SLOs, DORA Metrics & Production Alerting
Build a production SRE platform with SLO tracking, DORA metrics, and alerting.
Introduction
30 Second Summary
Every production service breaks eventually. The difference between a minor blip and a full-blown outage comes down to one thing: how fast your team detects, understands, and resolves the problem.
In this project, you will transform your existing observability stack into a complete SRE platform by adding SLO-driven alerting, DORA performance measurement, and unified dashboards that let you drill down from a metric spike to the exact log line and trace causing it.
What You'll Build
A production-grade reliability engineering platform built on top of your existing Prometheus, Grafana, Loki, and Tempo stack, with five dashboards provisioned entirely as code.
By the end of this project, you'll have:
- A working SLO framework with error budget tracking, multi-window burn rate alerts, and an Error Budget Policy that connects reliability engineering to business decisions.
- A DORA metrics dashboard with deployment frequency, lead time, change failure rate, and MTTR classified against Elite/High/Medium/Low industry benchmarks.
- A complete alerting system with Alertmanager route trees, inhibition rules, and structured Slack notifications using Go templates.
- A unified observability dashboard enabling metric-to-log-to-trace drill-down correlation across your entire stack.
- Secret Mission: inject chaos into your service to watch SLO burn rate alerts fire in real-time, practice incident response, and measure your MTTR against DORA benchmarks.
Are there any prerequisites?
Yes. This project is Part 2 of a series. You need a fully running observability stack from Part 1 ("Deploy an Observability Stack with Terraform") with Prometheus, Grafana, Loki, Tempo, Alertmanager, Node Exporter, Blackbox Exporter, OTel Collector, and an instrumented Flask sample app all deployed via Docker Compose and Terraform.
You will also need a free Slack workspace to receive alert notifications.
Verify Your Stack and Set Up Slack Notifications
Your observability stack from Project 1 is the foundation for everything you will build in this project. Before adding SLOs, alerting, and dashboards, you need to confirm all nine services are running and healthy.
You also need a real notification channel so alerts can reach you outside of the Grafana UI. In this step, you will bring your stack online, set up a Slack workspace for alert delivery, and wire it into Alertmanager.
In this step, get ready to:
- Verify your observability stack is running with all services healthy.
- Create a Slack channel and Incoming Webhook for alert notifications.
- Update your Alertmanager configuration with the real webhook URL and confirm connectivity.
Verify your observability stack is running
Your Project 1 stack includes Prometheus, Grafana, Loki, Tempo, Alertmanager, Node Exporter, Blackbox Exporter, the OTel Collector, and a Flask sample app. All of these need to be running before you proceed.
- Switch back to your terminal and navigate to the terraform/ directory inside your observability stack project folder.
- Bring up your stack by running this command:
terraform apply -auto-approveWhat does this command do?
Terraform reads your docker-compose.yaml and ensures all nine containers are created and running. If the stack is already up, Terraform reports no changes needed. If containers were stopped or removed, it recreates them.
- Confirm the stack is healthy by checking each service in your browser:
- Open http://localhost:3000 to verify Grafana loads.
- Open http://localhost:9090 to verify Prometheus loads.
- Open http://localhost:5000 to verify the Flask sample app responds.
Services not loading?
If any URL does not respond, check that Docker Desktop is running. Open Docker Desktop and confirm you see all containers listed as running.
If Terraform reports errors, try running terraform destroy -auto-approve followed by terraform apply -auto-approve to recreate from scratch.
Help me troubleshoot why my observability stack containers are not starting after terraform apply
Create a Slack workspace and Incoming Webhook
Alertmanager needs somewhere to send notifications when alerts fire. You will set up a Slack channel dedicated to DevOps alerts and create an Incoming Webhook that gives Alertmanager a URL to POST messages to.
- Go to https://slack.com/get-started#/createnew in your browser to create a free Slack workspace (or use an existing one).
- Once inside your workspace, create a new channel called #DevOps-Alerts.
Why a dedicated alerts channel?
A dedicated channel keeps alert noise separate from team conversation. On-call engineers can mute other channels and monitor only #DevOps-Alerts during an incident.
Now you need to create a Slack app with Incoming Webhooks enabled. This generates a unique URL that Alertmanager will use to deliver notifications.
- Open https://api.slack.com/apps in your browser.
- Click Create New App.
- Select From scratch.
- Enter a name for your app (e.g. Observability Alerts) and select your workspace.
- Click Create App.
- In the left sidebar of your app settings, click Incoming Webhooks.
- Toggle Activate Incoming Webhooks to On.
- Click Add New Webhook to Workspace.
- Select the #DevOps-Alerts channel from the dropdown.
- Click Allow.
You will be redirected back to your app settings page. Your new webhook URL appears under Webhook URLs for Your Workspace.
- Copy the webhook URL. It looks like https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX.
Keep your webhook URL private
This URL is a secret. Anyone with it can post messages to your Slack channel. Do not commit it to a public repository.
Not seeing Add New Webhook to Workspace?
Make sure you toggled Activate Incoming Webhooks to On first. The button only appears after activation.
If you are not a workspace admin, you may need the workspace owner to approve app installations.
Help me set up a Slack Incoming Webhook for my Alertmanager notifications
Update Alertmanager with your Slack webhook URL
Project 1 left a placeholder value in your Alertmanager configuration. You will now replace it with your real webhook URL so that alerts actually reach Slack.
- In VS Code, open configs/alertmanager/alertmanager.yml.
- Find this line:
slack_api_url: 'your-slack-webhook-url-here'- Replace your-slack-webhook-url-here with the actual webhook URL you copied from Slack, so the line looks like this:
slack_api_url: 'https://hooks.slack.com/services/YOUR/ACTUAL/WEBHOOK-URL'What does slack_api_url do?
This is the global Slack webhook URL that Alertmanager uses as the default destination for all slack_configs receivers. When an alert fires and routes to a Slack receiver, Alertmanager sends an HTTP POST to this URL with a JSON payload containing the alert details.
- Save alertmanager.yml.
Alertmanager reads its config file at startup. Since you changed the mounted config file, you need to restart the Alertmanager container so it picks up the new URL.
- Navigate to the root of your observability stack project (the directory containing docker-compose.yaml) and restart the Alertmanager service by running:
docker compose restart alertmanagerWhy restart instead of recreate?
The docker compose restart command stops and starts the container without removing it. Since the config file is volume-mounted, Alertmanager re-reads the updated alertmanager.yml on startup. No rebuild or recreation is needed.
- Verify Alertmanager is running by opening http://localhost:9093 in your browser. You should see the Alertmanager web UI.
- Verify Slack connectivity by sending a test message with this command (replace the URL at the end with your actual webhook URL):
curl -X POST -H 'Content-type: application/json' --data '{"text":"Alertmanager test - connection successful!"}' https://hooks.slack.com/services/YOUR/ACTUAL/WEBHOOK-URLReplace the URL in the curl command
Paste your actual webhook URL at the end of the curl command, replacing the placeholder. This sends a test message directly to your #DevOps-Alerts channel.
- Switch to Slack and check the #DevOps-Alerts channel. You should see the message "Alertmanager test - connection successful!" appear.
No message in Slack?
Double-check that the webhook URL in your curl command matches exactly what Slack generated. Even a single extra space or missing character will cause a failure.
If you see invalid_payload in the terminal response, check that the JSON in --data is properly quoted with no missing braces or quotes.
Help me debug why my Slack webhook test message is not arriving in my DevOps-Alerts channel
✔️ Awesome, I've got everything!
Great. Double check you have saved alertmanager.yml and confirmed the test message arrived in Slack.
ⓧ I'd like to double check the full code
Your configs/alertmanager/alertmanager.yml should have the real Slack webhook URL in place of the old placeholder. The key line to verify is:
slack_api_url: 'https://hooks.slack.com/services/YOUR/ACTUAL/WEBHOOK-URL'The rest of the file remains the same as Project 1. The only change in this step is replacing your-slack-webhook-url-here with your real URL.
Your observability stack is running and your alerting pipeline can reach Slack. Next up, you will define SLOs and build the recording rules that power error budget tracking.
Define SLOs and Error Budget Recording Rules
Your observability stack is collecting metrics, logs, and traces. But data alone does not tell you whether your service is reliable enough for its users.
In this step, you will define Service Level Objectives (SLOs) that set explicit reliability targets. You will then implement Prometheus recording rules that continuously compute error ratios and burn rates across multiple time windows, and build a Grafana dashboard to visualize your error budget in real-time.
In this step, get ready to:
- Define three SLOs with documented reasoning.
- Write Prometheus recording rules to compute error ratios and burn rates across 5m, 30m, 1h, and 6h windows.
- Build a provisioned-as-code Grafana dashboard showing SLI vs SLO gauges, error budget remaining, and burn rate trends.
Define three SLOs in a documentation file
Before writing any alerting rules, you need clearly defined reliability targets. An SLO states how reliable a service should be for its users over a specific time window.
- Create a new file at docs/slo-definitions.md by running:
mkdir -p docs
touch docs/slo-definitions.md- Open docs/slo-definitions.md in your editor and paste the following content:
# SLO Definitions
## 1. Availability SLO
- **Target**: 99.5% of HTTP probes return 2xx over a rolling 30-day window
- **SLI**: `probe_success{job="blackbox"}` metric from Blackbox Exporter
- **Error budget**: 0.5% = 216 minutes of allowed downtime per 30 days
- **Reasoning**: Our service is internal-facing. 99.5% balances reliability
with the team's capacity to ship features. A stricter target (99.9%) would
leave only 43 minutes of budget, making routine maintenance risky.
## 2. Latency SLO
- **Target**: 95% of requests complete under 500ms
- **SLI**: Request duration histogram from the sample app (`http_request_duration_seconds`)
- **Error budget**: 5% of requests may exceed 500ms
- **Reasoning**: 500ms is the threshold where users perceive delay. A 95%
target acknowledges that background tasks and cold starts will occasionally
exceed this without impacting user experience.
## 3. Error Rate SLO
- **Target**: 99% of requests succeed with non-5xx responses
- **SLI**: HTTP response status codes from the sample app
- **Error budget**: 1% of requests may return 5xx errors
- **Reasoning**: 1% error budget provides enough room for transient failures
during deployments while ensuring the vast majority of user requests succeed.
## Error Budget Calculations
| SLO | Target | Budget (30 days) |
|-----|--------|-------------------|
| Availability | 99.5% | 216 minutes of downtime |
| Latency | 95% | 5% of requests may be slow |
| Error Rate | 99% | 1% of requests may fail |What is an error budget?
An error budget is the inverse of your SLO target expressed as allowed unreliability. For 99.5% availability over 30 days: (1 - 0.995) x 30 x 24 x 60 = 216 minutes of allowed downtime.
As long as you stay within budget, the team can ship features freely. When the budget runs low, reliability work takes priority. This is the core mechanism that connects SLOs to business decisions.
- Save docs/slo-definitions.md.
- Confirm the file exists by running:
ls docs/You should see slo-definitions.md in the output.
Write Prometheus recording rules for SLO burn rates
Recording rules pre-compute expensive PromQL expressions on a schedule and store the results as new time series. This makes your SLO dashboard queries instant instead of computing ratios on every page load.
- Create the recording rules file by running:
touch configs/prometheus/slo-rules.yml- Open configs/prometheus/slo-rules.yml and paste the following recording rules:
# SLO Recording Rules
# Computes error ratios and burn rates across multiple time windows
# following the level:metric:operations naming convention.
groups:
- name: slo_availability_error_ratios
rules:
# Availability SLO: error ratio = 1 - probe_success
- record: job:slo_availability_errors:ratio_rate5m
expr: 1 - avg_over_time(probe_success{job="blackbox"}[5m])
labels:
slo: availability
- record: job:slo_availability_errors:ratio_rate30m
expr: 1 - avg_over_time(probe_success{job="blackbox"}[30m])
labels:
slo: availability
- record: job:slo_availability_errors:ratio_rate1h
expr: 1 - avg_over_time(probe_success{job="blackbox"}[1h])
labels:
slo: availability
- record: job:slo_availability_errors:ratio_rate6h
expr: 1 - avg_over_time(probe_success{job="blackbox"}[6h])
labels:
slo: availability
- name: slo_latency_error_ratios
rules:
# Latency SLO: fraction of requests exceeding 500ms
- record: job:slo_latency_errors:ratio_rate5m
expr: |
1 - (
sum(rate(http_request_duration_seconds_bucket{le="0.5"}[5m]))
/
sum(rate(http_request_duration_seconds_count[5m]))
)
labels:
slo: latency
- record: job:slo_latency_errors:ratio_rate30m
expr: |
1 - (
sum(rate(http_request_duration_seconds_bucket{le="0.5"}[30m]))
/
sum(rate(http_request_duration_seconds_count[30m]))
)
labels:
slo: latency
- record: job:slo_latency_errors:ratio_rate1h
expr: |
1 - (
sum(rate(http_request_duration_seconds_bucket{le="0.5"}[1h]))
/
sum(rate(http_request_duration_seconds_count[1h]))
)
labels:
slo: latency
- record: job:slo_latency_errors:ratio_rate6h
expr: |
1 - (
sum(rate(http_request_duration_seconds_bucket{le="0.5"}[6h]))
/
sum(rate(http_request_duration_seconds_count[6h]))
)
labels:
slo: latency
- name: slo_error_rate_ratios
rules:
# Error Rate SLO: fraction of 5xx responses
- record: job:slo_error_rate_errors:ratio_rate5m
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
labels:
slo: error_rate
- record: job:slo_error_rate_errors:ratio_rate30m
expr: |
sum(rate(http_requests_total{status=~"5.."}[30m]))
/
sum(rate(http_requests_total[30m]))
labels:
slo: error_rate
- record: job:slo_error_rate_errors:ratio_rate1h
expr: |
sum(rate(http_requests_total{status=~"5.."}[1h]))
/
sum(rate(http_requests_total[1h]))
labels:
slo: error_rate
- record: job:slo_error_rate_errors:ratio_rate6h
expr: |
sum(rate(http_requests_total{status=~"5.."}[6h]))
/
sum(rate(http_requests_total[6h]))
labels:
slo: error_rate
- name: slo_burn_rates
rules:
# Burn rate = error ratio / error budget
# Availability budget = 0.005 (for 99.5% target)
- record: job:slo_availability:burn_rate5m
expr: job:slo_availability_errors:ratio_rate5m / 0.005
labels:
slo: availability
- record: job:slo_availability:burn_rate30m
expr: job:slo_availability_errors:ratio_rate30m / 0.005
labels:
slo: availability
- record: job:slo_availability:burn_rate1h
expr: job:slo_availability_errors:ratio_rate1h / 0.005
labels:
slo: availability
- record: job:slo_availability:burn_rate6h
expr: job:slo_availability_errors:ratio_rate6h / 0.005
labels:
slo: availability
# Latency budget = 0.05 (for 95% target)
- record: job:slo_latency:burn_rate5m
expr: job:slo_latency_errors:ratio_rate5m / 0.05
labels:
slo: latency
- record: job:slo_latency:burn_rate30m
expr: job:slo_latency_errors:ratio_rate30m / 0.05
labels:
slo: latency
- record: job:slo_latency:burn_rate1h
expr: job:slo_latency_errors:ratio_rate1h / 0.05
labels:
slo: latency
- record: job:slo_latency:burn_rate6h
expr: job:slo_latency_errors:ratio_rate6h / 0.05
labels:
slo: latency
# Error rate budget = 0.01 (for 99% target)
- record: job:slo_error_rate:burn_rate5m
expr: job:slo_error_rate_errors:ratio_rate5m / 0.01
labels:
slo: error_rate
- record: job:slo_error_rate:burn_rate30m
expr: job:slo_error_rate_errors:ratio_rate30m / 0.01
labels:
slo: error_rate
- record: job:slo_error_rate:burn_rate1h
expr: job:slo_error_rate_errors:ratio_rate1h / 0.01
labels:
slo: error_rate
- record: job:slo_error_rate:burn_rate6h
expr: job:slo_error_rate_errors:ratio_rate6h / 0.01
labels:
slo: error_rateWhat does this code do?
This file defines four rule groups following the level:metric:operations naming convention from Prometheus best practices.
- The first three groups compute error ratios across 5m, 30m, 1h, and 6h windows for each SLO.
- The fourth group divides each error ratio by its error budget to produce a burn rate. A burn rate of 1x means you are consuming budget exactly at the rate your SLO allows.
- A burn rate of 14.4x (fast burn) means 2% of your 30-day budget is consumed in just 1 hour. A burn rate of 5x (slow burn) means 5% consumed in 6 hours.
Now you need to tell Prometheus to load this new rules file. Two config changes are required.
- Open configs/prometheus/prometheus.yml and find the rule_files section.
- Find this line:
rule_files:
- /etc/prometheus/alert-rules.yml- Add slo-rules.yml to the list so it looks like this:
rule_files:
- /etc/prometheus/alert-rules.yml
- /etc/prometheus/slo-rules.yml- Open docker-compose.yaml and find the Prometheus service's volumes section.
- Find the existing alert-rules volume mount:
- ./configs/prometheus/alert-rules.yml:/etc/prometheus/alert-rules.yml:ro- Add a new line directly below it for the SLO rules file:
- ./configs/prometheus/slo-rules.yml:/etc/prometheus/slo-rules.yml:ro- Save both files.
- Restart the Prometheus container to pick up the new configuration by running:
docker compose restart prometheus- Verify the recording rules loaded successfully by opening Prometheus at http://localhost:9090/rules in your browser.
You should see your four rule groups listed: slo_availability_error_ratios, slo_latency_error_ratios, slo_error_rate_ratios, and slo_burn_rates.
Don't see your rule groups?
- Check the Prometheus logs for YAML parsing errors by running docker compose logs prometheus. A missing colon or incorrect indentation in slo-rules.yml will prevent Prometheus from starting.
- Confirm the volume mount path in docker-compose.yaml matches exactly: ./configs/prometheus/slo-rules.yml:/etc/prometheus/slo-rules.yml:ro.
- Verify that prometheus.yml lists the file as /etc/prometheus/slo-rules.yml (the container path, not the host path).
Help me debug why my Prometheus recording rules aren't loading.
Build the SLO and error budget Grafana dashboard
Your recording rules are now producing time series. The next step is to visualize them in a provisioned-as-code dashboard that shows SLI vs SLO gauges, error budget remaining, and burn rate trends.
- Create the dashboard JSON file by running:
touch configs/grafana/dashboards/slo-error-budget.json- Open configs/grafana/dashboards/slo-error-budget.json and paste the following dashboard definition:
{
"uid": "slo-error-budget",
"title": "SLO & Error Budget",
"tags": ["slo", "error-budget", "sre"],
"timezone": "browser",
"editable": true,
"graphTooltip": 1,
"time": {
"from": "now-6h",
"to": "now"
},
"refresh": "30s",
"schemaVersion": 39,
"version": 1,
"panels": [
{
"id": 1,
"type": "gauge",
"title": "Availability SLI vs SLO (99.5%)",
"gridPos": { "x": 0, "y": 0, "w": 8, "h": 8 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [
{
"expr": "1 - job:slo_availability_errors:ratio_rate1h",
"legendFormat": "Availability"
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"min": 0.99,
"max": 1,
"thresholds": {
"mode": "absolute",
"steps": [
{ "value": null, "color": "red" },
{ "value": 0.995, "color": "green" }
]
}
},
"overrides": []
},
"options": {
"reduceOptions": { "calcs": ["lastNotNull"] },
"showThresholdLabels": false,
"showThresholdMarkers": true
}
},
{
"id": 2,
"type": "gauge",
"title": "Latency SLI vs SLO (95%)",
"gridPos": { "x": 8, "y": 0, "w": 8, "h": 8 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [
{
"expr": "1 - job:slo_latency_errors:ratio_rate1h",
"legendFormat": "Latency"
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"min": 0.9,
"max": 1,
"thresholds": {
"mode": "absolute",
"steps": [
{ "value": null, "color": "red" },
{ "value": 0.95, "color": "green" }
]
}
},
"overrides": []
},
"options": {
"reduceOptions": { "calcs": ["lastNotNull"] },
"showThresholdLabels": false,
"showThresholdMarkers": true
}
},
{
"id": 3,
"type": "gauge",
"title": "Error Rate SLI vs SLO (99%)",
"gridPos": { "x": 16, "y": 0, "w": 8, "h": 8 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [
{
"expr": "1 - job:slo_error_rate_errors:ratio_rate1h",
"legendFormat": "Error Rate"
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"min": 0.95,
"max": 1,
"thresholds": {
"mode": "absolute",
"steps": [
{ "value": null, "color": "red" },
{ "value": 0.99, "color": "green" }
]
}
},
"overrides": []
},
"options": {
"reduceOptions": { "calcs": ["lastNotNull"] },
"showThresholdLabels": false,
"showThresholdMarkers": true
}
},
{
"id": 4,
"type": "bargauge",
"title": "Error Budget Remaining",
"gridPos": { "x": 0, "y": 8, "w": 24, "h": 6 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [
{
"expr": "1 - (job:slo_availability_errors:ratio_rate1h / 0.005)",
"legendFormat": "Availability Budget"
},
{
"expr": "1 - (job:slo_latency_errors:ratio_rate1h / 0.05)",
"legendFormat": "Latency Budget"
},
{
"expr": "1 - (job:slo_error_rate_errors:ratio_rate1h / 0.01)",
"legendFormat": "Error Rate Budget"
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"min": 0,
"max": 1,
"thresholds": {
"mode": "absolute",
"steps": [
{ "value": null, "color": "red" },
{ "value": 0.25, "color": "orange" },
{ "value": 0.5, "color": "yellow" },
{ "value": 0.75, "color": "green" }
]
}
},
"overrides": []
},
"options": {
"reduceOptions": { "calcs": ["lastNotNull"] },
"orientation": "horizontal",
"displayMode": "gradient",
"showUnfilled": true
}
},
{
"id": 5,
"type": "timeseries",
"title": "Burn Rate (Fast 14.4x / Slow 5x Thresholds)",
"gridPos": { "x": 0, "y": 14, "w": 24, "h": 8 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [
{
"expr": "job:slo_availability:burn_rate5m",
"legendFormat": "Availability 5m burn"
},
{
"expr": "job:slo_availability:burn_rate1h",
"legendFormat": "Availability 1h burn"
},
{
"expr": "job:slo_availability:burn_rate6h",
"legendFormat": "Availability 6h burn"
}
],
"fieldConfig": {
"defaults": {
"custom": {
"lineWidth": 2,
"fillOpacity": 10,
"showPoints": "never",
"thresholdsStyle": { "mode": "dashed" }
},
"thresholds": {
"mode": "absolute",
"steps": [
{ "value": null, "color": "green" },
{ "value": 5, "color": "orange" },
{ "value": 14.4, "color": "red" }
]
}
},
"overrides": []
},
"options": {
"legend": { "displayMode": "table", "placement": "bottom" },
"tooltip": { "mode": "multi" }
}
},
{
"id": 6,
"type": "timeseries",
"title": "SLO Compliance History (7-day)",
"gridPos": { "x": 0, "y": 22, "w": 12, "h": 7 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [
{
"expr": "1 - avg_over_time(job:slo_availability_errors:ratio_rate1h[7d])",
"legendFormat": "Availability (7d)"
},
{
"expr": "1 - avg_over_time(job:slo_latency_errors:ratio_rate1h[7d])",
"legendFormat": "Latency (7d)"
},
{
"expr": "1 - avg_over_time(job:slo_error_rate_errors:ratio_rate1h[7d])",
"legendFormat": "Error Rate (7d)"
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"min": 0.9,
"max": 1,
"custom": {
"lineWidth": 2,
"fillOpacity": 5
}
},
"overrides": []
},
"options": {
"legend": { "displayMode": "list", "placement": "bottom" },
"tooltip": { "mode": "multi" }
}
},
{
"id": 7,
"type": "timeseries",
"title": "SLO Compliance History (30-day)",
"gridPos": { "x": 12, "y": 22, "w": 12, "h": 7 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [
{
"expr": "1 - avg_over_time(job:slo_availability_errors:ratio_rate1h[30d])",
"legendFormat": "Availability (30d)"
},
{
"expr": "1 - avg_over_time(job:slo_latency_errors:ratio_rate1h[30d])",
"legendFormat": "Latency (30d)"
},
{
"expr": "1 - avg_over_time(job:slo_error_rate_errors:ratio_rate1h[30d])",
"legendFormat": "Error Rate (30d)"
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"min": 0.9,
"max": 1,
"custom": {
"lineWidth": 2,
"fillOpacity": 5
}
},
"overrides": []
},
"options": {
"legend": { "displayMode": "list", "placement": "bottom" },
"tooltip": { "mode": "multi" }
}
}
],
"templating": { "list": [] },
"annotations": { "list": [] }
}What does this dashboard include?
- Three gauge panels (row 1) showing real-time SLI values compared to their SLO thresholds. Green means the SLI is above target.
- A bar gauge panel (row 2) showing remaining error budget for each SLO, colored from red (depleted) through orange and yellow to green (healthy).
- A time series panel (row 3) showing burn rate over time with dashed threshold lines at 5x (slow burn) and 14.4x (fast burn).
- Two compliance history panels (row 4) showing 7-day and 30-day rolling SLO compliance as line charts.
- Save configs/grafana/dashboards/slo-error-budget.json.
- Restart Grafana to pick up the new dashboard by running:
docker compose restart grafana- Open Grafana at http://localhost:3000 in your browser.
- Click **Dashboards** in the left sidebar.
- Look for SLO & Error Budget in the dashboard list.
You should see the dashboard listed and, when opened, the gauge panels showing current SLI values with green or red coloring based on whether they meet the SLO targets.
Dashboard not appearing in Grafana?
- Confirm the file is in the correct directory: configs/grafana/dashboards/slo-error-budget.json. The provisioning config from Project 1 points to this exact folder.
- Check for JSON syntax errors by running cat configs/grafana/dashboards/slo-error-budget.json | python3 -m json.tool. If the command fails, fix the JSON before restarting Grafana.
- Look at Grafana logs with docker compose logs grafana for provisioning errors.
Help me figure out why my provisioned dashboard isn't showing up in Grafana.
Why recording rules instead of raw queries?
Without recording rules, every time you open this dashboard Grafana would send complex PromQL queries spanning large time ranges to Prometheus. Recording rules pre-compute these results every evaluation cycle and store them as instant-lookup metrics.
This matters especially for burn rates, where the query involves dividing a windowed ratio by a constant. Pre-computing means your dashboard loads in milliseconds regardless of data volume.
Your SLO framework is now live. Prometheus is continuously computing error ratios and burn rates, and your dashboard shows exactly how much error budget remains. Next up, you will write the Error Budget Policy that turns these numbers into team decisions.
Write the Error Budget Policy
Your SLO recording rules are computing error ratios and burn rates in real-time. But SLOs without a documented response plan are just dashboards that nobody acts on.
In this step, you will create the policy document that turns your SLO numbers into organizational decisions. You will also validate that your dashboard math is correct by calculating expected error budget values and then depleting them with real traffic.
In this step, get ready to:
- Create the Error Budget Policy document defining escalation actions.
- Calculate error budget values and verify dashboard accuracy.
- Generate traffic and observe error budget depletion in real-time.
Create the error budget policy document
An Error Budget Policy is the contract between your reliability team and your feature teams. It defines what actions are triggered when your error budget crosses specific thresholds.
- Create the file docs/error-budget-policy.md by running:
touch docs/error-budget-policy.md- Open docs/error-budget-policy.md in your editor and paste the following content:
# Error Budget Policy
## Purpose
This policy defines the organizational response when error budgets are consumed
beyond acceptable thresholds. It connects SLO compliance to deployment decisions
and reliability investment.
## SLO Error Budget Values
| SLO | Target | Budget Formula | Allowed Failures (30-day window) |
|-----|--------|---------------|----------------------------------|
| Availability | 99.5% | (1 - 0.995) x 30 x 24 x 60 | 216 minutes of downtime |
| Latency | 95% | (1 - 0.95) x 30 x 24 x 60 | 2,160 minutes of slow requests |
| Error Rate | 99% | (1 - 0.99) x 30 x 24 x 60 | 432 minutes of elevated errors |
## Escalation Thresholds
### 50% Budget Consumed - Caution
**Trigger**: Error budget for any SLO drops below 50% remaining.
**Actions**:
- Increase monitoring frequency (review dashboards daily instead of weekly)
- Halt risky deployments (feature flags, large refactors, infrastructure migrations)
- Notify the on-call SRE and engineering lead
- Begin root-cause investigation of budget consumption pattern
### 100% Budget Consumed - Critical
**Trigger**: Error budget for any SLO is fully exhausted (0% remaining).
**Actions**:
- Immediate feature freeze: no new feature deployments until budget recovers
- Reliability sprint: all engineering effort redirected to stability fixes
- Incident review within 24 hours to identify contributing factors
- Stakeholder communication: product and leadership notified of freeze
## Decision Ownership
| Role | Responsibility |
|------|---------------|
| SRE On-Call | Monitors budget consumption, triggers 50% escalation |
| Engineering Lead | Approves or overrides deployment freeze at 100% |
| Product Owner | Acknowledges feature freeze, reprioritizes sprint |
| VP Engineering | Final escalation authority for policy exceptions |
## SLO Review Cadence
- **Weekly**: Review error budget burn rate trends in team standup
- **Monthly**: Full SLO review meeting (adjust targets if consistently over/under)
- **Quarterly**: Recalibrate SLO targets based on business requirements and user feedback
- **After incidents**: Emergency review if a single event consumes >25% of any budget
## Policy Exceptions
Exceptions to the feature freeze require written approval from the Engineering Lead
and must document the business justification and rollback plan.What does this policy do?
The two escalation thresholds (50% and 100%) create a progressive response. At 50%, teams shift from feature work to caution mode. At 100%, all engineering effort goes to reliability.
Without this policy, SLO violations become just another ignored dashboard metric. The policy gives teams explicit authority to halt risky changes when reliability is at stake.
- Save docs/error-budget-policy.md.
- Confirm the file exists by running:
ls docs/- You should see both slo-definitions.md and error-budget-policy.md listed.
Don't see the file?
Make sure you are running ls docs/ from the root of your observability-stack project directory. If the docs/ folder does not exist, create it first with mkdir -p docs.
Help me create the error budget policy document.
Calculate error budget values and verify the dashboard
The error budget formula is: budget = (1 - SLO target) x measurement window. Verifying this math against your dashboard panels confirms your recording rules are computing correctly.
- Work through the budget calculation for each SLO to confirm the numbers match what your dashboard displays:
Availability SLO (99.5%):
Budget = (1 - 0.995) x 30 days x 24 hours x 60 minutes
Budget = 0.005 x 43,200
Budget = 216 minutes of allowed downtime
Latency SLO (95%):
Budget = (1 - 0.95) x 30 days x 24 hours x 60 minutes
Budget = 0.05 x 43,200
Budget = 2,160 minutes where slow requests are acceptable
Error Rate SLO (99%):
Budget = (1 - 0.99) x 30 days x 24 hours x 60 minutes
Budget = 0.01 x 43,200
Budget = 432 minutes of allowed elevated errorsWhy minutes instead of percentages?
Converting error budgets to minutes makes them tangible. Saying "you have 216 minutes of allowed downtime per month" is more actionable than "your budget is 0.5%". Teams can immediately estimate whether a planned maintenance window fits within the budget.
- Open your Grafana SLO dashboard at http://localhost:3000.
- Navigate to the SLO and Error Budget dashboard you provisioned in Step 2.
- Check the error budget remaining panels. With no failures generated yet, all three budgets should show close to 100% remaining.
What should I see?
The bar gauge panels should display green (or near-green) bars showing approximately 100% budget remaining for each SLO. The exact percentage depends on how long your stack has been running and whether any probes have failed naturally.
Dashboard showing 0% or no data?
Your recording rules may not have had enough time to generate data. Prometheus needs at least one full evaluation interval (usually 1-5 minutes) to populate recording rule metrics.
Check that your slo-rules.yml is loaded by querying http://localhost:9090/rules in your browser. If the rules are missing, verify your docker-compose.yaml volume mount and prometheus.yml rule_files list from Step 2.
Help me troubleshoot my SLO dashboard panels showing no data.
Generate traffic and observe error budget depletion
To validate the full loop from SLO violation to budget depletion on your dashboard, you need to send a mix of successful and failing requests to your sample app.
- Generate a mix of successful and failing traffic by running this script in your terminal:
# Send 80 successful requests and 20 failing requests
for i in $(seq 1 80); do
curl -s http://localhost:5000/api/orders > /dev/null
done
for i in $(seq 1 20); do
curl -s http://localhost:5000/api/orders/fail > /dev/null
done
echo "Traffic generation complete: 80 success, 20 failures"What does this traffic do?
The /api/orders endpoint returns successful 2xx responses. The /api/orders/fail endpoint returns 5xx errors. A 20% failure rate significantly exceeds your 99% Error Rate SLO target, which should cause rapid error budget consumption visible on your dashboard.
- Wait 1-2 minutes for Prometheus to scrape the new metrics and evaluate your recording rules.
- Return to your SLO and Error Budget dashboard in Grafana and refresh the page.
- Observe the Error Rate SLO panel. The error budget remaining should have decreased noticeably from its previous value.
What should I see?
The Error Rate SLO budget remaining panel should show a visible drop. The burn rate time series panel should show a spike corresponding to when you generated the failing traffic. The Availability SLO budget may also show some depletion if the Blackbox Exporter detected the errors.
- To see more dramatic depletion, run additional waves of failing traffic by running:
# Generate another burst of failures to accelerate budget depletion
for i in $(seq 1 50); do
curl -s http://localhost:5000/api/orders/fail > /dev/null
sleep 0.5
done
echo "Additional failure traffic sent"Why space out the requests?
The sleep 0.5 between requests spreads failures across multiple Prometheus scrape intervals. This makes the burn rate trend line more visible as a sustained elevation rather than a single spike.
- Refresh your Grafana dashboard again after 2-3 minutes.
- Confirm that the error budget remaining percentage has dropped further and the burn rate panel shows sustained elevation above the baseline.
Error budget not depleting?
Make sure your sample app is running and responding. Test by running curl http://localhost:5000/api/orders manually. You should get a JSON response.
If the app responds but the dashboard does not update, check that your recording rules reference the correct metric names from your app's Prometheus exporter. Query http://localhost:9090/graph and search for your SLO recording rule metrics to see if they have data.
Help me troubleshoot why my error budget is not depleting after sending traffic.
Your error budget policy is now documented and validated with real traffic. Next up, you will build the DORA metrics dashboard to track your team's deployment performance against industry benchmarks.
Build the DORA Metrics Dashboard
Your SLO framework tells you whether your service is reliable enough. But how do you know if your team is shipping fast enough and recovering quickly from failures?
DORA metrics answer that question. They measure software delivery performance across four dimensions: how often you deploy, how long changes take to reach production, how often deployments fail, and how quickly you recover. In this step, you will create Prometheus recording rules for all four DORA metrics, build a Grafana dashboard with benchmark classification, and document toil in your workflow.
In this step, get ready to:
- Create Prometheus recording rules tracking Deployment Frequency, Lead Time, Change Failure Rate, and MTTR.
- Build a provisioned DORA dashboard with Elite/High/Medium/Low benchmark panels.
- Identify toil in your workflow and propose automation strategies.
Create DORA recording rules
DORA metrics rely on custom metrics that your deployment pipeline pushes to Prometheus. The sample app from Project 1 can expose these via its metrics endpoint. Your recording rules will aggregate these raw counters into meaningful rates and averages.
- Create a new file at configs/prometheus/dora-rules.yml by pasting the following recording rules:
# DORA Metrics Recording Rules
# Tracks Deployment Frequency, Lead Time for Changes,
# Change Failure Rate, and Mean Time to Restore (MTTR)
groups:
- name: dora-metrics
interval: 1m
rules:
# --- Deployment Frequency ---
# Rate of deployments per hour over different windows
- record: job:deployments:rate1h
expr: sum(rate(deployments_total[1h]))
- record: job:deployments:rate24h
expr: sum(rate(deployments_total[24h]))
# Daily deployment count for trend analysis
- record: job:deployments:count24h
expr: sum(increase(deployments_total[24h]))
# --- Lead Time for Changes (sub-intervals) ---
# Time from commit to pipeline trigger
- record: job:deployment_lead_time_commit_seconds:mean5m
expr: |
sum(rate(deployment_lead_time_commit_seconds_sum[5m]))
/
sum(rate(deployment_lead_time_commit_seconds_count[5m]))
# Time from pipeline trigger to pipeline complete
- record: job:deployment_lead_time_pipeline_trigger_seconds:mean5m
expr: |
sum(rate(deployment_lead_time_pipeline_trigger_seconds_sum[5m]))
/
sum(rate(deployment_lead_time_pipeline_trigger_seconds_count[5m]))
# Time from pipeline complete to deployment confirmed
- record: job:deployment_lead_time_pipeline_complete_seconds:mean5m
expr: |
sum(rate(deployment_lead_time_pipeline_complete_seconds_sum[5m]))
/
sum(rate(deployment_lead_time_pipeline_complete_seconds_count[5m]))
# Total lead time (commit to deployment confirmed)
- record: job:deployment_lead_time_deploy_confirm_seconds:mean5m
expr: |
sum(rate(deployment_lead_time_deploy_confirm_seconds_sum[5m]))
/
sum(rate(deployment_lead_time_deploy_confirm_seconds_count[5m]))
# --- Change Failure Rate ---
# Ratio of failed deployments to total deployments
- record: job:deployment_failures_per_deployments:ratio_rate1h
expr: |
sum(rate(deployment_failures_total[1h]))
/
sum(rate(deployments_total[1h]))
- record: job:deployment_failures_per_deployments:ratio_rate24h
expr: |
sum(rate(deployment_failures_total[24h]))
/
sum(rate(deployments_total[24h]))
# --- Mean Time to Restore (MTTR) ---
# Average incident resolution time
- record: job:incident_resolution_seconds:mean5m
expr: |
sum(rate(incident_resolution_seconds_sum[5m]))
/
sum(rate(incident_resolution_seconds_count[5m]))
# 95th percentile recovery time
- record: job:incident_resolution_seconds:p95
expr: histogram_quantile(0.95, sum(rate(incident_resolution_seconds_bucket[1h])) by (le))What does this file do?
Each recording rule pre-computes an expensive PromQL query and stores the result as a new time series. This makes dashboard queries instant instead of recalculating across thousands of raw samples every refresh.
The naming convention follows the Prometheus standard: level:metric:operations. For example, job:deployments:rate1h means aggregated at the job level, the deployments metric, computed as a rate over 1 hour.
What are the DORA metrics?
- Deployment Frequency measures how often you deploy to production.
- Lead Time for Changes tracks the time from code commit to production. This file breaks it into sub-intervals: commit to pipeline trigger, pipeline trigger to pipeline complete, and pipeline complete to deployment confirmed.
- Change Failure Rate is the ratio of deployments that require immediate intervention (hotfixes, rollbacks).
- Mean Time to Restore (MTTR) measures how quickly you recover from a failed deployment.
- Save configs/prometheus/dora-rules.yml.
Now you need to tell Prometheus to load this new rule file. You will update both prometheus.yml and docker-compose.yaml.
- In configs/prometheus/prometheus.yml, find the rule_files section. It currently includes slo-rules.yml. Find this block:
rule_files:
- /etc/prometheus/alert-rules.yml
- /etc/prometheus/slo-rules.yml- Add /etc/prometheus/dora-rules.yml to the list, so it now looks like this:
rule_files:
- /etc/prometheus/alert-rules.yml
- /etc/prometheus/slo-rules.yml
- /etc/prometheus/dora-rules.yml- In docker-compose.yaml, find the Prometheus service volume mounts. You already have a mount for slo-rules.yml. Find this line:
- ./configs/prometheus/slo-rules.yml:/etc/prometheus/slo-rules.yml:ro- Add a new volume mount directly below it for the DORA rules file:
- ./configs/prometheus/dora-rules.yml:/etc/prometheus/dora-rules.yml:ro- Save both prometheus.yml and docker-compose.yaml.
- Restart the Prometheus container to load the new rules by running:
docker compose up -d prometheus- Open Prometheus at http://localhost:9090/rules in your browser. You should see a dora-metrics group listed with all your recording rules.
Don't see the dora-metrics group?
- Check that the volume mount path in docker-compose.yaml matches exactly: ./configs/prometheus/dora-rules.yml:/etc/prometheus/dora-rules.yml:ro.
- Verify the rule_files path in prometheus.yml uses the container path /etc/prometheus/dora-rules.yml, not the host path.
- Check Prometheus logs for YAML syntax errors by running docker compose logs prometheus.
Help me troubleshoot why my DORA recording rules are not loading in Prometheus.
Build the DORA metrics Grafana dashboard
With the recording rules active, you can now build a dashboard that classifies your team's performance against industry benchmarks. The DORA research classifies teams into four tiers based on their metric values.
What are the DORA benchmark tiers?
- Elite: on-demand deployment (multiple times/day), lead time under 1 hour, 0-2% change failure rate, recovery under 1 hour.
- High: daily to weekly deployment, lead time under 1 day, under 15% change failure rate, recovery under 1 day.
- Medium: weekly to monthly deployment, lead time under 1 week, under 30% change failure rate, recovery under 1 week.
- Low: less than monthly deployment, lead time over 1 week, over 30% change failure rate, recovery over 1 week.
These are not fixed thresholds. DORA determines tiers annually via cluster analysis. The values above are commonly-cited approximate benchmarks.
- Create the DORA dashboard by pasting the following JSON into a new file at configs/grafana/dashboards/dora-metrics.json:
{
"uid": "dora-metrics",
"title": "DORA Metrics",
"tags": ["dora", "sre", "performance"],
"timezone": "browser",
"editable": true,
"graphTooltip": 1,
"time": {
"from": "now-7d",
"to": "now"
},
"refresh": "30s",
"schemaVersion": 39,
"version": 1,
"panels": [
{
"type": "row",
"title": "Benchmark Classification",
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 },
"id": 1,
"collapsed": false
},
{
"type": "stat",
"title": "Deployment Frequency",
"description": "Elite: on-demand | High: daily-weekly | Medium: weekly-monthly | Low: <monthly",
"gridPos": { "h": 6, "w": 6, "x": 0, "y": 1 },
"id": 2,
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [
{
"expr": "job:deployments:count24h",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"displayName": "Deploys/day",
"color": { "mode": "thresholds" },
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "red", "value": null },
{ "color": "orange", "value": 0.25 },
{ "color": "yellow", "value": 1 },
{ "color": "green", "value": 7 }
]
}
},
"overrides": []
},
"options": {
"reduceOptions": { "calcs": ["lastNotNull"] },
"textMode": "auto",
"colorMode": "background",
"graphMode": "area"
}
},
{
"type": "stat",
"title": "Lead Time for Changes",
"description": "Elite: <1hr | High: <1 day | Medium: <1 week | Low: >1 week",
"gridPos": { "h": 6, "w": 6, "x": 6, "y": 1 },
"id": 3,
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [
{
"expr": "job:deployment_lead_time_deploy_confirm_seconds:mean5m",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "s",
"displayName": "Lead Time",
"color": { "mode": "thresholds" },
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "yellow", "value": 3600 },
{ "color": "orange", "value": 86400 },
{ "color": "red", "value": 604800 }
]
}
},
"overrides": []
},
"options": {
"reduceOptions": { "calcs": ["lastNotNull"] },
"textMode": "auto",
"colorMode": "background",
"graphMode": "area"
}
},
{
"type": "stat",
"title": "Change Failure Rate",
"description": "Elite: 0-2% | High: <15% | Medium: <30% | Low: >30%",
"gridPos": { "h": 6, "w": 6, "x": 12, "y": 1 },
"id": 4,
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [
{
"expr": "job:deployment_failures_per_deployments:ratio_rate24h * 100",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"displayName": "CFR",
"min": 0,
"max": 100,
"color": { "mode": "thresholds" },
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "yellow", "value": 2 },
{ "color": "orange", "value": 15 },
{ "color": "red", "value": 30 }
]
}
},
"overrides": []
},
"options": {
"reduceOptions": { "calcs": ["lastNotNull"] },
"textMode": "auto",
"colorMode": "background",
"graphMode": "area"
}
},
{
"type": "stat",
"title": "Mean Time to Restore",
"description": "Elite: <1hr | High: <1 day | Medium: <1 week | Low: >1 week",
"gridPos": { "h": 6, "w": 6, "x": 18, "y": 1 },
"id": 5,
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [
{
"expr": "job:incident_resolution_seconds:mean5m",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "s",
"displayName": "MTTR",
"color": { "mode": "thresholds" },
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "yellow", "value": 3600 },
{ "color": "orange", "value": 86400 },
{ "color": "red", "value": 604800 }
]
}
},
"overrides": []
},
"options": {
"reduceOptions": { "calcs": ["lastNotNull"] },
"textMode": "auto",
"colorMode": "background",
"graphMode": "area"
}
},
{
"type": "row",
"title": "Trends",
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 7 },
"id": 6,
"collapsed": false
},
{
"type": "timeseries",
"title": "Deployment Frequency Trend",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 },
"id": 7,
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [
{
"expr": "job:deployments:count24h",
"legendFormat": "Deploys/day",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"color": { "mode": "palette-classic" },
"custom": {
"drawStyle": "line",
"lineWidth": 2,
"fillOpacity": 10,
"pointSize": 5,
"showPoints": "auto"
}
},
"overrides": []
},
"options": {
"legend": { "displayMode": "list", "placement": "bottom" },
"tooltip": { "mode": "single" }
}
},
{
"type": "timeseries",
"title": "Change Failure Rate with Alert Threshold",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 },
"id": 8,
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [
{
"expr": "job:deployment_failures_per_deployments:ratio_rate24h * 100",
"legendFormat": "CFR %",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"color": { "mode": "palette-classic" },
"custom": {
"drawStyle": "line",
"lineWidth": 2,
"fillOpacity": 10,
"thresholdsStyle": { "mode": "line" }
},
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "red", "value": 15 }
]
}
},
"overrides": []
},
"options": {
"legend": { "displayMode": "list", "placement": "bottom" },
"tooltip": { "mode": "single" }
}
},
{
"type": "timeseries",
"title": "Lead Time Sub-Intervals",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 16 },
"id": 9,
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [
{
"expr": "job:deployment_lead_time_commit_seconds:mean5m",
"legendFormat": "Commit to Pipeline Trigger",
"refId": "A"
},
{
"expr": "job:deployment_lead_time_pipeline_trigger_seconds:mean5m",
"legendFormat": "Pipeline Trigger to Complete",
"refId": "B"
},
{
"expr": "job:deployment_lead_time_pipeline_complete_seconds:mean5m",
"legendFormat": "Pipeline Complete to Deploy",
"refId": "C"
}
],
"fieldConfig": {
"defaults": {
"unit": "s",
"color": { "mode": "palette-classic" },
"custom": {
"drawStyle": "bars",
"lineWidth": 1,
"fillOpacity": 50,
"stacking": { "mode": "normal", "group": "A" }
}
},
"overrides": []
},
"options": {
"legend": { "displayMode": "list", "placement": "bottom" },
"tooltip": { "mode": "multi" }
}
},
{
"type": "timeseries",
"title": "MTTR Trend",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 16 },
"id": 10,
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [
{
"expr": "job:incident_resolution_seconds:mean5m",
"legendFormat": "Mean Recovery",
"refId": "A"
},
{
"expr": "job:incident_resolution_seconds:p95",
"legendFormat": "p95 Recovery",
"refId": "B"
}
],
"fieldConfig": {
"defaults": {
"unit": "s",
"color": { "mode": "palette-classic" },
"custom": {
"drawStyle": "line",
"lineWidth": 2,
"fillOpacity": 10
}
},
"overrides": []
},
"options": {
"legend": { "displayMode": "list", "placement": "bottom" },
"tooltip": { "mode": "multi" }
}
}
],
"templating": { "list": [] },
"annotations": { "list": [] }
}What does this dashboard include?
- Four stat panels (one per DORA metric) with color-coded thresholds matching the Elite/High/Medium/Low tiers. Green means Elite, yellow means High, orange means Medium, and red means Low.
- A Deployment Frequency trend showing deploys per day over time.
- A Change Failure Rate panel with a visible threshold line at 15% (the boundary between High and Medium performance).
- A Lead Time Sub-Intervals panel showing stacked bars for each pipeline phase, so you can identify bottlenecks.
- An MTTR Trend panel tracking both mean and p95 recovery time.
- Save configs/grafana/dashboards/dora-metrics.json.
- Restart the Grafana container to pick up the new dashboard by running:
docker compose restart grafana- Open Grafana at http://localhost:3000 and navigate to Dashboards. You should see DORA Metrics listed alongside your existing dashboards, with no manual UI configuration needed.
Dashboard not appearing in Grafana?
- Confirm the JSON file is saved in configs/grafana/dashboards/ (the same directory that holds golden-signals.json and slo-error-budget.json from earlier steps).
- The provisioning provider from Project 1 loads all .json files from that directory automatically. Check the volume mount maps to /var/lib/grafana/dashboards inside the container.
- Check Grafana logs for JSON parse errors by running docker compose logs grafana.
Help me troubleshoot why my provisioned DORA dashboard is not appearing in Grafana.
Identify toil and propose automation strategies
Toil is repetitive, manual work that scales linearly with service growth and produces no lasting value. Identifying toil is a core SRE practice because it reveals where automation delivers the highest return.
- Create a new file at docs/toil-identification.md by pasting the following template:
# Toil Identification
## What is Toil?
Toil is work that is manual, repetitive, automatable, tactical (interrupt-driven),
scales linearly with service size, and has no enduring value.
Identifying and eliminating toil frees engineering time for reliability improvements.
## Identified Toil Examples
### 1. Manual Dashboard Refresh After Config Changes
**Description:** Every time a Prometheus rule file or Grafana dashboard JSON is
updated, the operator must manually restart the relevant Docker container
(e.g., `docker compose restart prometheus`) and verify the change loaded
correctly by checking the /rules or /dashboards endpoint.
**Frequency:** Multiple times per day during active development.
**Time per occurrence:** 2-5 minutes (restart + verify + troubleshoot if failed).
**Automation strategy:** Implement a file-watcher script (using `inotifywait` on
Linux or `fswatch` on macOS) that automatically triggers container restarts when
config files in `configs/` are modified. Add a health-check assertion that
verifies the rule count matches expectations after reload.
**Status:** Not yet implemented. Could use Prometheus HTTP reload endpoint
(`POST /-/reload`) instead of full container restart for rule changes.
### 2. Manual Traffic Generation for Dashboard Testing
**Description:** To verify that DORA and SLO dashboards show meaningful data,
the operator must manually run curl commands or scripts to generate traffic
against the sample app endpoints (`/api/orders`, `/api/orders/fail`).
This must be repeated every time the stack is brought up fresh.
**Frequency:** Every time the stack starts or after a `terraform destroy` and
re-apply cycle.
**Time per occurrence:** 5-10 minutes (write/find script, run it, wait for
metrics to propagate, verify panels).
**Automation strategy:** Create a dedicated `traffic-generator` Docker service
in `docker-compose.yaml` that continuously sends a configurable mix of
success/failure requests. Use environment variables to control request rate
and failure percentage. This eliminates manual testing entirely.
**Status:** Not yet implemented. Would require adding a new service definition
to docker-compose.yaml with a lightweight container running a shell loop or
a Python script.
## Summary
| Toil Example | Frequency | Time/Occurrence | Automation ROI |
|---|---|---|---|
| Manual config reload | Multiple times/day | 2-5 min | High |
| Manual traffic generation | Every stack restart | 5-10 min | High |
## Next Steps
- Prioritize the traffic-generator automation (highest time savings).
- Investigate Prometheus `/-/reload` endpoint for zero-downtime rule updates.
- Consider adding a CI step that validates rule files with `promtool check rules`
before deployment.Why document toil?
Documenting toil creates visibility. Teams that track toil can measure how much engineering time goes to repetitive work versus building new capabilities. This document becomes your backlog for automation improvements.
In a real production environment, you would track toil hours per quarter and set a target. Google SRE recommends no more than 50% of an SRE's time goes to toil.
- Save docs/toil-identification.md.
- Confirm the file exists by running:
ls docs/- You should see slo-definitions.md, error-budget-policy.md, and toil-identification.md listed.
Customize your toil examples
The template above covers two common examples from this project. Feel free to add more toil you have noticed in your own workflow. The more specific your examples, the more useful this document becomes as a real engineering artifact.
✔️ Awesome, I've got everything!
Great. Double check you have saved all three new files and that your Prometheus rules page shows the dora-metrics group.
ⓧ I'd like to double check the full code
Here is the full configs/prometheus/dora-rules.yml:
# DORA Metrics Recording Rules
# Tracks Deployment Frequency, Lead Time for Changes,
# Change Failure Rate, and Mean Time to Restore (MTTR)
groups:
- name: dora-metrics
interval: 1m
rules:
# --- Deployment Frequency ---
# Rate of deployments per hour over different windows
- record: job:deployments:rate1h
expr: sum(rate(deployments_total[1h]))
- record: job:deployments:rate24h
expr: sum(rate(deployments_total[24h]))
# Daily deployment count for trend analysis
- record: job:deployments:count24h
expr: sum(increase(deployments_total[24h]))
# --- Lead Time for Changes (sub-intervals) ---
# Time from commit to pipeline trigger
- record: job:deployment_lead_time_commit_seconds:mean5m
expr: |
sum(rate(deployment_lead_time_commit_seconds_sum[5m]))
/
sum(rate(deployment_lead_time_commit_seconds_count[5m]))
# Time from pipeline trigger to pipeline complete
- record: job:deployment_lead_time_pipeline_trigger_seconds:mean5m
expr: |
sum(rate(deployment_lead_time_pipeline_trigger_seconds_sum[5m]))
/
sum(rate(deployment_lead_time_pipeline_trigger_seconds_count[5m]))
# Time from pipeline complete to deployment confirmed
- record: job:deployment_lead_time_pipeline_complete_seconds:mean5m
expr: |
sum(rate(deployment_lead_time_pipeline_complete_seconds_sum[5m]))
/
sum(rate(deployment_lead_time_pipeline_complete_seconds_count[5m]))
# Total lead time (commit to deployment confirmed)
- record: job:deployment_lead_time_deploy_confirm_seconds:mean5m
expr: |
sum(rate(deployment_lead_time_deploy_confirm_seconds_sum[5m]))
/
sum(rate(deployment_lead_time_deploy_confirm_seconds_count[5m]))
# --- Change Failure Rate ---
# Ratio of failed deployments to total deployments
- record: job:deployment_failures_per_deployments:ratio_rate1h
expr: |
sum(rate(deployment_failures_total[1h]))
/
sum(rate(deployments_total[1h]))
- record: job:deployment_failures_per_deployments:ratio_rate24h
expr: |
sum(rate(deployment_failures_total[24h]))
/
sum(rate(deployments_total[24h]))
# --- Mean Time to Restore (MTTR) ---
# Average incident resolution time
- record: job:incident_resolution_seconds:mean5m
expr: |
sum(rate(incident_resolution_seconds_sum[5m]))
/
sum(rate(incident_resolution_seconds_count[5m]))
# 95th percentile recovery time
- record: job:incident_resolution_seconds:p95
expr: histogram_quantile(0.95, sum(rate(incident_resolution_seconds_bucket[1h])) by (le))Your prometheus.yml should include this in the rule_files section:
rule_files:
- /etc/prometheus/alert-rules.yml
- /etc/prometheus/slo-rules.yml
- /etc/prometheus/dora-rules.ymlYour docker-compose.yaml Prometheus volumes section should include:
- ./configs/prometheus/slo-rules.yml:/etc/prometheus/slo-rules.yml:ro
- ./configs/prometheus/dora-rules.yml:/etc/prometheus/dora-rules.yml:roYou now have a complete DORA metrics framework that classifies your team's delivery performance against industry benchmarks. Next up, you will build infrastructure dashboards for Node Exporter and Blackbox Exporter to round out your observability coverage.
Build Infrastructure Dashboards
You have SLO and DORA dashboards tracking service reliability and team performance. The next layer of observability covers the infrastructure itself.
In this step, you will create two dashboards that surface host-level signals from Node Exporter (CPU, memory, disk, network, load) and endpoint-level signals from Blackbox Exporter (uptime, response time, SSL expiry). Both are provisioned as JSON files so they load automatically with zero manual configuration.
In this step, get ready to:
- Create a Node Exporter dashboard with CPU, memory, disk I/O, network I/O, and load average panels.
- Create a Blackbox Exporter dashboard with uptime timeline, response time percentiles, SSL expiry, and probe success panels.
- Restart Grafana and verify both dashboards appear automatically with live data.
Create the Node Exporter dashboard
Node Exporter exposes hardware and kernel metrics from the Docker host. This dashboard turns those raw counters into actionable time-series panels covering the most important infrastructure signals.
- Create a new file at configs/grafana/dashboards/node-exporter.json and paste the following dashboard JSON:
{
"uid": "node-exporter",
"title": "Node Exporter - Infrastructure Health",
"tags": ["infrastructure", "node-exporter"],
"timezone": "browser",
"editable": true,
"graphTooltip": 1,
"time": {
"from": "now-1h",
"to": "now"
},
"refresh": "30s",
"schemaVersion": 39,
"version": 1,
"panels": [
{
"id": 1,
"type": "timeseries",
"title": "CPU Usage - Total",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "100 - (avg(rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)",
"legendFormat": "CPU Usage %",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"max": 100,
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "yellow", "value": 80 },
{ "color": "red", "value": 90 }
]
}
}
}
},
{
"id": 2,
"type": "timeseries",
"title": "CPU Usage - Per Core",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "100 - (rate(node_cpu_seconds_total{mode=\"idle\"}[5m]) * 100)",
"legendFormat": "Core {{cpu}}",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"max": 100
}
}
},
{
"id": 3,
"type": "timeseries",
"title": "Memory Usage",
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 8 },
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes",
"legendFormat": "Used",
"refId": "A"
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "node_memory_Cached_bytes",
"legendFormat": "Cached",
"refId": "B"
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "node_memory_MemAvailable_bytes",
"legendFormat": "Available",
"refId": "C"
}
],
"fieldConfig": {
"defaults": {
"unit": "bytes"
}
}
},
{
"id": 4,
"type": "timeseries",
"title": "Disk I/O",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 16 },
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "rate(node_disk_read_bytes_total[5m])",
"legendFormat": "Read {{device}}",
"refId": "A"
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "rate(node_disk_written_bytes_total[5m])",
"legendFormat": "Write {{device}}",
"refId": "B"
}
],
"fieldConfig": {
"defaults": {
"unit": "Bps"
}
}
},
{
"id": 5,
"type": "timeseries",
"title": "Network I/O",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 16 },
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "rate(node_network_receive_bytes_total[5m])",
"legendFormat": "Receive {{device}}",
"refId": "A"
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "rate(node_network_transmit_bytes_total[5m])",
"legendFormat": "Transmit {{device}}",
"refId": "B"
}
],
"fieldConfig": {
"defaults": {
"unit": "Bps"
}
}
},
{
"id": 6,
"type": "timeseries",
"title": "System Load Averages",
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 24 },
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "node_load1",
"legendFormat": "1m",
"refId": "A"
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "node_load5",
"legendFormat": "5m",
"refId": "B"
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "node_load15",
"legendFormat": "15m",
"refId": "C"
}
],
"fieldConfig": {
"defaults": {
"unit": "short"
}
}
}
]
}What does this dashboard cover?
Each panel queries a different category of Node Exporter metrics:
- CPU Usage - Total calculates the percentage of non-idle CPU time averaged across all cores using node_cpu_seconds_total.
- CPU Usage - Per Core breaks idle time down by individual CPU, so you can spot a single hot core.
- Memory Usage shows used, cached, and available bytes side by side using node_memory_MemTotal_bytes, node_memory_Cached_bytes, and node_memory_MemAvailable_bytes.
- Disk I/O and Network I/O use rate() to convert cumulative byte counters into bytes-per-second throughput.
- System Load Averages plots node_load1, node_load5, and node_load15 to show how system demand trends over 1, 5, and 15-minute windows.
- Save configs/grafana/dashboards/node-exporter.json.
- You won't see the dashboard in Grafana yet. The next substep creates the Blackbox Exporter dashboard, and then you will restart Grafana to load both.
JSON syntax error when saving?
- Make sure every opening brace, bracket, and quote has a matching close. A missing comma after a panel object is the most common issue.
- Check that all \" sequences inside expr strings are properly escaped. Your editor should highlight unbalanced quotes.
Help me fix a JSON syntax error in my Grafana dashboard file
✔️ Awesome, I've got everything!
Great. Make sure you have saved the file before moving on.
ⓧ I'd like to double check the full code
Here is the complete node-exporter.json for comparison:
{
"uid": "node-exporter",
"title": "Node Exporter - Infrastructure Health",
"tags": ["infrastructure", "node-exporter"],
"timezone": "browser",
"editable": true,
"graphTooltip": 1,
"time": {
"from": "now-1h",
"to": "now"
},
"refresh": "30s",
"schemaVersion": 39,
"version": 1,
"panels": [
{
"id": 1,
"type": "timeseries",
"title": "CPU Usage - Total",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "100 - (avg(rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)",
"legendFormat": "CPU Usage %",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"max": 100,
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "yellow", "value": 80 },
{ "color": "red", "value": 90 }
]
}
}
}
},
{
"id": 2,
"type": "timeseries",
"title": "CPU Usage - Per Core",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "100 - (rate(node_cpu_seconds_total{mode=\"idle\"}[5m]) * 100)",
"legendFormat": "Core {{cpu}}",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"max": 100
}
}
},
{
"id": 3,
"type": "timeseries",
"title": "Memory Usage",
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 8 },
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes",
"legendFormat": "Used",
"refId": "A"
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "node_memory_Cached_bytes",
"legendFormat": "Cached",
"refId": "B"
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "node_memory_MemAvailable_bytes",
"legendFormat": "Available",
"refId": "C"
}
],
"fieldConfig": {
"defaults": {
"unit": "bytes"
}
}
},
{
"id": 4,
"type": "timeseries",
"title": "Disk I/O",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 16 },
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "rate(node_disk_read_bytes_total[5m])",
"legendFormat": "Read {{device}}",
"refId": "A"
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "rate(node_disk_written_bytes_total[5m])",
"legendFormat": "Write {{device}}",
"refId": "B"
}
],
"fieldConfig": {
"defaults": {
"unit": "Bps"
}
}
},
{
"id": 5,
"type": "timeseries",
"title": "Network I/O",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 16 },
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "rate(node_network_receive_bytes_total[5m])",
"legendFormat": "Receive {{device}}",
"refId": "A"
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "rate(node_network_transmit_bytes_total[5m])",
"legendFormat": "Transmit {{device}}",
"refId": "B"
}
],
"fieldConfig": {
"defaults": {
"unit": "Bps"
}
}
},
{
"id": 6,
"type": "timeseries",
"title": "System Load Averages",
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 24 },
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "node_load1",
"legendFormat": "1m",
"refId": "A"
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "node_load5",
"legendFormat": "5m",
"refId": "B"
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "node_load15",
"legendFormat": "15m",
"refId": "C"
}
],
"fieldConfig": {
"defaults": {
"unit": "short"
}
}
}
]
}Create the Blackbox Exporter dashboard
Blackbox Exporter probes endpoints from the outside, the same way a real user would reach your service. This dashboard shows whether endpoints are up, how fast they respond, and when SSL certificates are about to expire.
- Create a new file at configs/grafana/dashboards/blackbox-exporter.json and paste the following dashboard JSON:
{
"uid": "blackbox-exporter",
"title": "Blackbox Exporter - Endpoint Health",
"tags": ["infrastructure", "blackbox-exporter"],
"timezone": "browser",
"editable": true,
"graphTooltip": 1,
"time": {
"from": "now-1h",
"to": "now"
},
"refresh": "30s",
"schemaVersion": 39,
"version": 1,
"panels": [
{
"id": 1,
"type": "stat",
"title": "Probe Success Rate",
"gridPos": { "h": 6, "w": 12, "x": 0, "y": 0 },
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "avg_over_time(probe_success{job=\"blackbox\"}[1h]) * 100",
"legendFormat": "{{instance}}",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "red", "value": null },
{ "color": "yellow", "value": 95 },
{ "color": "green", "value": 99 }
]
}
}
}
},
{
"id": 2,
"type": "stat",
"title": "SSL Certificate Expiry",
"gridPos": { "h": 6, "w": 12, "x": 12, "y": 0 },
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "(probe_ssl_earliest_cert_expiry{job=\"blackbox\"} - time()) / 86400",
"legendFormat": "{{instance}}",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "d",
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "red", "value": null },
{ "color": "yellow", "value": 14 },
{ "color": "green", "value": 30 }
]
}
}
}
},
{
"id": 3,
"type": "state-timeline",
"title": "Uptime / Downtime Timeline",
"gridPos": { "h": 6, "w": 24, "x": 0, "y": 6 },
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "probe_success{job=\"blackbox\"}",
"legendFormat": "{{instance}}",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"color": {
"mode": "fixed"
},
"mappings": [
{
"type": "value",
"options": {
"0": { "text": "DOWN", "color": "red" },
"1": { "text": "UP", "color": "green" }
}
}
]
}
}
},
{
"id": 4,
"type": "timeseries",
"title": "HTTP Response Time",
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 12 },
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "quantile_over_time(0.5, probe_duration_seconds{job=\"blackbox\"}[5m])",
"legendFormat": "p50 {{instance}}",
"refId": "A"
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "quantile_over_time(0.9, probe_duration_seconds{job=\"blackbox\"}[5m])",
"legendFormat": "p90 {{instance}}",
"refId": "B"
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "quantile_over_time(0.99, probe_duration_seconds{job=\"blackbox\"}[5m])",
"legendFormat": "p99 {{instance}}",
"refId": "C"
}
],
"fieldConfig": {
"defaults": {
"unit": "s"
}
}
}
]
}What does this dashboard cover?
- Probe Success Rate averages the binary probe_success metric over the last hour and displays it as a percentage. Below 99% turns yellow. Below 95% turns red.
- SSL Certificate Expiry subtracts the current Unix timestamp from probe_ssl_earliest_cert_expiry and converts seconds to days. Red when fewer than 14 days remain.
- Uptime / Downtime Timeline uses a state-timeline panel type with value mappings that color 1 as green (UP) and 0 as red (DOWN). This gives you a visual bar of uptime history.
- HTTP Response Time computes p50, p90, and p99 percentiles of probe_duration_seconds over a rolling 5-minute window using quantile_over_time().
- Save configs/grafana/dashboards/blackbox-exporter.json.
- Both dashboard files are now ready. The next substep restarts Grafana to provision them.
SSL panel showing no data?
If your Blackbox Exporter targets only use HTTP (not HTTPS), the probe_ssl_earliest_cert_expiry metric will not exist. The panel will remain empty. This is expected for HTTP-only targets. Once you add an HTTPS target to your Blackbox Exporter scrape configuration, the SSL panel will populate.
Help me understand why my SSL certificate panel shows no data
✔️ Awesome, I've got everything!
Great. Make sure you have saved the file before moving on.
ⓧ I'd like to double check the full code
Here is the complete blackbox-exporter.json for comparison:
{
"uid": "blackbox-exporter",
"title": "Blackbox Exporter - Endpoint Health",
"tags": ["infrastructure", "blackbox-exporter"],
"timezone": "browser",
"editable": true,
"graphTooltip": 1,
"time": {
"from": "now-1h",
"to": "now"
},
"refresh": "30s",
"schemaVersion": 39,
"version": 1,
"panels": [
{
"id": 1,
"type": "stat",
"title": "Probe Success Rate",
"gridPos": { "h": 6, "w": 12, "x": 0, "y": 0 },
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "avg_over_time(probe_success{job=\"blackbox\"}[1h]) * 100",
"legendFormat": "{{instance}}",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "red", "value": null },
{ "color": "yellow", "value": 95 },
{ "color": "green", "value": 99 }
]
}
}
}
},
{
"id": 2,
"type": "stat",
"title": "SSL Certificate Expiry",
"gridPos": { "h": 6, "w": 12, "x": 12, "y": 0 },
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "(probe_ssl_earliest_cert_expiry{job=\"blackbox\"} - time()) / 86400",
"legendFormat": "{{instance}}",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "d",
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "red", "value": null },
{ "color": "yellow", "value": 14 },
{ "color": "green", "value": 30 }
]
}
}
}
},
{
"id": 3,
"type": "state-timeline",
"title": "Uptime / Downtime Timeline",
"gridPos": { "h": 6, "w": 24, "x": 0, "y": 6 },
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "probe_success{job=\"blackbox\"}",
"legendFormat": "{{instance}}",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"color": {
"mode": "fixed"
},
"mappings": [
{
"type": "value",
"options": {
"0": { "text": "DOWN", "color": "red" },
"1": { "text": "UP", "color": "green" }
}
}
]
}
}
},
{
"id": 4,
"type": "timeseries",
"title": "HTTP Response Time",
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 12 },
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "quantile_over_time(0.5, probe_duration_seconds{job=\"blackbox\"}[5m])",
"legendFormat": "p50 {{instance}}",
"refId": "A"
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "quantile_over_time(0.9, probe_duration_seconds{job=\"blackbox\"}[5m])",
"legendFormat": "p90 {{instance}}",
"refId": "B"
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "quantile_over_time(0.99, probe_duration_seconds{job=\"blackbox\"}[5m])",
"legendFormat": "p99 {{instance}}",
"refId": "C"
}
],
"fieldConfig": {
"defaults": {
"unit": "s"
}
}
}
]
}Restart Grafana and verify both dashboards appear
Because dashboards are provisioned from files, Grafana needs a restart to pick up new JSON files placed in the dashboards directory.
- Restart the Grafana container by running this command from your project root:
docker compose restart grafana- Wait about 10 seconds for Grafana to finish starting up.
- Open Grafana at http://localhost:3000 in your browser.
- Click the hamburger menu on the left, then click Dashboards.
- Confirm you see both Node Exporter - Infrastructure Health and Blackbox Exporter - Endpoint Health in the list.
- Open the Node Exporter dashboard. You should see live time-series data populating the CPU, memory, disk, network, and load average panels.
- Open the Blackbox Exporter dashboard. The Probe Success Rate stat should show a percentage value, and the Uptime / Downtime Timeline should display green UP bars.
Why no manual UI configuration?
The dashboard provisioning system you set up in Project 1 (the dashboards.yaml provider file pointing to /var/lib/grafana/dashboards) automatically scans for JSON files on startup. Any new file placed in the mounted configs/grafana/dashboards/ directory appears without manual import.
This is dashboards-as-code. Every dashboard definition lives in version control. If you ever need to rebuild your stack from scratch, the dashboards reprovision automatically.
Dashboards not appearing after restart?
- Check that the JSON filenames end in .json and are saved inside configs/grafana/dashboards/ (the same directory as your existing golden-signals.json and slo-error-budget.json).
- Run docker compose logs grafana and look for provisioning errors. A common cause is invalid JSON in one of the dashboard files.
- Verify the volume mount in docker-compose.yaml maps ./configs/grafana/dashboards to /var/lib/grafana/dashboards:ro. If this mount is missing, Grafana cannot see the files.
Help me troubleshoot why my provisioned dashboards are not showing up in Grafana
Your infrastructure monitoring layer is complete. Both dashboards load automatically from code, with zero manual clicks in the Grafana UI. Next up, you will build the Unified Observability dashboard that ties metrics, logs, and traces together for full drill-down correlation.
Build the Unified Observability Dashboard
You have four dashboards provisioned as code and a full set of recording rules tracking SLOs and DORA metrics. The missing piece is the ability to jump from a metric spike directly into the logs that caused it, and from those logs into the exact distributed trace showing where the failure happened.
In this step, you will verify that your Loki-Tempo correlation from Project 1 is working, then build a Unified Observability dashboard that ties metrics, logs, and traces into one drill-down workflow.
In this step, get ready to:
- Verify bidirectional Loki-Tempo correlation (derived fields and tracesToLogsV2).
- Build the Unified Observability dashboard with metric-to-log and log-to-trace panels.
- Test the full drill-down acceptance criterion end-to-end.
Verify Loki-Tempo correlation is working
Bidirectional correlation means two things must be configured: Loki needs a derived field that extracts trace IDs from log lines and links them to Tempo, and Tempo needs a tracesToLogsV2 block pointing back to Loki.
- Open your datasource provisioning file at configs/grafana/provisioning/datasources/datasources.yml in VS Code.
- Confirm the Loki datasource section contains a derivedFields entry matching this pattern:
# Inside the Loki datasource's jsonData section
derivedFields:
- datasourceUid: tempo
matcherRegex: "trace_id=(\\w+)"
name: TraceID
url: '$${__value.raw}'
urlDisplayLabel: 'View Trace'- Confirm the Tempo datasource section contains a tracesToLogsV2 block pointing to Loki:
# Inside the Tempo datasource's jsonData section
tracesToLogsV2:
datasourceUid: loki
filterByTraceID: true
spanStartTimeShift: '-5m'
spanEndTimeShift: '5m'Why do both directions matter?
The derived field in Loki lets you click a trace ID in a log line and jump to the trace in Tempo. The tracesToLogsV2 in Tempo lets you click a span and see the correlated log lines in Loki.
Without both configured, you can only navigate one direction. The Unified Observability dashboard relies on both paths being functional.
- If either section is missing or misconfigured, add or fix it now, then restart the Grafana container by running:
docker restart grafana- Verify the correlation by opening Grafana at http://localhost:3000 and navigating to Explore.
- Select the Loki datasource and run a query like {job="sample-app"}.
- Expand a log line that contains a trace_id field.
- You should see a View Trace link in the log details. Click it to confirm it opens the trace in Tempo.
No View Trace link appearing?
Check that the datasourceUid in Loki's derived field matches the uid assigned to your Tempo datasource in the same datasources.yml file.
Also confirm your sample app logs include trace_id= in the log body (not just as structured metadata). The matcherRegex must match the exact format your app emits.
Help me troubleshoot why my Loki derived field link to Tempo is not appearing in log details.
Create the Unified Observability dashboard
This dashboard connects all three observability signals into one view. Metrics panels include data links that open Loki with a scoped query for the same time window. The logs panel relies on the derived field to make trace IDs clickable, linking directly to Tempo.
- Create the dashboard file by running:
touch configs/grafana/dashboards/unified-observability.json- Confirm the file exists by running:
ls configs/grafana/dashboards/You should see unified-observability.json alongside your four existing dashboard files.
- Paste the following dashboard JSON into configs/grafana/dashboards/unified-observability.json:
{
"id": null,
"uid": "unified-observability",
"title": "Unified Observability",
"tags": ["observability", "correlation", "sre"],
"timezone": "browser",
"editable": true,
"graphTooltip": 2,
"time": { "from": "now-1h", "to": "now" },
"refresh": "10s",
"schemaVersion": 39,
"version": 1,
"panels": [
{
"id": 1,
"type": "row",
"title": "Metrics Overview",
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 },
"collapsed": false
},
{
"id": 2,
"type": "timeseries",
"title": "HTTP Error Rate (5xx)",
"description": "Rate of 5xx responses from the sample app. Click a data point to view correlated logs.",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 1 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [
{
"expr": "sum(rate(http_requests_total{job=\"sample-app\", status=~\"5..\"}[5m])) / sum(rate(http_requests_total{job=\"sample-app\"}[5m]))",
"legendFormat": "Error Rate",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "yellow", "value": 0.01 },
{ "color": "red", "value": 0.05 }
]
},
"links": [
{
"title": "View correlated logs in Loki",
"url": "/explore?orgId=1&left={\"datasource\":\"loki\",\"queries\":[{\"expr\":\"{job=\\\"sample-app\\\"} |= \\\"status=5\\\"\",\"refId\":\"A\"}],\"range\":{\"from\":\"${__from}\",\"to\":\"${__to}\"}}",
"targetBlank": true
}
]
},
"overrides": []
},
"options": {
"tooltip": { "mode": "single" },
"legend": { "displayMode": "list", "placement": "bottom" }
}
},
{
"id": 3,
"type": "timeseries",
"title": "Request Latency (p95)",
"description": "95th percentile latency. Click to view correlated logs for slow requests.",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 1 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [
{
"expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job=\"sample-app\"}[5m])) by (le))",
"legendFormat": "p95 Latency",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "s",
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "yellow", "value": 0.3 },
{ "color": "red", "value": 0.5 }
]
},
"links": [
{
"title": "View correlated logs in Loki",
"url": "/explore?orgId=1&left={\"datasource\":\"loki\",\"queries\":[{\"expr\":\"{job=\\\"sample-app\\\"} | json | duration > 500ms\",\"refId\":\"A\"}],\"range\":{\"from\":\"${__from}\",\"to\":\"${__to}\"}}",
"targetBlank": true
}
]
},
"overrides": []
},
"options": {
"tooltip": { "mode": "single" },
"legend": { "displayMode": "list", "placement": "bottom" }
}
},
{
"id": 4,
"type": "row",
"title": "Correlated Logs",
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 9 },
"collapsed": false
},
{
"id": 5,
"type": "logs",
"title": "Application Logs (trace IDs are clickable)",
"description": "Logs from the sample app. Expand a log line and click the trace ID link to open the full trace in Tempo.",
"gridPos": { "h": 10, "w": 24, "x": 0, "y": 10 },
"datasource": { "type": "loki", "uid": "loki" },
"targets": [
{
"expr": "{job=\"sample-app\"} | json",
"refId": "A"
}
],
"options": {
"showTime": true,
"showLabels": false,
"showCommonLabels": false,
"wrapLogMessage": true,
"prettifyLogMessage": false,
"enableLogDetails": true,
"sortOrder": "Descending",
"dedupStrategy": "none"
}
},
{
"id": 6,
"type": "row",
"title": "Error Logs",
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 20 },
"collapsed": false
},
{
"id": 7,
"type": "logs",
"title": "Error Logs Only (5xx and exceptions)",
"description": "Filtered to error-level logs. Trace IDs link directly to Tempo for root cause analysis.",
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 21 },
"datasource": { "type": "loki", "uid": "loki" },
"targets": [
{
"expr": "{job=\"sample-app\"} | json | level=\"ERROR\" or status >= 500",
"refId": "A"
}
],
"options": {
"showTime": true,
"showLabels": true,
"showCommonLabels": false,
"wrapLogMessage": true,
"prettifyLogMessage": false,
"enableLogDetails": true,
"sortOrder": "Descending",
"dedupStrategy": "none"
}
}
]
}What does this dashboard do?
The dashboard has three sections working together to enable full observability drill-down:
- The HTTP Error Rate and Request Latency panels query Prometheus. Each has a data link using ${__from} and ${__to} to open Loki Explore with the same time window automatically scoped.
- The Application Logs panel queries Loki with a JSON parser. When you expand a log line, the derivedFields config makes trace IDs clickable links that open directly in Tempo.
- The Error Logs Only panel filters to error-level entries so you can quickly find the failing requests and jump to their traces.
- Save configs/grafana/dashboards/unified-observability.json.
- Restart the Grafana container to pick up the new dashboard by running:
docker restart grafana- Open Grafana at http://localhost:3000 and navigate to Dashboards.
- You should see Unified Observability listed alongside your other four dashboards, all loaded automatically without any manual import.
Dashboard not appearing in Grafana?
Confirm the JSON file is valid by running cat configs/grafana/dashboards/unified-observability.json | python3 -m json.tool. A syntax error will prevent Grafana from loading it.
Also check that the dashboard provisioning provider at configs/grafana/provisioning/dashboards/dashboards.yaml points to the correct path (/var/lib/grafana/dashboards) and that your docker-compose.yaml mounts ./configs/grafana/dashboards to that path.
Help me figure out why my provisioned Grafana dashboard JSON file is not loading.
✔️ Awesome, I've got everything!
Great. Double check you have saved unified-observability.json and that the dashboard appears in Grafana's dashboard list.
ⓧ I'd like to double check the full code
Here is the complete configs/grafana/dashboards/unified-observability.json file:
{
"id": null,
"uid": "unified-observability",
"title": "Unified Observability",
"tags": ["observability", "correlation", "sre"],
"timezone": "browser",
"editable": true,
"graphTooltip": 2,
"time": { "from": "now-1h", "to": "now" },
"refresh": "10s",
"schemaVersion": 39,
"version": 1,
"panels": [
{
"id": 1,
"type": "row",
"title": "Metrics Overview",
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 },
"collapsed": false
},
{
"id": 2,
"type": "timeseries",
"title": "HTTP Error Rate (5xx)",
"description": "Rate of 5xx responses from the sample app. Click a data point to view correlated logs.",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 1 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [
{
"expr": "sum(rate(http_requests_total{job=\"sample-app\", status=~\"5..\"}[5m])) / sum(rate(http_requests_total{job=\"sample-app\"}[5m]))",
"legendFormat": "Error Rate",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "yellow", "value": 0.01 },
{ "color": "red", "value": 0.05 }
]
},
"links": [
{
"title": "View correlated logs in Loki",
"url": "/explore?orgId=1&left={\"datasource\":\"loki\",\"queries\":[{\"expr\":\"{job=\\\"sample-app\\\"} |= \\\"status=5\\\"\",\"refId\":\"A\"}],\"range\":{\"from\":\"${__from}\",\"to\":\"${__to}\"}}",
"targetBlank": true
}
]
},
"overrides": []
},
"options": {
"tooltip": { "mode": "single" },
"legend": { "displayMode": "list", "placement": "bottom" }
}
},
{
"id": 3,
"type": "timeseries",
"title": "Request Latency (p95)",
"description": "95th percentile latency. Click to view correlated logs for slow requests.",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 1 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [
{
"expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job=\"sample-app\"}[5m])) by (le))",
"legendFormat": "p95 Latency",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "s",
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "yellow", "value": 0.3 },
{ "color": "red", "value": 0.5 }
]
},
"links": [
{
"title": "View correlated logs in Loki",
"url": "/explore?orgId=1&left={\"datasource\":\"loki\",\"queries\":[{\"expr\":\"{job=\\\"sample-app\\\"} | json | duration > 500ms\",\"refId\":\"A\"}],\"range\":{\"from\":\"${__from}\",\"to\":\"${__to}\"}}",
"targetBlank": true
}
]
},
"overrides": []
},
"options": {
"tooltip": { "mode": "single" },
"legend": { "displayMode": "list", "placement": "bottom" }
}
},
{
"id": 4,
"type": "row",
"title": "Correlated Logs",
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 9 },
"collapsed": false
},
{
"id": 5,
"type": "logs",
"title": "Application Logs (trace IDs are clickable)",
"description": "Logs from the sample app. Expand a log line and click the trace ID link to open the full trace in Tempo.",
"gridPos": { "h": 10, "w": 24, "x": 0, "y": 10 },
"datasource": { "type": "loki", "uid": "loki" },
"targets": [
{
"expr": "{job=\"sample-app\"} | json",
"refId": "A"
}
],
"options": {
"showTime": true,
"showLabels": false,
"showCommonLabels": false,
"wrapLogMessage": true,
"prettifyLogMessage": false,
"enableLogDetails": true,
"sortOrder": "Descending",
"dedupStrategy": "none"
}
},
{
"id": 6,
"type": "row",
"title": "Error Logs",
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 20 },
"collapsed": false
},
{
"id": 7,
"type": "logs",
"title": "Error Logs Only (5xx and exceptions)",
"description": "Filtered to error-level logs. Trace IDs link directly to Tempo for root cause analysis.",
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 21 },
"datasource": { "type": "loki", "uid": "loki" },
"targets": [
{
"expr": "{job=\"sample-app\"} | json | level=\"ERROR\" or status >= 500",
"refId": "A"
}
],
"options": {
"showTime": true,
"showLabels": true,
"showCommonLabels": false,
"wrapLogMessage": true,
"prettifyLogMessage": false,
"enableLogDetails": true,
"sortOrder": "Descending",
"dedupStrategy": "none"
}
}
]
}Verify the full drill-down acceptance criterion
The real test of unified observability is navigating from a metric anomaly to its root cause in a single workflow. You will generate error traffic, spot the spike, drill into logs, and trace the failure.
- Generate a burst of error traffic against your sample app by running:
# Send 20 requests to the /api/orders/fail endpoint to create error spikes
for i in $(seq 1 20); do curl -s http://localhost:5000/api/orders/fail > /dev/null; done- Wait about 30 seconds for Prometheus to scrape the new data points.
- Open the Unified Observability dashboard in Grafana.
- Observe the spike in the HTTP Error Rate (5xx) panel.
- Click on the spike data point. Select the View correlated logs in Loki link that appears.
- Grafana Explore opens with the Loki datasource and a time-scoped query showing the error logs from that exact window.
- Expand one of the error log lines. In the log details, you should see a TraceID field with a View Trace link.
- Click View Trace to open the full distributed trace in Tempo.
- In Tempo, inspect the trace spans to identify the exact service and endpoint that produced the 5xx error.
What did you just prove?
You completed the observability loop: metric spike (Prometheus) to correlated logs (Loki) to the exact failing trace (Tempo). This is the acceptance criterion for a production-ready unified observability platform.
In a real incident, this workflow reduces Mean Time to Identify (MTTI) from minutes of manual searching to seconds of clicking.
Data link opens Loki but shows no logs?
The time range in the link uses ${__from} and ${__to} from the dashboard. If you zoomed into a narrow time range on the dashboard before clicking, broaden it to Last 15 minutes and try again.
Also confirm your sample app is logging to the {job="sample-app"} label stream. Run {job="sample-app"} in Loki Explore directly to confirm logs are flowing.
Help me debug why my Grafana data link opens Loki Explore but shows no log results.
You now have five dashboards provisioned entirely as code, with the Unified Observability dashboard completing the metric-to-log-to-trace correlation loop. Next, you will configure the complete alerting system to make sure these signals trigger structured notifications before anyone even opens a dashboard.
Configure the Complete Alerting System
Your dashboards visualize what's happening across your infrastructure, but dashboards alone require someone to be watching them 24/7. A production-grade alerting system automatically detects problems and notifies your team before users are impacted.
In this step, you'll build a complete alerting pipeline. You'll define infrastructure alerts with anti-flapping protections, add multi-window SLO burn rate alerts based on the Google SRE methodology, and overhaul Alertmanager with intelligent routing, inhibition rules, and structured Slack notifications.
In this step, get ready to:
- Replace the basic alert rules with comprehensive infrastructure alerts that include recovery detection.
- Add multi-window SLO burn rate alerts and CI/CD performance alerts with runbook annotations.
- Overhaul Alertmanager with route trees, inhibition rules, and structured Go notification templates.
Replace the basic alert rules with production-grade infrastructure alerts
The existing alert-rules.yml from Project 1 contains only a single InstanceDown alert. Production systems need alerts at multiple severity levels with anti-flapping protections to prevent noisy notifications during transient issues.
- Replace the entire contents of configs/prometheus/alert-rules.yml with this comprehensive infrastructure alert configuration:
# Comprehensive alerting rules for infrastructure, SLOs, and CI/CD
groups:
- name: infrastructure_alerts
rules:
# CPU Warning - sustained high CPU usage
- alert: HighCpuUsageWarning
expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
for: 5m
keep_firing_for: 5m
labels:
severity: warning
service: infrastructure
annotations:
summary: "High CPU usage on {{ $labels.instance }}"
description: "CPU usage is above 80% for more than 5 minutes. Current value: {{ $value | printf \"%.1f\" }}%"
dashboard: "http://localhost:3000/d/node-exporter/node-exporter"
runbook: "docs/runbooks/high-cpu.md"
# CPU Critical - dangerously high CPU usage
- alert: HighCpuUsageCritical
expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90
for: 10m
keep_firing_for: 10m
labels:
severity: critical
service: infrastructure
annotations:
summary: "Critical CPU usage on {{ $labels.instance }}"
description: "CPU usage is above 90% for more than 10 minutes. Current value: {{ $value | printf \"%.1f\" }}%"
dashboard: "http://localhost:3000/d/node-exporter/node-exporter"
runbook: "docs/runbooks/high-cpu.md"
# Memory Warning
- alert: HighMemoryUsageWarning
expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 80
for: 5m
keep_firing_for: 5m
labels:
severity: warning
service: infrastructure
annotations:
summary: "High memory usage on {{ $labels.instance }}"
description: "Memory usage is above 80%. Current value: {{ $value | printf \"%.1f\" }}%"
dashboard: "http://localhost:3000/d/node-exporter/node-exporter"
runbook: "docs/runbooks/high-memory.md"
# Memory Critical
- alert: HighMemoryUsageCritical
expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90
for: 5m
keep_firing_for: 10m
labels:
severity: critical
service: infrastructure
annotations:
summary: "Critical memory usage on {{ $labels.instance }}"
description: "Memory usage is above 90%. Current value: {{ $value | printf \"%.1f\" }}%"
dashboard: "http://localhost:3000/d/node-exporter/node-exporter"
runbook: "docs/runbooks/high-memory.md"
# Disk Warning
- alert: DiskSpaceWarning
expr: (1 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"})) * 100 > 75
for: 10m
keep_firing_for: 5m
labels:
severity: warning
service: infrastructure
annotations:
summary: "Disk space running low on {{ $labels.instance }}"
description: "Disk usage is above 75%. Current value: {{ $value | printf \"%.1f\" }}%"
dashboard: "http://localhost:3000/d/node-exporter/node-exporter"
runbook: "docs/runbooks/disk-space.md"
# Disk Critical
- alert: DiskSpaceCritical
expr: (1 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"})) * 100 > 90
for: 5m
keep_firing_for: 10m
labels:
severity: critical
service: infrastructure
annotations:
summary: "Critical disk space on {{ $labels.instance }}"
description: "Disk usage is above 90%. Current value: {{ $value | printf \"%.1f\" }}%"
dashboard: "http://localhost:3000/d/node-exporter/node-exporter"
runbook: "docs/runbooks/disk-space.md"
# Server Downtime - Blackbox probe failure with recovery
- alert: ServerDown
expr: probe_success{job="blackbox"} == 0
for: 2m
keep_firing_for: 5m
labels:
severity: critical
service: availability
annotations:
summary: "Server {{ $labels.instance }} is unreachable"
description: "Blackbox probe has been failing for more than 2 consecutive minutes."
dashboard: "http://localhost:3000/d/blackbox-exporter/blackbox-exporter"
runbook: "docs/runbooks/server-down.md"What does this configuration do?
Each alert uses two key anti-flapping mechanisms from Prometheus:
- for: causes Prometheus to wait the specified duration before counting an alert as firing. This prevents brief spikes from triggering notifications.
- keep_firing_for: keeps the alert in the firing state after the condition clears, preventing rapid firing/resolved flip-flopping.
- The severity and service labels enable Alertmanager to route alerts to different channels with different urgency levels.
- Annotations include a dashboard link and runbook link so responders can immediately jump to context when paged.
Why warning AND critical for the same metric?
Two-tier severity gives your team time to react before things become urgent. A warning at 80% CPU means "investigate soon" while a critical at 90% means "act immediately."
The ServerDown alert is intentionally only critical because a probe failure means real user impact right now.
- Save configs/prometheus/alert-rules.yml.
You won't restart Prometheus yet. You'll add the SLO and CI/CD alerts next, then restart once with the complete rule set.
Seeing YAML syntax errors later?
YAML is indentation-sensitive. Each alert: block must be indented exactly 6 spaces (under rules:). The labels: and annotations: blocks must be 8 spaces. Mixing tabs and spaces will break the config silently.
help me debug YAML indentation errors in my Prometheus alert rules
Add multi-window SLO burn rate alerts and CI/CD alerts
Infrastructure alerts catch host-level problems, but they don't tell you whether your service-level objectives are at risk. Multi-window burn rate alerts detect when your error budget is being consumed faster than sustainable.
- Add the following two rule groups below the infrastructure_alerts group in configs/prometheus/alert-rules.yml:
- name: slo_burn_rate_alerts
rules:
# Fast Burn - Critical
# 14.4x burn rate = 2% of 30-day error budget consumed in 1 hour
- alert: SloAvailabilityFastBurn
expr: |
job:slo_errors_per_request:ratio_rate5m{job="blackbox"} > (14.4 * 0.005)
and
job:slo_errors_per_request:ratio_rate1h{job="blackbox"} > (14.4 * 0.005)
for: 2m
keep_firing_for: 5m
labels:
severity: critical
service: slo
slo: availability
annotations:
summary: "Availability SLO fast burn rate alert"
description: "Error budget is being consumed at 14.4x the sustainable rate. At this pace, the entire 30-day budget will be exhausted in under 2 hours."
dashboard: "http://localhost:3000/d/slo-error-budget/slo-error-budget"
runbook: "docs/runbooks/slo-fast-burn.md"
# Slow Burn - Warning
# 5x burn rate = 5% of 30-day error budget consumed in 6 hours
- alert: SloAvailabilitySlowBurn
expr: |
job:slo_errors_per_request:ratio_rate30m{job="blackbox"} > (5 * 0.005)
and
job:slo_errors_per_request:ratio_rate6h{job="blackbox"} > (5 * 0.005)
for: 5m
keep_firing_for: 15m
labels:
severity: warning
service: slo
slo: availability
annotations:
summary: "Availability SLO slow burn rate alert"
description: "Error budget is being consumed at 5x the sustainable rate. Needs attention within the next few hours."
dashboard: "http://localhost:3000/d/slo-error-budget/slo-error-budget"
runbook: "docs/runbooks/slo-slow-burn.md"
- name: cicd_alerts
rules:
# Change Failure Rate exceeds threshold
- alert: HighChangeFailureRate
expr: job:change_failure_rate:ratio > 0.15
for: 5m
keep_firing_for: 10m
labels:
severity: warning
service: cicd
annotations:
summary: "Change Failure Rate exceeds threshold"
description: "CFR is above 15% (High benchmark). Current value: {{ $value | printf \"%.1f\" }}%"
dashboard: "http://localhost:3000/d/dora-metrics/dora-metrics"
runbook: "docs/runbooks/high-cfr.md"
# MTTR exceeds maximum
- alert: HighMttr
expr: job:mttr_hours:avg > 24
for: 5m
keep_firing_for: 10m
labels:
severity: warning
service: cicd
annotations:
summary: "MTTR exceeds maximum threshold"
description: "Mean Time to Restore is above 24 hours. Current value: {{ $value | printf \"%.1f\" }} hours"
dashboard: "http://localhost:3000/d/dora-metrics/dora-metrics"
runbook: "docs/runbooks/high-mttr.md"How do multi-window burn rate alerts work?
The multi-window approach uses two time windows simultaneously to balance speed and accuracy. The short window (5m or 30m) catches recent spikes, while the long window (1h or 6h) confirms the problem is sustained.
Fast Burn at 14.4x means your error budget is depleting 14.4 times faster than its sustainable rate. At that pace, 2% of a 30-day budget is consumed in just 1 hour. This demands immediate action.
Slow Burn at 5x is less urgent but still dangerous. It consumes 5% of the budget in 6 hours. Left unaddressed, it exhausts the budget within days.
Why do burn rate alerts reference recording rules?
The job:slo_errors_per_request:ratio_rate5m metrics come from the recording rules you created in slo-rules.yml back in Step 2. Recording rules pre-compute expensive queries so alerts can evaluate quickly without re-querying raw data every cycle.
- Save configs/prometheus/alert-rules.yml with both the infrastructure and SLO/CI/CD rule groups.
Your complete file now has three rule groups: infrastructure_alerts, slo_burn_rate_alerts, and cicd_alerts.
Burn rate alerts referencing metrics that don't exist?
If Prometheus shows "no data" for burn rate alerts, confirm your slo-rules.yml recording rules are loaded and evaluating. Check the Prometheus UI at http://localhost:9090/rules to verify all rule groups appear.
help me debug why my SLO burn rate alerts show no data in Prometheus
Overhaul Alertmanager with route trees, inhibition, and structured templates
Alertmanager decides what happens after Prometheus fires an alert. The basic config from Project 1 sends everything to a single Slack channel with the same urgency.
A production overhaul adds route trees that send critical alerts faster, inhibition rules that suppress noise during outages, and structured notification templates that give responders immediate context.
- Replace the entire contents of configs/alertmanager/alertmanager.yml with this configuration:
# Alertmanager configuration with route trees, inhibition, and structured templates
global:
resolve_timeout: 5m
slack_api_url: 'https://hooks.slack.com/services/YOUR/WEBHOOK/URL'
templates:
- '/etc/alertmanager/templates/slack.tmpl'
route:
receiver: slack-default
group_by: ['alertname', 'severity', 'service']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
# Critical infrastructure alerts - immediate notification
- match:
severity: critical
service: infrastructure
receiver: slack-critical
group_wait: 10s
repeat_interval: 1h
# SLO burn rate alerts - fast burn gets immediate attention
- match:
severity: critical
service: slo
receiver: slack-critical
group_wait: 10s
repeat_interval: 30m
# SLO slow burn - warning level
- match:
severity: warning
service: slo
receiver: slack-warning
group_wait: 1m
repeat_interval: 2h
# Availability alerts - server down
- match:
service: availability
receiver: slack-critical
group_wait: 10s
repeat_interval: 1h
# CI/CD alerts
- match:
service: cicd
receiver: slack-warning
group_wait: 5m
repeat_interval: 4h
# Warning-level infrastructure
- match:
severity: warning
service: infrastructure
receiver: slack-warning
group_wait: 1m
repeat_interval: 4h
# Inhibition rules - suppress noise when host is unreachable
inhibit_rules:
# If a host is completely down, suppress CPU/memory/disk alerts for that host
- source_matchers:
- alertname = "ServerDown"
target_matchers:
- service = "infrastructure"
equal: ['instance']
# If a critical alert is firing, suppress warning for the same alert
- source_matchers:
- severity = "critical"
target_matchers:
- severity = "warning"
equal: ['alertname', 'instance']
# Silencing documentation:
# Use the Alertmanager UI (http://localhost:9093/#/silences) to create temporary silences.
# Common use cases:
# - Planned maintenance windows: silence ServerDown for the maintenance target
# - Known noisy alerts during deployments: silence infrastructure warnings
# - Investigating a known issue: silence specific alertname + instance
# To create a silence via API:
# curl -X POST http://localhost:9093/api/v2/silences -d '{...}'
receivers:
- name: slack-default
slack_configs:
- channel: '#DevOps-Alerts'
send_resolved: true
title: '{{ template "slack.title" . }}'
text: '{{ template "slack.text" . }}'
- name: slack-critical
slack_configs:
- channel: '#DevOps-Alerts'
send_resolved: true
title: '{{ template "slack.title" . }}'
text: '{{ template "slack.text" . }}'
color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}'
- name: slack-warning
slack_configs:
- channel: '#DevOps-Alerts'
send_resolved: true
title: '{{ template "slack.title" . }}'
text: '{{ template "slack.text" . }}'
color: '{{ if eq .Status "firing" }}warning{{ else }}good{{ end }}'How do route trees and inhibition rules work together?
Every alert enters the top-level route and traverses child routes until it finds a match. Critical alerts get shorter group_wait (10 seconds) so they notify faster, while warnings use longer waits to batch related alerts together.
Inhibition rules prevent alert storms. If ServerDown fires for a host, all CPU/memory/disk alerts for that same host are suppressed. When the server is completely unreachable, those resource alerts are just noise.
The equal: ['instance'] field ensures inhibition only applies when both the source and target alert share the same instance label value.
- Replace the placeholder slack_api_url value with your real Slack webhook URL (the one you configured in Step 1).
- Save configs/alertmanager/alertmanager.yml.
Now create the Go notification template file that the receivers reference.
- Create the templates directory and file by running:
mkdir -p configs/alertmanager/templates- Create configs/alertmanager/templates/slack.tmpl with the following structured notification template:
{{ define "slack.title" -}}
[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .GroupLabels.SortedPairs.Values | join " " }}
{{- end }}
{{ define "slack.text" -}}
{{ range .Alerts -}}
*Alert:* {{ .Labels.alertname }}
*Severity:* {{ .Labels.severity }}
*Host:* {{ .Labels.instance }}
*Value:* {{ .Annotations.description }}
*Dashboard:* {{ .Annotations.dashboard }}
*Runbook:* {{ .Annotations.runbook }}
*Status:* {{ .Status }}
*Started:* {{ .StartsAt.Format "2006-01-02 15:04:05 UTC" }}
{{ if eq .Status "resolved" }}*Resolved:* {{ .EndsAt.Format "2006-01-02 15:04:05 UTC" }}{{ end }}
---
{{ end -}}
{{- end }}What does this template produce?
The slack.title template shows the alert status (FIRING or RESOLVED) plus a count and the group labels. The slack.text template loops through each alert and outputs a structured block with all the context a responder needs.
- Alert name and severity tell you what broke and how urgent it is.
- The host field identifies which instance is affected.
- Dashboard and runbook links let you click directly to investigation tools.
- Resolved status with timestamps helps calculate MTTR for your DORA metrics.
Now update docker-compose.yaml to mount the templates directory into the Alertmanager container.
- In docker-compose.yaml, find the alertmanager service's volumes: section.
- Add this volume mount below the existing alertmanager.yml mount:
- ./configs/alertmanager/templates:/etc/alertmanager/templates:roWhy mount the templates directory?
The templates: field in alertmanager.yml references /etc/alertmanager/templates/slack.tmpl inside the container. This volume mount maps your local configs/alertmanager/templates/ directory to that container path so Alertmanager can read the template file.
- Save docker-compose.yaml.
- Restart both the Prometheus and Alertmanager containers to load the new configs by running:
docker restart prometheus alertmanager- Verify Prometheus loaded the alert rules by opening http://localhost:9090/rules in your browser. You should see all three rule groups (infrastructure_alerts, slo_burn_rate_alerts, cicd_alerts) listed with their individual rules.
- Verify Alertmanager loaded its config by opening http://localhost:9093/#/status in your browser. You should see your route tree and the three receivers (slack-default, slack-critical, slack-warning) listed under the config section.
Containers failing to start after restart?
Check the container logs for configuration errors by running docker logs prometheus and docker logs alertmanager. Common issues include YAML indentation errors, missing template files (check your volume mount path matches exactly), or invalid PromQL in alert expressions.
If Alertmanager reports a template error, verify that configs/alertmanager/templates/slack.tmpl exists and that the file doesn't have any encoding issues from copy-pasting.
help me debug why my Prometheus or Alertmanager container won't start after configuration changes
✔️ Awesome, I've got everything!
Great. Double check you've saved all three files: alert-rules.yml, alertmanager.yml, and slack.tmpl.
ⓧ I'd like to double check the full code
Here is the complete configs/prometheus/alert-rules.yml:
# Comprehensive alerting rules for infrastructure, SLOs, and CI/CD
groups:
- name: infrastructure_alerts
rules:
# CPU Warning - sustained high CPU usage
- alert: HighCpuUsageWarning
expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
for: 5m
keep_firing_for: 5m
labels:
severity: warning
service: infrastructure
annotations:
summary: "High CPU usage on {{ $labels.instance }}"
description: "CPU usage is above 80% for more than 5 minutes. Current value: {{ $value | printf \"%.1f\" }}%"
dashboard: "http://localhost:3000/d/node-exporter/node-exporter"
runbook: "docs/runbooks/high-cpu.md"
# CPU Critical - dangerously high CPU usage
- alert: HighCpuUsageCritical
expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90
for: 10m
keep_firing_for: 10m
labels:
severity: critical
service: infrastructure
annotations:
summary: "Critical CPU usage on {{ $labels.instance }}"
description: "CPU usage is above 90% for more than 10 minutes. Current value: {{ $value | printf \"%.1f\" }}%"
dashboard: "http://localhost:3000/d/node-exporter/node-exporter"
runbook: "docs/runbooks/high-cpu.md"
# Memory Warning
- alert: HighMemoryUsageWarning
expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 80
for: 5m
keep_firing_for: 5m
labels:
severity: warning
service: infrastructure
annotations:
summary: "High memory usage on {{ $labels.instance }}"
description: "Memory usage is above 80%. Current value: {{ $value | printf \"%.1f\" }}%"
dashboard: "http://localhost:3000/d/node-exporter/node-exporter"
runbook: "docs/runbooks/high-memory.md"
# Memory Critical
- alert: HighMemoryUsageCritical
expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90
for: 5m
keep_firing_for: 10m
labels:
severity: critical
service: infrastructure
annotations:
summary: "Critical memory usage on {{ $labels.instance }}"
description: "Memory usage is above 90%. Current value: {{ $value | printf \"%.1f\" }}%"
dashboard: "http://localhost:3000/d/node-exporter/node-exporter"
runbook: "docs/runbooks/high-memory.md"
# Disk Warning
- alert: DiskSpaceWarning
expr: (1 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"})) * 100 > 75
for: 10m
keep_firing_for: 5m
labels:
severity: warning
service: infrastructure
annotations:
summary: "Disk space running low on {{ $labels.instance }}"
description: "Disk usage is above 75%. Current value: {{ $value | printf \"%.1f\" }}%"
dashboard: "http://localhost:3000/d/node-exporter/node-exporter"
runbook: "docs/runbooks/disk-space.md"
# Disk Critical
- alert: DiskSpaceCritical
expr: (1 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"})) * 100 > 90
for: 5m
keep_firing_for: 10m
labels:
severity: critical
service: infrastructure
annotations:
summary: "Critical disk space on {{ $labels.instance }}"
description: "Disk usage is above 90%. Current value: {{ $value | printf \"%.1f\" }}%"
dashboard: "http://localhost:3000/d/node-exporter/node-exporter"
runbook: "docs/runbooks/disk-space.md"
# Server Downtime - Blackbox probe failure with recovery
- alert: ServerDown
expr: probe_success{job="blackbox"} == 0
for: 2m
keep_firing_for: 5m
labels:
severity: critical
service: availability
annotations:
summary: "Server {{ $labels.instance }} is unreachable"
description: "Blackbox probe has been failing for more than 2 consecutive minutes."
dashboard: "http://localhost:3000/d/blackbox-exporter/blackbox-exporter"
runbook: "docs/runbooks/server-down.md"
- name: slo_burn_rate_alerts
rules:
# Fast Burn - Critical
# 14.4x burn rate = 2% of 30-day error budget consumed in 1 hour
- alert: SloAvailabilityFastBurn
expr: |
job:slo_errors_per_request:ratio_rate5m{job="blackbox"} > (14.4 * 0.005)
and
job:slo_errors_per_request:ratio_rate1h{job="blackbox"} > (14.4 * 0.005)
for: 2m
keep_firing_for: 5m
labels:
severity: critical
service: slo
slo: availability
annotations:
summary: "Availability SLO fast burn rate alert"
description: "Error budget is being consumed at 14.4x the sustainable rate. At this pace, the entire 30-day budget will be exhausted in under 2 hours."
dashboard: "http://localhost:3000/d/slo-error-budget/slo-error-budget"
runbook: "docs/runbooks/slo-fast-burn.md"
# Slow Burn - Warning
# 5x burn rate = 5% of 30-day error budget consumed in 6 hours
- alert: SloAvailabilitySlowBurn
expr: |
job:slo_errors_per_request:ratio_rate30m{job="blackbox"} > (5 * 0.005)
and
job:slo_errors_per_request:ratio_rate6h{job="blackbox"} > (5 * 0.005)
for: 5m
keep_firing_for: 15m
labels:
severity: warning
service: slo
slo: availability
annotations:
summary: "Availability SLO slow burn rate alert"
description: "Error budget is being consumed at 5x the sustainable rate. Needs attention within the next few hours."
dashboard: "http://localhost:3000/d/slo-error-budget/slo-error-budget"
runbook: "docs/runbooks/slo-slow-burn.md"
- name: cicd_alerts
rules:
# Change Failure Rate exceeds threshold
- alert: HighChangeFailureRate
expr: job:change_failure_rate:ratio > 0.15
for: 5m
keep_firing_for: 10m
labels:
severity: warning
service: cicd
annotations:
summary: "Change Failure Rate exceeds threshold"
description: "CFR is above 15% (High benchmark). Current value: {{ $value | printf \"%.1f\" }}%"
dashboard: "http://localhost:3000/d/dora-metrics/dora-metrics"
runbook: "docs/runbooks/high-cfr.md"
# MTTR exceeds maximum
- alert: HighMttr
expr: job:mttr_hours:avg > 24
for: 5m
keep_firing_for: 10m
labels:
severity: warning
service: cicd
annotations:
summary: "MTTR exceeds maximum threshold"
description: "Mean Time to Restore is above 24 hours. Current value: {{ $value | printf \"%.1f\" }} hours"
dashboard: "http://localhost:3000/d/dora-metrics/dora-metrics"
runbook: "docs/runbooks/high-mttr.md"Here is the complete configs/alertmanager/alertmanager.yml:
# Alertmanager configuration with route trees, inhibition, and structured templates
global:
resolve_timeout: 5m
slack_api_url: 'https://hooks.slack.com/services/YOUR/WEBHOOK/URL'
templates:
- '/etc/alertmanager/templates/slack.tmpl'
route:
receiver: slack-default
group_by: ['alertname', 'severity', 'service']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
# Critical infrastructure alerts - immediate notification
- match:
severity: critical
service: infrastructure
receiver: slack-critical
group_wait: 10s
repeat_interval: 1h
# SLO burn rate alerts - fast burn gets immediate attention
- match:
severity: critical
service: slo
receiver: slack-critical
group_wait: 10s
repeat_interval: 30m
# SLO slow burn - warning level
- match:
severity: warning
service: slo
receiver: slack-warning
group_wait: 1m
repeat_interval: 2h
# Availability alerts - server down
- match:
service: availability
receiver: slack-critical
group_wait: 10s
repeat_interval: 1h
# CI/CD alerts
- match:
service: cicd
receiver: slack-warning
group_wait: 5m
repeat_interval: 4h
# Warning-level infrastructure
- match:
severity: warning
service: infrastructure
receiver: slack-warning
group_wait: 1m
repeat_interval: 4h
# Inhibition rules - suppress noise when host is unreachable
inhibit_rules:
# If a host is completely down, suppress CPU/memory/disk alerts for that host
- source_matchers:
- alertname = "ServerDown"
target_matchers:
- service = "infrastructure"
equal: ['instance']
# If a critical alert is firing, suppress warning for the same alert
- source_matchers:
- severity = "critical"
target_matchers:
- severity = "warning"
equal: ['alertname', 'instance']
# Silencing documentation:
# Use the Alertmanager UI (http://localhost:9093/#/silences) to create temporary silences.
# Common use cases:
# - Planned maintenance windows: silence ServerDown for the maintenance target
# - Known noisy alerts during deployments: silence infrastructure warnings
# - Investigating a known issue: silence specific alertname + instance
# To create a silence via API:
# curl -X POST http://localhost:9093/api/v2/silences -d '{...}'
receivers:
- name: slack-default
slack_configs:
- channel: '#DevOps-Alerts'
send_resolved: true
title: '{{ template "slack.title" . }}'
text: '{{ template "slack.text" . }}'
- name: slack-critical
slack_configs:
- channel: '#DevOps-Alerts'
send_resolved: true
title: '{{ template "slack.title" . }}'
text: '{{ template "slack.text" . }}'
color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}'
- name: slack-warning
slack_configs:
- channel: '#DevOps-Alerts'
send_resolved: true
title: '{{ template "slack.title" . }}'
text: '{{ template "slack.text" . }}'
color: '{{ if eq .Status "firing" }}warning{{ else }}good{{ end }}'Here is the complete configs/alertmanager/templates/slack.tmpl:
{{ define "slack.title" -}}
[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .GroupLabels.SortedPairs.Values | join " " }}
{{- end }}
{{ define "slack.text" -}}
{{ range .Alerts -}}
*Alert:* {{ .Labels.alertname }}
*Severity:* {{ .Labels.severity }}
*Host:* {{ .Labels.instance }}
*Value:* {{ .Annotations.description }}
*Dashboard:* {{ .Annotations.dashboard }}
*Runbook:* {{ .Annotations.runbook }}
*Status:* {{ .Status }}
*Started:* {{ .StartsAt.Format "2006-01-02 15:04:05 UTC" }}
{{ if eq .Status "resolved" }}*Resolved:* {{ .EndsAt.Format "2006-01-02 15:04:05 UTC" }}{{ end }}
---
{{ end -}}
{{- end }}Your alerting system is now production-grade. Alerts fire at the right severity, route to the right channel with the right urgency, suppress noise during real outages, and deliver structured notifications that give responders everything they need to act.
Secret mission
Chaos Injection and Incident Response
Your SLO dashboards are humming, your alerts are configured, and your Slack channel is wired up. But how do you know any of it actually works under pressure? In this secret mission, you will build a chaos injection mechanism that deliberately breaks your service so you can watch your entire observability platform respond in real-time.
Clean Up Your Resources
Clean Up Your Resources
Decide whether to keep your resources running, pause them to come back later, or delete them entirely.
This project runs entirely locally via Docker Compose, so there are no ongoing cloud costs. The stack uses approximately 2-4GB of RAM while running.
Resources you used:
- Docker containers (Prometheus, Grafana, Loki, Tempo, Alertmanager, Node Exporter, Blackbox Exporter, OTel Collector, Flask sample app).
- Local project files in your observability-stack directory (configs, dashboards, docs, chaos scripts).
- Docker volumes storing Prometheus, Grafana, Loki, and Tempo data.
- Slack incoming webhook in your #DevOps-Alerts channel.
Keep everything running
No action needed. Choose this if you're still actively experimenting with chaos injection, tuning alert thresholds, or building additional dashboards.
- Your Docker containers continue running and serving data to all five dashboards.
- Your Slack webhook remains active and will continue to deliver alert notifications.
- RAM usage stays at approximately 2-4GB while all containers are running.
Pause - I'll come back to this later
Shut down running containers to free up memory. Your configuration files and Docker volumes persist, so you can bring everything back up later.
- Stop all containers by running this command from your project directory:
cd terraform
terraform destroy -auto-approve- Your data persists in Docker volumes. When you're ready to resume, run terraform apply -auto-approve from the same terraform/ directory to bring everything back.
- All dashboards, recording rules, alert rules, and Alertmanager configuration remain on disk in your configs/ directory.
Delete - I don't want to use this again
Remove all project resources and start fresh. This destroys containers, volumes, and local files.
- Destroy all Docker containers by running:
cd terraform
terraform destroy -auto-approve- Remove all Docker volumes (this deletes stored metrics, logs, and traces) by running:
docker volume prune -f- Delete the project directory entirely. On macOS or Linux, run:
cd ~/Desktop
rm -rf observability-stack- On Windows, run:
cd ~\Desktop
rmdir /s /q observability-stack- To remove the Slack webhook, open your Slack workspace settings, navigate to Apps, find Incoming WebHooks, and delete the webhook configuration for #DevOps-Alerts.
That's a Wrap!
That's a Wrap!
Nice work! You just transformed a basic observability stack into a production-grade SRE platform with SLO-driven alerting, DORA performance tracking, and unified drill-down correlation.
You've learned how to:
- Define SLOs with error budgets and implement multi-window burn rate alerting (fast burn 14.4x, slow burn 5x) that routes structured notifications to Slack.
- Build five Grafana dashboards provisioned as code with unified metric-to-log-to-trace drill-down correlation across Prometheus, Loki, and Tempo.
- Configure a complete Alertmanager system with routing trees, inhibition rules, Go notification templates, and an Error Budget Policy connecting reliability engineering to business decisions.
- Injected chaos to validate the entire alerting and observability pipeline end-to-end, measured MTTR, and classified your incident response against DORA benchmarks.
Ready to quiz yourself?