Error Budget Control Room
Derive burn-rate alert thresholds from first principles using SLOs.
Introduction
30 Second Summary
Open a contributor graph for a platform like Nextstrain's Nextclade: one engineer holds 88% of the commits, a second holds most of the rest. These services are not dying, they are single-maintainer-fragile, and when two people share night-time alert duty, every false alarm costs half the roster a night of sleep.
In this project, you will build a local Error Budget Control Room that derives alert thresholds from first principles and proves they fire in the correct order. You will ship the result through a private GitHub repository with every decision documented and every issue closed.
What You'll Build
You will run a single command that brings up Prometheus, Grafana, a sample HTTP service, and k6 as a load driver, then inject a 10% error rate and watch the fast-burn alert fire exactly when your hand-derived math predicts.
By the end of this project, you'll have:
- A running observability stack where one command proves the system green, with error injection that fires alerts on a mathematically predictable schedule.
- Hand-derived burn-rate thresholds verified against the Google SRE Workbook, with lab-compressed windows that fire within the session so you can observe firing order live.
- Four signature artifacts delivered through a tagged GitHub release: the validated system, a stakeholder presentation, a teach-back README a non-engineer can follow, and documentation a person arriving tomorrow can run and understand.
- Secret Mission: Write an honest after-action review scoring the AI's accuracy against your derivation, with real numbers.
Are there any prerequisites?
You need Docker Desktop installed and running, a GitHub account (Free tier), and any flat-rate AI client (Claude, ChatGPT, or local Ollama) for design drafting. No metered API is required and the project costs nothing to complete.
Before We Start
You are about to build the arithmetic that decides whether two people on a small on-call roster get to sleep. Before any containers run, lock in what you are building and why the math has to be shown rather than copied.
Set Up the Repository and Stack
The plan exists before the work. A GitHub Project board with issues opened before any code gives you a tracker that proves the sequence was honest.
The compose file is the contract. If the stack does not come up from a clean clone in one command, nothing after it is trustworthy.
In this step, get ready to:
- Create a private GitHub repository with a Project board and six issues.
- Bring up four containers (Prometheus, Grafana, sample-service, k6) with one command.
- Confirm metrics flow from the sample service into Prometheus.
Confirm your tools
This project needs a container engine (Docker or Podman), Git, and a code editor. The observability stack (Prometheus, Grafana, Sloth, k6) ships inside the starter repo's pinned compose file, so there is nothing extra to install for those.
- Check your container engine is running by executing this command:
docker infoUsing Podman instead of Docker?
Run podman info instead. Every docker compose command in this project works the same way with podman compose.
✔️ I see server info
Your container engine is running and ready. Skip ahead to the Git check below.
ⓧ Cannot connect to the Docker daemon
The engine is installed but the daemon is not running. Start it now.
- macOS or Windows: Open Docker Desktop (or Podman Desktop) from your applications and wait for the status indicator to turn green.
- Linux: Start the Docker service by running:
sudo systemctl start dockerRe-run docker info to confirm you now see server details.
ⓧ Command not found
You need to install a container engine. Choose the path that matches your operating system.
- macOS or Windows: Download and install Docker Desktop from https://docs.docker.com/desktop/ or Podman Desktop from https://podman-desktop.io/downloads. Both include the Compose plugin.
- Linux: Install Docker Engine and the Compose plugin by following the official guide at https://docs.docker.com/engine/install/. After installing, start the service with sudo systemctl start docker.
Once installed, run docker info (or podman info) again. You should see server details confirming the engine is active.
- Check Git is installed by running:
git --version✔️ I see a version number
Git is ready. Move on to the editor check below.
ⓧ Command not found
Install Git from the official site at https://git-scm.com/downloads. Follow the prompts for your operating system, accept the defaults, then open a new terminal window.
Run git --version again in the new terminal to confirm Git is available.
For your editor, any editor that can open a folder works (VS Code, Cursor, IntelliJ, Vim). You will open the cloned repo after the next substep.
What about Prometheus, Grafana, Sloth, and k6?
All four are container images pinned in the starter repo's compose.yaml. When you run docker compose up later in this step, Docker pulls and starts them automatically. No separate install required.
Create the repository and project board
You need a GitHub account, a GitHub organization to house your portfolio repos, and a private repository created from the starter template. The org is a one-time setup that keeps project repos grouped instead of scattered across your personal account.
- Sign in to GitHub at https://github.com. If you don't have an account, click Sign up and create one for free.
- In the top-right corner, click the + dropdown.
- Click New organization.
- Select the Free plan.
- Enter an organization name (e.g. yourname-portfolio) and your contact email.
- Click Create organization. Skip the optional steps that follow.
Why create a GitHub organization?
A repo in a personal account sits alongside forks, experiments, and half-finished side projects. An organization groups your portfolio work in one URL that you can share with hiring managers or teammates.
You only create the org once. Every future project repo goes inside it.
Now create the project repository from the starter template. The template includes the compose file, sample service, load scripts, and issue templates you need.
- Navigate to the starter template repository.
- Click Use this template above the file list.
- Select Create a new repository.
- Use the Owner dropdown to select your organization (not your personal account).
- Type error-budget-control-room as the repository name.
- Set visibility to Private.
- Click Create repository from template.
You now have your own private copy of the starter repo. Next, add a Project board to track every piece of work.
- In the top-right corner of GitHub, click your profile picture.
- Click Your profile.
- On your profile page, click Projects.
- Click New project.
- Under "Start from scratch", click Board.
- In the Project name field, type Error Budget Control Room.
- Click Create project.
What is a GitHub Project board?
GitHub Projects (free tier) gives you a kanban-style board that tracks issues across columns like To Do, In Progress, and Done. Each issue you create maps to a piece of work in the build.
Now populate the board with six issues that map to each remaining piece of work. You can create them directly from the board without leaving the page.
- In the Todo column, click the + button at the bottom.
- Type the first title from the list below and press Enter.
- Repeat for each remaining title (six total):
Error Budget Control Room
Design: SLO spec and decision records
Build: Multi-window multi-burn-rate rules
Prove: Firing-order validation under load
Deliver: Dashboard, runbook, and tagged release
After-Action: Transfer review and README- Once all six items appear in the column, click the ... menu on each item and select Convert to issue.
- In the dialog, select your error-budget-control-room repository and confirm.
Your board should now show six issues in the Todo column, each linked to a real issue in the repository.
Clone and bring up the stack
Before running anything else, take a look at what the template gave you.
- Move to your Desktop by running this command:
cd ~/Desktop- Clone your repository by running this command:
git clone [[YOUR_REPO_URL="https://github.com/your-org/error-budget-control-room.git"]]You should see Git download the repository files. The output ends with "done" when the clone is complete.
- Move into the project folder by running:
cd error-budget-control-room- See what the starter template gave you by running:
lsYou should see files including compose.yaml, app.py, and requirements.txt. The compose.yaml file defines every container to start, and app.py is the sample service source code. One thing is missing: the Dockerfile that tells Docker how to package app.py into a container image.
What is Docker Compose?
Docker Compose reads the compose.yaml file and starts every container listed in it. Instead of running four separate commands to start four services, you describe them all in one file and Compose handles the rest.
Each service gets its container one of two ways. It either pulls a pre-built image from Docker Hub (like downloading an app someone else already packaged), or it builds one locally from a Dockerfile in the repo. Three services (Prometheus, Grafana, k6) use pre-built images. The sample service expects a Dockerfile that you will write next.
- In your editor, create a new file called Dockerfile (no file extension) in the root of the error-budget-control-room folder.
- Add the following build instructions to Dockerfile:
# Use a lightweight Python base image
FROM python:3.11-slim
# Set the working directory inside the container
WORKDIR /app
# Copy dependency list and install packages
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Copy the application source code
COPY app.py .
# Expose the port the service listens on
EXPOSE 8080
# Start the service
CMD ["python", "app.py"]What does this Dockerfile do?
FROM python:3.11-slim picks a minimal Python image as the starting point. Slim images are smaller and faster to pull than full ones.
COPY requirements.txt and RUN pip install install the Python packages the service needs (Flask and the Prometheus client library). Copying dependencies first means Docker can cache this layer and skip reinstalling if only app.py changes.
EXPOSE 8080 documents which port the service listens on. CMD tells Docker how to start the app when the container launches.
- Save Dockerfile.
- Confirm the file exists by running:
ls DockerfileYou should see Dockerfile printed back. The compose file can now find it when it builds the sample service image.
ls: cannot access 'Dockerfile'?
Make sure you created the file in the project root (error-budget-control-room/), not inside a subfolder. The file name must be exactly Dockerfile with a capital D and no extension.
If you used touch to create it, confirm your terminal is in the right directory with pwd. It should end in /error-budget-control-room.
Help me find my Dockerfile
- Start all four containers defined in compose.yaml by running:
docker compose up -d --buildWhat do these flags mean?
-d runs the containers in the background (detached mode), so you get your terminal back instead of watching logs scroll forever.
--build tells Compose to look at the Dockerfile in the repo and build a fresh image for the sample service before starting it. The other three services (Prometheus, Grafana, k6) don't need building — Compose downloads their ready-made images from Docker Hub the first time. This download may take a minute or two depending on your internet speed.
- Wait about 30 seconds for health checks to pass, then confirm all containers are healthy by running:
docker compose psYou should see all four services listed with a status of healthy or running.
- Open http://localhost:9090/targets in your browser. Confirm that sample-service shows a state of UP.
- Open http://localhost:3000 in your browser.
- Log in with username admin and password admin.
- Grafana will prompt you to change the password. You can skip this for local work by clicking Skip.
- Confirm you see the pre-provisioned Prometheus datasource (under Connections in the left sidebar) and the burn-down dashboard (under Dashboards). The dashboard will be empty since no alerting rules exist yet.
Containers not starting?
If docker compose ps shows containers as exited, check whether port 9090 or 3000 is already in use by another process. On macOS, run lsof -i :9090 to find the conflicting process.
If Docker reports an image pull error, confirm Docker Desktop is running and your network connection is active.
Help me diagnose my containers
Confirm metrics flow into Prometheus
A green target page means Prometheus can scrape the service. But scraping alone does not prove metrics are arriving. A quick burst of load proves the full pipeline.
- Run a 30-second baseline load test by executing:
docker compose exec k6 k6 run --duration 30s /scripts/load.jsWhat does this command do?
This runs the pre-written k6/load.js script inside the k6 container. It drives a constant 50 requests per second against the sample service for 30 seconds, generating http_requests_total metrics that Prometheus scrapes.
Before you check the result, predict: if k6 sent requests for 30 seconds and Prometheus scrapes every few seconds, do you expect the request rate query to return zero or non-zero?
- Open http://localhost:9090/graph in your browser.
- In the expression field, paste this PromQL query and click Execute:
rate(http_requests_total[1m])You should see non-zero values. This confirms the full pipeline is working: k6 drives load, sample-service records metrics, and Prometheus scrapes and stores them.
- Return to your Error Budget Control Room project board on GitHub.
- Drag the setup issue from Todo to Done.
Query returning empty results?
If the query returns no data, the k6 load might not have had time to register. Wait 15 seconds and click Execute again. Prometheus needs at least one full scrape interval to calculate a rate.
If it still shows nothing, go back to http://localhost:9090/targets and confirm the sample-service target is UP. A DOWN state means Prometheus cannot reach the service.
Help me debug my Prometheus query
Your stack is green and your board tracks every remaining piece of work. Next up, you will design the SLO and its indicator with AI-drafted decision records that you review and reject where the math does not hold.
Design with AI Assistance
Your stack is running and your board is tracking progress. Now you need a design before you build anything.
A two-person roster inheriting a service with no SLI needs one on day one, from metrics that already exist. This project builds the arithmetic that decides whether those two people get to sleep.
In this step, get ready to:
- Prompt your AI to draft candidate SLIs and decision records.
- Review the AI's output and reject at least one SLI candidate.
- Write ADR-004 yourself and draw the container view diagram.
Prompt your AI for design drafts
The design artifacts come before any code. You will ask your AI client to draft a requirements brief, three SLI candidates, and three of the four decision records.
- Create a new branch for the design work by running:
git checkout -b design/decisions- Create the directories for your design documents by running:
mkdir -p docs/decisions- Confirm both directories exist by running:
ls docs/You should see the decisions directory listed alongside the existing workbook-table.md file.
Now open your AI client (Claude, ChatGPT, or local Ollama) and paste the following prompt:
I am building an error-budget alerting system for a sample HTTP service.
System context:
- The service exposes two metrics:
http_requests_total{code, method}
http_request_duration_seconds_bucket{le}
- Two engineers share on-call. Waking one is a 50% capacity hit.
- The service has no existing SLI. We need one on day one from proxy metrics.
Please draft the following:
1. A requirements brief (docs/requirements-brief.md) naming the service,
the two-person roster constraint, and the business context.
2. Three candidate SLI definitions. For each, argue the valid-events
denominator in a comment (what counts as a valid event, excluding
health checks and client cancellations).
3. Three decision records in MADR v4 format (with status: accepted):
- ADR-001: Choose 99.9% SLO over 99.5% and 99.95%. Reversal trigger:
first month of real data.
- ADR-002: Choose proxy request metrics over application instrumentation.
Accept that HTTP-200-wrapped errors are invisible.
- ADR-003: Choose generated rules over hand-written PromQL. Accept that
the generator hides arithmetic so thresholds must be hand-derived.
Each record must name the alternative beaten, the tradeoff accepted,
and the trigger that would reverse the decision.What does this prompt do?
You are asking the AI to produce the first pass of your design documents. The AI drafts quickly, but the denominator argument (what counts as a valid event) is where engineers add value.
The three MADR v4 records each capture a single architectural decision: the context, the options considered, and the tradeoff accepted. Version 4.0.0 uses YAML frontmatter for status and date.
- Save the AI's requirements brief output to docs/requirements-brief.md.
- Save the AI's three ADR outputs to docs/decisions/ADR-001.md, docs/decisions/ADR-002.md, and docs/decisions/ADR-003.md.
- Keep the three SLI candidates visible (in a file or your AI chat history) for the next substep.
- Confirm the files exist by running:
ls docs/decisions/ && ls docs/requirements-brief.mdYou should see ADR-001.md, ADR-002.md, ADR-003.md listed, plus docs/requirements-brief.md confirmed.
Files not showing up?
Make sure you saved the AI output into the correct paths inside your repository's docs/ folder. If you see "No such file or directory", double-check that you created docs/decisions/ with the mkdir -p command above.
help me figure out why my docs files aren't in the right place
Review the AI's output and reject at least one SLI
The AI generated three SLI candidates. Your job now is to check each one against the metrics that actually exist on the service.
- Read through each of the three SLI candidates the AI proposed.
- For each candidate, ask: can this be measured using only http_requests_total{code, method} and http_request_duration_seconds_bucket{le}?
- Reject at least one candidate as unmeasurable from the available metrics.
- Write one sentence explaining why it cannot work (for example, a "correctness" SLI requires application-level assertions the proxy cannot see).
- Annotate the rejected candidate with REJECTED and your one-sentence reason directly in the SLI candidates file or in a comment at the top of the relevant ADR.
- Accept or revise the surviving SLI definition so that its good/total query references real metric labels.
- Record your rejection count (0-3) somewhere you can find it later. You will need this number for the after-action review.
Why reject an AI-generated candidate?
A valid event in SLI terms is one the service was meant to serve. Health checks and client cancellations are excluded because they do not reflect real user experience.
The AI cannot know which metrics your proxy actually exposes versus which it merely named. Your rejection proves you checked the denominator argument rather than rubber-stamping it.
Write ADR-004 and the container view, then merge
ADR-004 is the one the AI does not write. This record captures the decision to compress lab windows so the alert fires within your session.
- Create docs/decisions/ADR-004.md and write the following content:
---
status: accepted
date: 2026-07-22
---
# Lab Windows Are Compressed and This Is Stated
## Context and Problem Statement
A 45-minute session does not contain a one-hour burn window.
We need the alert to fire within the lab without changing the
arithmetic that decides whether it fires.
## Decision Drivers
- The session must observe a firing alert, not just assert one would fire.
- The multipliers (14.4x and 6x) and budget (0.001) must remain identical.
- The window ratio (1:12 long-to-short) must be preserved.
## Considered Options
- Skip the lab and trust the math on paper.
- Compress the budget period (changes semantics).
- Compress only the windows by factor 12 (preserves rate threshold).What does the first half establish?
The context names the constraint (45 minutes is too short for a 1-hour window). The decision drivers lock down what must not change: the multipliers, budget fraction, and window ratio.
Three options are listed so the record shows the alternatives that were beaten, not just the winner.
- Continue the file with the decision outcome section:
## Decision Outcome
Chosen option: Compress only the windows by factor 12.
- Fast-burn: 1h becomes 5m (long), 5m becomes 25s (short).
- Slow-burn: 6h becomes 30m (long), 30m becomes 2m30s (short).
- The budget period (30 days) is NOT rescaled.
### What is preserved
- Rate threshold (14.4x and 6x multipliers).
- Window ratio (long window is 12x the short window).
- The formula r_th = (p x T) / w still governs firing.
### What is NOT preserved
- Budget-consumption semantics (draining 2% of a 30-day budget
in 5 minutes does not mean the same thing as draining it in 1 hour).
### Reversal trigger
First month of real production data. If the SLO target moves
per ADR-001's reversal clause, the lab windows move with it.Why state what is NOT preserved?
Compressed windows prove that the threshold fires at the right burn rate. They do not prove that the budget drains at a meaningful pace. Stating this limit openly prevents someone from mistaking the lab for a production configuration.
- Save docs/decisions/ADR-004.md.
- Confirm all four ADRs exist by running:
ls docs/decisions/You should see ADR-001.md, ADR-002.md, ADR-003.md, and ADR-004.md listed.
Now draw the container view diagram. This diagram shows how the four containers connect before you plan the build.
- Create docs/container-view.md with the following content:
# Container View
```
+-----------+ scrape +----------------+
| prometheus| <------------------ | sample-service |
+-----------+ +----------------+
^ ^
| query | HTTP load
v |
+-----------+ +------+
| grafana | | k6 |
+-----------+ +------+
+-----------+ generates rules +-----------+
| sloth | ------------------> | prometheus|
+-----------+ (consumed via +-----------+
volume mount)
```
## Connections
- **prometheus** scrapes metrics from **sample-service** on :8080.
- **k6** drives constant HTTP load to **sample-service** on :8080.
- **grafana** queries **prometheus** on :9090.
- **sloth** generates recording/alerting rules consumed by prometheus.Why draw this before building?
The diagram names every piece and its connections. If the plan and the diagram disagree about which containers exist, you fix the one that is wrong before writing any rules.
- Save docs/container-view.md.
- Stage and commit all design documents by running:
git add docs/
git commit -m "Add design artifacts: ADRs, requirements brief, container view"- Push the branch to your remote by running:
git push -u origin design/decisions- Open your repository on GitHub.
- Click Compare & pull request for the design/decisions branch.
- In the PR description, add Closes #N where #N is the issue number of your design task issue.
- Click Create pull request, then click Merge pull request.
- Click Confirm merge.
Before you check, predict: what status should the design issue show on your Project board now that the PR is merged?
- Open your GitHub Project board and confirm the design issue has moved to Done.
The Closes #N keyword automatically closed the issue when the PR merged. Your board should show the design issue in the Done column.
Issue still showing as open?
Make sure the Closes #N text is in the PR description (not a comment) and that #N matches the exact issue number for your design task. If the PR is already merged without it, close the issue manually on the board.
help me link my PR to the correct GitHub issue so it closes automatically
Your design is locked in. Next up, you will take the surviving SLI and turn it into an OpenSLO document that generates real alerting rules.
Build the SLO Rules
The decisions are made. Now you need a machine-readable source of truth for the SLO that a tool can consume and turn into alerting rules.
A single OpenSLO document captures the target, the compliance window, and the indicator queries. Sloth reads that document and generates multi-window multi-burn-rate rules, eliminating hand-written PromQL drift. But the generator hides the arithmetic, so you will hand-verify every production threshold against the Workbook table before trusting it.
In this step, get ready to:
- Write the OpenSLO source document and generate production rules with Sloth.
- Create lab rules with compressed windows that fire within the session.
- Load both rule sets into Prometheus and verify they parse without error.
Write the OpenSLO source and generate production rules
The OpenSLO spec is a vendor-neutral way to state an SLO. Sloth reads it and outputs Prometheus-native recording and alerting rules. Your job is to define what "good" and "total" mean using metrics the service already exposes.
- Create the branch and the slo/ directory by running these commands:
git checkout -b build/rules
mkdir slo- Create the file slo/availability.yml and paste the following OpenSLO document:
apiVersion: openslo/v1alpha
kind: SLO
metadata:
name: sample-service-availability
displayName: Sample Service Availability
spec:
service: sample-service
description: "Availability SLO measured at the proxy layer"
budgetingMethod: Occurrences
objectives:
- target: 0.999
ratioMetrics:
good:
source: prometheus
queryType: promql
query: sum(rate(http_requests_total{job="sample-service",code!~"5.."}[{{.window}}]))
total:
source: prometheus
queryType: promql
query: sum(rate(http_requests_total{job="sample-service"}[{{.window}}]))
timeWindows:
- count: 30
unit: DayWhat does this document do?
This is the single source of truth for your SLO. The target: 0.999 sets the 99.9% objective over a 30-day window.
The good query counts requests that did NOT return a 5xx status. The total query counts all requests. The {{.window}} placeholder is replaced by Sloth with the actual window duration when generating rules.
- Create the rules/ directory, then generate production rules by running:
mkdir rules
docker compose exec -T sloth sloth generate -i /slo/availability.yml > rules/production.ymlWhy the -T flag?
The -T flag disables pseudo-TTY allocation. Without it, the terminal injects formatting characters into the output and corrupts the YAML file you are redirecting into.
- Open rules/production.yml and verify it against docs/workbook-table.md. Look for these two expressions:
- Fast-burn page alert: the expr must contain 14.4 * 0.001 with 1h and 5m windows.
- Slow-burn page alert: the expr must contain 6 * 0.001 with 6h and 30m windows.
These are the Google SRE Workbook Table 5-8 production values. If either value differs from the Workbook table, the generator is wrong and the hand derivation governs.
Where do 14.4 and 6 come from?
The Workbook formula is: burn_rate_threshold = (fraction_of_budget_consumed x compliance_period) / long_window. For 2% budget consumed in 1 hour with a 30-day (720h) period: 0.02 x 720 / 1 = 14.4. For 5% budget consumed in 6 hours: 0.05 x 720 / 6 = 6.
The short window is always 1/12 of the long window. That gives 5m for the fast-burn (1h / 12) and 30m for the slow-burn (6h / 12). Both windows must exceed the threshold simultaneously for the alert to fire.
🙋♀️ Sloth exited with an error?
Check that slo/availability.yml uses apiVersion: openslo/v1alpha exactly. Sloth does not accept other OpenSLO versions. Also confirm the slo/ directory is mounted into the Sloth container in your docker-compose.yml.
help me debug a Sloth generate error with my OpenSLO file
Create the lab rules with compressed windows
The production rules have windows too long to observe in a 45-minute session. ADR-004 decided to compress them by a factor of 12 while preserving the rate thresholds. The multipliers stay identical. Only the timescale changes.
- Copy the production rules to a lab file by running:
cp rules/production.yml rules/lab.yml- Open rules/lab.yml in your editor. Add this comment block at the very top of the file, before any existing content:
# Lab rules for error-budget-control-room
# These rules preserve rate thresholds and window ratios.
# They do NOT preserve budget-consumption semantics.
# The budget period is not rescaled.
#
# Production -> Lab window compression (factor 12):
# Fast-burn long: 1h -> 5m
# Fast-burn short: 5m -> 25s
# Slow-burn long: 6h -> 30m
# Slow-burn short: 30m -> 2m30s- Now replace the window durations throughout the file. Use find-and-replace in your editor:
- Replace the fast-burn long window from 1h to 5m.
- Replace the fast-burn short window from 5m to 25s.
- Replace the slow-burn long window from 6h to 30m.
- Replace the slow-burn short window from 30m to 2m30s.
What stays the same and what changes?
The multipliers (14.4 and 6) and the budget (0.001) remain identical to production. Only the window durations change. The ratio between long and short windows (12:1) is also preserved.
What this means: the fast-burn rule now fires after observing a sustained burn for 5 minutes instead of 1 hour. The slow-burn rule fires after 30 minutes instead of 6 hours. The math that decides whether to page is the same. The observation period is compressed so you can see it happen in this session.
- Save rules/lab.yml.
- Confirm both rule files exist by running:
ls rules/You should see lab.yml and production.yml listed.
Only see one file?
If production.yml is missing, the Sloth generate command in the previous substep may have failed silently. Re-run it and check for error output. If lab.yml is missing, confirm the cp command ran in the correct directory.
help me figure out why my rules directory is missing a file
Load both rule sets into Prometheus and verify
The rules exist on disk but Prometheus does not know about them yet. You need to ensure the rules/ directory is mounted into the Prometheus container and then trigger a configuration reload.
- Open docker-compose.yml and find the prometheus service's volumes section. Add a mount for the rules directory:
- ./rules:/etc/prometheus/rulesWhat does this mount do?
This makes your local rules/ directory appear inside the Prometheus container at /etc/prometheus/rules. The starter repo's prometheus.yml already has a rule_files entry pointing to that path, so Prometheus will load any .yml files placed there.
- Apply the volume change by restarting the Prometheus container:
docker compose up -d prometheus- Once the container is healthy, trigger a config reload by running:
docker compose exec prometheus wget -qO- --post-data= http://localhost:9090/-/reloadWhy wget instead of curl?
The Prometheus container image does not include curl. The wget --post-data= trick sends an empty POST request, which is all the /-/reload endpoint requires. This endpoint only works because the starter repo starts Prometheus with --web.enable-lifecycle.
Before you check the rules page, predict: do you expect to see both production AND lab rules loaded, or just one set?
- Open http://localhost:9090/rules in your browser.
You should see both rule groups listed without any errors. The fast-burn and slow-burn alerting rules appear for both the production and lab configurations. The Grafana burn-down panel now has rules to query.
Rules page shows an error or is empty?
If you see a YAML parse error, open the file mentioned in the error message and check for indentation issues or stray characters from the TTY redirect. If the page is empty, confirm the volume mount line was added to the correct service (prometheus, not another container) and that you restarted with docker compose up -d prometheus.
help me debug why Prometheus is not loading my rule files
- Commit all new files and open a pull request by running:
git add slo/ rules/ docker-compose.yml
git commit -m "Add OpenSLO source, production and lab rules"
git push -u origin build/rules- Open a pull request on GitHub from build/rules into main.
- In the PR description, reference the build issue so merging closes it automatically (e.g. "Closes #N").
- Merge the pull request. Confirm the build issue moves to Done on your Project board.
Your SLO now lives in a single source of truth, and Prometheus has both the production reference set and the lab set ready to fire. Next up, you will inject errors and watch the fast-burn rule fire exactly when the math predicts.
Prove Firing Order
Your rules are loaded and the stack is green. The claim you need to prove is about ORDER: the fast-burn alert fires before the slow-burn alert because its 5-minute window elapses first.
This is not exclusion. A severe outage trips both rules in the Workbook design. The fast page simply arrives first. That arrival is what justifies waking one of two engineers rather than filing a ticket.
In this step, get ready to:
- Start sustained load and inject a 10% error rate into the sample service.
- Observe the fast-burn alert transition to FIRING while the slow-burn alert remains pending.
- Verify the burn-down panel slope against a hand-computed prediction.
Start load and inject errors
The lab's compressed windows mean the fast-burn rule needs only 5 minutes of sustained errors to fire. You will start a 10-minute load run, inject a 10% error rate, and note the wall-clock time so you can verify the alert timeline.
- Create the prove branch by running this command:
git checkout -b prove/firing-order- Start the sustained load run by running this command:
docker compose exec k6 k6 run --duration 10m /scripts/load.jsWhat does this command do?
This executes the k6 load script inside the k6 container, driving a constant 50 RPS against the sample service for 10 minutes. The terminal will show a progress bar and real-time metrics while the test runs.
This command occupies your current terminal for the full 10 minutes. You need a second terminal for the next step.
You should see k6 print its banner and a progress bar showing virtual users and elapsed time.
Not seeing output from k6?
Confirm the k6 container is running with docker compose ps. If it shows as exited, run docker compose up -d k6 to restart it.
help me debug why the k6 container is not running
- Open a new terminal window. Press Cmd+T (macOS) or right-click the terminal tab bar and select New Tab (Windows).
- Inject a 10% error rate into the sample service by running this command:
docker compose exec k6 wget -qO- "http://sample-service:8080/inject-errors?rate=10"What does this command do?
This hits the sample service's error-injection toggle from inside the k6 container. After this call, the service returns HTTP 500 on 10% of all incoming requests. The remaining 90% continue to succeed normally.
- Note the current wall-clock time (e.g. 14:32). You will use this to verify the alert timeline in the next substep.
You should see a confirmation response from the service indicating the error rate is now set to 10%.
No response from the inject-errors endpoint?
Make sure you are running the command from inside the k6 container (via docker compose exec k6), not from your host. The sample-service hostname resolves only inside the Docker network.
help me troubleshoot the error injection endpoint
Observe firing order
The fast-burn alert uses a 5-minute long window and a 25-second short window (lab-compressed from the Workbook's 1h/5m). The slow-burn alert uses a 30-minute long window and a 2-minute-30-second short window (lab-compressed from 6h/30m). Both need their long AND short windows to exceed the threshold simultaneously before firing.
Before you check the alerts page, predict: which alert will fire first, and why?
- Open http://localhost:9090/alerts in your browser.
- Watch the page over the next 5 minutes. Refresh every 30-60 seconds to see state changes.
Within approximately 5 minutes of your error injection time, the fast-burn alert (14.4x multiplier) transitions from inactive to pending to FIRING. The slow-burn alert (6x multiplier) remains inactive or pending because its 30-minute long window has not yet elapsed.
Why does the fast-burn alert fire first?
Both rules require the burn rate to exceed a threshold in BOTH a long window and a short window simultaneously. The fast-burn rule's long window is 5 minutes. After 5 minutes of 10% errors, both its windows have enough data to confirm the burn is real AND still happening.
The slow-burn rule's long window is 30 minutes. Even though the errors started at the same time, the rule cannot confirm the burn is real until 30 minutes of data accumulates. The fast rule pages first because its window is shorter, not because it excludes the slow rule.
- Take a screenshot of the Prometheus alerts page showing the fast-burn alert in FIRING state and the slow-burn alert still inactive or pending. Note the timestamps.
Verify the burn-down slope and commit evidence
The Grafana burn-down panel visualises how quickly the error budget drains. You can predict the slope from arithmetic and compare it to the observed drain rate.
- Open the burn-down dashboard at http://localhost:3000 (admin/admin if prompted).
- Locate the error-budget burn-down panel. It should show a downward slope starting from when you injected errors.
Compute the expected slope. At 10% error rate with a 0.1% budget (0.001), the burn rate equals 0.10 / 0.001 = 100. The budget drains at 100x the sustainable rate.
- Compare the panel's observed slope to the predicted burn rate of 100. Record the percentage difference between observed and predicted.
- Your target is within 10% of the predicted rate.
What does the burn rate of 100 mean?
A burn rate of 1 means you spend your entire 30-day error budget exactly on time. A burn rate of 100 means you spend that entire budget in 0.3 days (about 7 hours). The Grafana panel should show the budget draining approximately 100x faster than the sustainable baseline.
- Create a file to record your evidence. In your second terminal, run:
cat > docs/firing-order-evidence.md << 'EOF'
# Firing Order Evidence
## Timeline
- Error injection time: [NOTE YOUR TIME]
- Fast-burn alert FIRING at: [NOTE TIMESTAMP]
- Slow-burn alert status at that time: inactive/pending
## Burn-Down Slope
- Predicted burn rate: 100 (0.10 / 0.001)
- Observed slope: [NOTE FROM PANEL]
- Percentage difference: [CALCULATE]
## Conclusion
The fast-burn alert fired first because its 5-minute window
elapsed before the slow-burn rule's 30-minute window.
EOFWhat is this file?
This markdown file captures your experimental evidence: the timestamps proving firing order, and the slope comparison proving the arithmetic predicts reality. Replace the bracketed placeholders with your actual observed values before committing.
- Edit docs/firing-order-evidence.md to fill in your actual timestamps, observed slope, and percentage difference.
- Commit the evidence and push the branch by running these commands:
git add docs/firing-order-evidence.md
git commit -m "Add firing-order evidence with timestamps and slope comparison"
git push -u origin prove/firing-order- On GitHub, open a pull request from prove/firing-order into main.
- In the PR description, reference the prove issue so it closes automatically (e.g. "Closes #N" where N is the prove issue number).
- Merge the PR.
Confirm the prove issue is now closed on your Project board.
Alert never transitioned to FIRING?
Check that both rule files are loaded at http://localhost:9090/rules. If the lab rules are missing, verify the volume mount in your compose file points to rules/lab.yml.
Also confirm the error injection is active. Run the inject command again and check that http_req_failed shows approximately 10% in the k6 output. If errors are not flowing, the alert has nothing to trigger on.
help me debug why my burn-rate alert is not firing
Your firing-order evidence is committed and the prove issue is closed. Next up, you will derive the burn-rate thresholds from first principles and deliver the stakeholder artifacts.
Derive and Deliver
You proved the fast-burn alert fires first. That is Step 4's artifact. But firing an alert you were handed is not the skill this build exists to teach.
The graded artifact is the derivation. A learner who can show why 14.4x is the page threshold (and not, say, 10x or 20x) has something a copy-paste operator does not. In this step you produce three of the four signature artifacts: the hand-worked derivation, the stakeholder presentation, and the teach-back. The fourth (the after-action review) is the Secret Mission.
In this step, get ready to:
- Write a hand-worked derivation showing production and lab thresholds side by side.
- Write a teach-back README a non-pager-carrier can follow.
- Write a 3-slide error-budget policy readout and tag the release.
Write the hand-worked derivation
The Workbook's Table 5-8 gives thresholds, but not the reasoning that produced them. Your derivation shows the formula, plugs in both production and lab values, and states explicitly what the lab preserves and what it does not.
- Create and check out the delivery branch by running:
git checkout -b deliver/docs- Create docs/derivation.md with the following content:
# Burn-Rate Threshold Derivation
## Formula (SRE Workbook Chapter 5)
r_th = (p x T) / w
- p = fraction of error budget consumed before alert fires
- T = compliance period in hours (720h for a 30-day window)
- w = long window in hours
## Production Thresholds (Table 5-8)
| Severity | p | w | Calculation | r_th | Windows (long/short) |
|----------|------|-----|-------------|------|----------------------|
| Fast page | 0.02 | 1h | 0.02 x 720 / 1 | 14.4 | 1h / 5m |
| Slow page | 0.05 | 6h | 0.05 x 720 / 6 | 6 | 6h / 30m |## Lab Thresholds (factor 12 compression)
| Severity | r_th | Windows (long/short) | Compressed from |
|----------|------|----------------------|-----------------|
| Fast page | 14.4 | 5m / 25s | 1h / 5m |
| Slow page | 6 | 30m / 2m30s | 6h / 30m |
## What is preserved
- Rate thresholds: 14.4 and 6 (identical multipliers)
- Window ratio: long / short = 12 (identical ratio)
- Alert expression: multiplier x budget (14.4 * 0.001 and 6 * 0.001)
## What is NOT preserved
- The 30-day budget period is NOT compressed
- Budget-consumption semantics do not hold at lab timescale
- A full budget drain cannot be reproduced in 45 minutesWhat does this derivation prove?
The formula r_th = (p x T) / w descends from a single question: how fast must the budget drain for you to care within this time window?
At 14.4x, the entire 30-day budget is gone in about 50 hours (roughly 2 days). That is worth waking one of two engineers. At 6x, it lasts about 5 days. Still a page, but a slower one. The lab uses the same multipliers at a compressed timescale so you can observe firing without waiting hours.
- Confirm the file exists by running:
cat docs/derivation.md | head -20You should see the formula and the production threshold table at the top of the output.
File not showing up?
Make sure you created the file inside the docs/ directory that already exists in your repository (it holds your decision records and other docs from earlier steps).
help me find where docs/derivation.md should go in my project
Write the teach-back README
The handover test for this project: someone who has never carried a pager reads your teach-back and can say what a burn rate is, why a fast burn pages, and why a slow burn only tickets. If it needs a diagram they cannot follow, it is not finished.
- Create docs/teach-back.md with the following structure:
# Teach-Back: Error Budget Alerting
## What is an error budget?
At 99.9% availability over 30 days, you are allowed 43 minutes of
downtime per month. That 43 minutes is your error budget.
## What is burn rate?
Burn rate measures how fast you spend that budget.
- Burn rate 1 = spending exactly on pace (budget lasts the full month)
- Burn rate 14.4 = budget gone in about 2 days
- Burn rate 6 = budget gone in about 5 days
## Why two windows?
The long window confirms the burn is real (not a one-second spike).
The short window confirms it is still happening right now.
Both must exceed the threshold simultaneously for the alert to fire.## Why does 14.4x justify a page?
With a two-person on-call roster, waking one engineer is a 50% capacity
hit. Only a burn that exhausts the budget in 2 days justifies that cost.
14.4x is that rate. A 6x burn is slower (5 days) but still a page.
A 1x burn over 3 days is a ticket handled in working hours.
## Decision record defenses
**ADR-001 (99.9% target):** Every threshold descends from this choice.
If the first month of real data shows it is too tight, the SLO moves
to 99.5% and all thresholds recalculate.
**ADR-002 (Proxy metrics):** A day-one indicator from metrics that
already exist. Tradeoff: HTTP-200-wrapped errors are invisible.
**ADR-003 (Generated rules):** Eliminates hand-written PromQL drift.
Thresholds are hand-derived to compensate for hidden arithmetic.
**ADR-004 (Lab compression):** Preserves rate thresholds and window
ratios. Does NOT preserve budget-consumption semantics.## Where does this transfer?
Any service watched by a roster too small to absorb a bad night has
this same problem. The skill is deriving a threshold from a budget
rather than inheriting one, and being able to show the derivation
when someone asks why they were woken.What makes a good teach-back?
The test is comprehension, not length. Every term is defined once in one plain clause on first use. If a reader has to jump back to understand a later section, the definitions are not self-contained enough. Read your own teach-back as if you had never held a pager. Does each sentence land without the next one?
- Verify the teach-back reads end to end by running:
cat docs/teach-back.mdRead through the output. Every section should be understandable on its own without referencing another section.
Teach-back feels too jargon-heavy?
Check that you define each term (error budget, burn rate, long window, short window) before you use it in a later section. Move definitions up if any section references an undefined term.
help me simplify my teach-back language
Write the readout and tag the release
The stakeholder presentation is the artifact most often dropped. It is the one that separates a solutions engineer from an engineer. Three slides, each with speaker notes that say what you would say out loud.
- Create docs/readout.md with the following structure:
# Error-Budget Policy Readout
## Slide 1: Our SLO and What It Costs
- Target: 99.9% availability over 30 days
- Error budget: 43 minutes of allowed downtime per month
- Current burn rate: [insert observed rate from Step 4]
- Constraint: two-person on-call roster
**Speaker notes:** We chose 99.9% because [reference ADR-001].
Every threshold in this system descends from this single number.
Our roster has two engineers. Waking one is a 50% capacity hit.## Slide 2: When We Page and When We Ticket
| Burn rate | Action | Why |
|-----------|--------|-----|
| 14.4x | Page | Budget gone in ~2 days. Half the roster is asleep. |
| 6x | Slower page | Budget gone in ~5 days. Still urgent. |
| 1x over 3 days | Ticket | Time to handle in working hours. |
**Speaker notes:** The 6x row is a second, slower page, not a ticket.
Both page-severity rows are pages. The difference is urgency.
14.4x means "wake someone now." 6x means "wake someone soon."
1x over 3 days means "file it and handle it tomorrow morning."## Slide 3: What Changes If the First Month Disagrees
- ADR-001 reversal: if 99.9% is too tight, SLO moves to 99.5%
- ADR-002 reversal: if HTTP-200-wrapped errors cause missed incidents,
switch to application-level instrumentation
- ADR-003 reversal: if generated rules hide a bug, hand-write the PromQL
- ADR-004 reversal: first month of real data retires the lab windows
**Speaker notes:** Every decision has a named trigger that reverses it.
We do not defend choices past the point where evidence contradicts them.
The first month of production data is the test.Why name the reversal triggers?
A decision record without a reversal trigger is a policy carved in stone. Naming the trigger up front shows the audience you are defending a position, not a belief. If the data contradicts the position, the system updates automatically because the trigger was already defined.
- Confirm all three new files exist by running:
ls docs/derivation.md docs/teach-back.md docs/readout.mdYou should see all three filenames listed without errors.
- Stage, commit, and push the delivery branch by running:
git add docs/derivation.md docs/teach-back.md docs/readout.md
git commit -m "Add derivation, teach-back, and readout docs"
git push -u origin deliver/docs- Switch back to your GitHub repository in the browser.
- Open a Pull Request from deliver/docs into main.
- In the PR description, add Closes #N referencing your deliver issue number.
- Merge the Pull Request.
- Pull the merged changes and tag the release by running:
git checkout main
git pull
git tag -a v1.0.0 -m "Release v1.0.0: Error Budget Control Room"
git push origin v1.0.0Why tag a release?
The tag marks a known-good state of the repository. Anyone who clones at v1.0.0 gets the complete system: stack, rules, derivation, teach-back, and readout. It is the line between "in progress" and "shipped."
Before you verify, predict: what should you see when you check the tag and the board?
- Confirm the tag exists by running:
git tagYou should see v1.0.0 in the output.
- Return to your GitHub Project board.
- Confirm the deliver issue is now closed (the PR's Closes #N keyword closed it on merge).
Tag not showing on GitHub?
Make sure you pushed the tag with git push origin v1.0.0. Tags are not pushed by a regular git push unless you specify them explicitly.
help me push my release tag to GitHub
Three of four signature artifacts are shipped. The validated system proved the math. The readout presents it. The teach-back transfers it. One artifact remains: the after-action review, where you score the AI honestly. That is the Secret Mission.
Secret mission
After-Action Review and AI Scoring
The AI drafted fast, but was it right? Score its accuracy against the arithmetic you verified by hand. Write the after-action review while the frustration is fresh, because honest answers do not survive a night's sleep.
Clean Up Your Resources
Clean Up Your Resources
Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally with Docker containers and a free GitHub repository, so there are no ongoing costs.
Resources you used:
- Four Docker containers (Prometheus, Grafana, sample-service, k6) running locally.
- Docker volumes storing Prometheus and Grafana data.
- One private GitHub repository with Project board, issues, and release v1.0.0.
Keep everything running
No action needed. Choose this if you are still actively building or want to experiment further with the alerting rules.
- The stack uses approximately 500MB RAM total and no external paid resources.
- Your GitHub repository is on the free tier and incurs no cost regardless of how long it persists.
- Docker Desktop will continue running all four containers in the background.
Pause - I'll come back to this later
Shut down running containers to free up memory while keeping all data intact. You can restart later without rebuilding.
- Stop all containers by running:
docker compose stop- When you are ready to resume, restart the containers by running:
docker compose start- Your Prometheus data, Grafana dashboards, and rule files remain on disk.
Delete - I don't want to use this again
Remove all project resources and start fresh. This is not reversible for local data.
- Remove all containers, networks, and volumes by running:
docker compose down -v- Delete the local project folder. On macOS or Linux, run:
cd ~
rm -rf ~/Desktop/error-budget-control-room- On Windows, run:
cd ~
rmdir /s /q %USERPROFILE%\Desktop\error-budget-control-room- To delete the GitHub repository, open your repository page and click the Settings tab.
- Scroll down to the Danger Zone section and click Delete this repository.
- Follow the confirmation prompts: click I want to delete this repository, click I have read and understand these effects, then type the repository name and confirm.
Nice Work!
Nice Work!
You just built the arithmetic that decides whether two people get to sleep. From a single OpenSLO document, you generated burn-rate alerting rules, proved their firing order experimentally, and shipped the whole thing through a private GitHub repository with every PR merged and every issue closed.
You've learned how to:
- Hand-derive burn-rate thresholds from first principles using the Workbook formula r_th = (p x T) / w, proving both the 14.4x fast-burn and 6x slow-burn page thresholds rather than copying them.
- Prove firing order experimentally: the fast-burn page arrives before the slow-burn page because its window is shorter. That order is the arithmetic that justifies waking one of two engineers.
- Deliver four signature artifacts from a tagged release: the validated working system, the stakeholder presentation (3-slide readout with the two-person math), the teach-back README a non-pager-carrier can follow, and four MADR v4 decision records each naming the alternative, the tradeoff, and the reversal trigger.
- Secret Mission: Completed honest meta-analysis of the AI-assisted workflow, reporting both the AI's accuracy and the engineer's review rigour as real numbers. After-action written while the frustration is fresh, so a person arriving tomorrow can run the whole thing and understand why each decision was made.
Ready to quiz yourself?