SRE Error Budget Arithmetic

Introduction

30 Second Summary

When two people share an on-call rotation, every alert that wakes someone at 3 a.m. needs to be worth it. The difference between paging someone now and letting it wait until morning comes down to arithmetic you can derive by hand.

In this project, you will calculate the error budget for a 99.9% availability target, derive the alerting thresholds by hand, and prove those thresholds with synthetic tests that run in seconds on your local machine. No server, no scraped data, no cloud account.

What You'll Build

You will open a single HTML page, point at proof results that ran in under five seconds, and walk a reviewer through exactly why one burn rate wakes someone while another only raises a ticket.

By the end of this project, you'll have:

• A validated alerting proof that runs in under five seconds from one command, showing a 10% error ratio fires both page and ticket while a 0.2% error ratio fires ticket alone and wakes nobody.
• A hand-derived threshold table with four burn rates (14.4, 6, 3, 1) placed beside the values the generator emitted, so every number traces back to a formula you can defend out loud.
• A stakeholder readout as a self-contained HTML page showing the SLO, the derivation, the proof results, four documented decisions, and what the proof does not cover.
• Secret Mission: Write a teach-back page that explains burn rate to someone who has never carried a pager, an after-action review written while the frustration is fresh, and an AI accuracy score.

Are there any prerequisites?

You need a terminal, git installed, comfort reading YAML, and an AI chat client you already have access to (Claude, ChatGPT, a local model via Ollama, or any equivalent). No Prometheus, Go, or SRE experience is required.

Before We Start

Before any tool is opened, this is the moment to lock in what you are about to build and why it matters. A two-person on-call roster cannot absorb a bad night, so every threshold you set must be derived by hand and defended out loud.

Init the Repo and Verify Both CLIs

Without verified tool versions, the generated rules and unit tests you build later cannot be trusted. This step gets two CLI programs onto your machine and confirms they work.

You will also set up a git repo with a starter folder that contains reference materials, test inputs, and a task checklist. Everything is tracked from the first commit.

In this step, get ready to:

• Initialize a git repo with the starter folder committed.
• Download Sloth v0.16.0 and promtool from Prometheus 3.13.0.
• Verify both CLIs pass their host checks.

Create the project folder and initialize git

The only two prerequisites are git and a text editor. Everything else is downloaded in this step.

Open your terminal and the IDE of your choice (Cursor, VS Code, or another editor) to follow along with the commands below.

• Check that git is installed by running this command:

git --version

You should see a version number in the output.

Command not found?

Install git from https://git-scm.com/downloads and restart your terminal.

Help me install git on my machine.

• Move to your Desktop by running this command:

cd ~/Desktop

• Create the project folder, move into it, initialize a git repo, and create the starter/ directory by running these commands:

mkdir error-budget-proof
cd error-budget-proof
git init
mkdir starter

You should see a message that includes "Initialized empty Git repository" in the output.

Stage the starter files and make the initial commit

The starter folder ships reference materials, test input templates, and a task checklist so you spend your time on the argument and the arithmetic rather than the typing.

• Create a file called .gitattributes in the root of your error-budget-proof folder, and paste the following content:

* text=auto eol=lf

Why pin line endings to LF?

Sloth on Windows fails silently when it reads a spec file with CRLF line endings. This one-line file tells git to normalize every file to LF, preventing that silent failure on any platform.

• Create a file called TASKS.md in the project root, and paste this content:

# Tasks

- [ ] Setup: init repo, download and verify both CLIs
- [ ] Design: four ADRs, three candidate SLIs, SLI review, requirements brief, component view, plan
- [ ] Generate: Sloth spec, generated rules, threshold derivations (four rows, twelve values checked)
- [ ] Test: promtool tests pass (10% fires both, 0.2% fires ticket alone, burn rates 100 and 2 proven)
- [ ] Readout: stakeholder HTML, README, all boxes ticked, working tree clean, tagged v1.0.0

What is the task checklist?

One checkbox per task, ticked as you complete it. This is your single source of truth for project progress. You will tick the first box at the end of this step.

• Confirm both files exist by running:

macOS

ls -a

Windows

ls -Force

You should see .git, .gitattributes, TASKS.md, and starter in the output.

• Create starter/reference-table.md in your text editor and paste this content:

# Reference Table (SRE Workbook Chapter 5)

SLO: 99.9% over 30 days. Error budget: 0.001. Period: 720 hours.

| Budget share | Window  | Burn rate | Long window | Short window | Notification |
|------------- |-------- |---------- |------------ |------------- |------------- |
| 2% (0.02)    | 1 hour  | 14.4      | 1 hour      | 5 minutes    | Page         |
| 5% (0.05)    | 6 hours | 6         | 6 hours     | 30 minutes   | Page         |
| 10% (0.10)   | 1 day   | 3         | 1 day       | 2 hours      | Ticket       |
| 10% (0.10)   | 3 days  | 1         | 3 days      | 6 hours      | Ticket       |

Source: Google SRE Workbook, Chapter 5, "Alerting on SLOs."

• Create starter/series-outage.txt and paste:

# Severe outage: 10% sustained error ratio.
# 900 good + 100 failing per minute. Burn rate = 0.1 / 0.001 = 100.
# Use in promtool input_series:
#   series: 'http_request_duration_seconds_count{job="bordercontrol",code="200"}'
#     values: '0+900x120'
#   series: 'http_request_duration_seconds_count{job="bordercontrol",code="500"}'
#     values: '0+100x120'

• Confirm these two files exist by running:

ls starter/

You should see reference-table.md and series-outage.txt listed.

• Create starter/series-slowleak.txt and paste:

# Slow leak: 0.2% sustained error ratio.
# 998 good + 2 failing per minute. Burn rate = 0.002 / 0.001 = 2.
# Use in promtool input_series:
#   series: 'http_request_duration_seconds_count{job="bordercontrol",code="200"}'
#     values: '0+998x420'
#   series: 'http_request_duration_seconds_count{job="bordercontrol",code="500"}'
#     values: '0+2x420'

• Create starter/worked-example.md and paste:

# Worked Example: Deriving One Row

Take the first row of the reference table.

The question: how fast does the error budget burn if 2% of it
disappears in one hour?

The period is 30 days = 720 hours.
The budget share is 2% = 0.02.
The window is 1 hour.

burn_rate = budget_share * (period / window)
burn_rate = 0.02 * (720 / 1)
burn_rate = 0.02 * 720
burn_rate = 14.4

A burn rate of 14.4 means the service is consuming its error
budget 14.4 times faster than the pace that would use it up
exactly over the 30-day window. At this pace the entire budget
runs out in 720 / 14.4 = 50 hours, about two days.

That warrants waking someone. The alert is a page.

Repeat this for the other three rows, placing each derived value
beside the value the generator emitted.

• Confirm the starter directory now has four files by running:

ls starter/

You should see reference-table.md, series-outage.txt, series-slowleak.txt, and worked-example.md.

What are these reference files?

The reference table contains the four-row burn-rate table from the SRE Workbook. You will check your hand-derived thresholds against it in Step 3.

The two series files are templates for the promtool test scenarios you write in Step 4: one models a severe outage (10% error ratio, burn rate 100), the other a slow leak (0.2% error ratio, burn rate 2).

The worked example walks through deriving one row of the reference table so you have a model for the other three.

Now create the fetch script that automates the CLI download for your platform. The script belongs in the starter/ directory.

macOS

• Create starter/fetch.sh and paste the platform-detection block:

#!/usr/bin/env bash
# Fetch Sloth v0.16.0 and promtool from Prometheus 3.13.0.
# Detects macOS vs Linux. Run from the project root.
set -euo pipefail

OS="$(uname -s)"
ARCH="$(uname -m)"

case "$OS" in
  Darwin)
    case "$ARCH" in
      arm64) SLOTH_BIN="sloth-darwin-arm64"; PROM_ARCHIVE="prometheus-3.13.0.darwin-arm64.tar.gz" ;;
      *)     SLOTH_BIN="sloth-darwin-amd64"; PROM_ARCHIVE="prometheus-3.13.0.darwin-amd64.tar.gz" ;;
    esac ;;
  Linux)
    SLOTH_BIN="sloth-linux-amd64"; PROM_ARCHIVE="prometheus-3.13.0.linux-amd64.tar.gz" ;;
  *) echo "Unsupported OS: $OS. Use fetch.ps1 on Windows."; exit 1 ;;
esac

• Still in starter/fetch.sh, add the download and verification block below the esac line:

echo "Downloading Sloth v0.16.0..."
curl -fSL "https://github.com/slok/sloth/releases/download/v0.16.0/${SLOTH_BIN}" -o sloth
chmod +x sloth

echo "Downloading Prometheus 3.13.0..."
curl -fSL "https://github.com/prometheus/prometheus/releases/download/v3.13.0/${PROM_ARCHIVE}" -o prom.tar.gz
tar xzf prom.tar.gz --strip-components=1 "*/promtool"
rm prom.tar.gz
chmod +x promtool

echo "Verifying..."
./sloth version
./promtool --version
echo "Done. Move sloth and promtool to a directory on your PATH."

✔️ Awesome, I've got everything!

Save starter/fetch.sh.

ⓧ I'd like to double check the full code

Your starter/fetch.sh should contain the platform-detection block (starting with #!/usr/bin/env bash) immediately followed by the download-and-verify block (starting with echo "Downloading Sloth v0.16.0..."). Scroll up and confirm both pieces are present and in order.

Windows

• Create starter/fetch.ps1 and paste:

# Fetch Sloth v0.16.0 and promtool from Prometheus 3.13.0 on Windows.
# Run from the project root.
$ErrorActionPreference = "Stop"

Write-Host "Downloading Sloth v0.16.0..."
Invoke-WebRequest -Uri "https://github.com/slok/sloth/releases/download/v0.16.0/sloth-windows-amd64.exe" -OutFile "sloth.exe"

Write-Host "Downloading Prometheus 3.13.0..."
Invoke-WebRequest -Uri "https://github.com/prometheus/prometheus/releases/download/v3.13.0/prometheus-3.13.0.windows-amd64.zip" -OutFile "prom.zip"
Expand-Archive -Path "prom.zip" -DestinationPath "prom-extract" -Force
Copy-Item "prom-extract\prometheus-3.13.0.windows-amd64\promtool.exe" -Destination "promtool.exe"
Remove-Item -Recurse -Force "prom-extract", "prom.zip"

Write-Host "Verifying..."
.\sloth.exe version
.\promtool.exe --version
Write-Host "Done. Move sloth.exe and promtool.exe to a directory on your PATH."

Save starter/fetch.ps1.

What does this script do?

On macOS/Linux, the platform-detection block picks the correct binary name for your OS and architecture. The download block fetches the Sloth binary and the Prometheus archive, extracts promtool, and runs a quick verification.

On Windows, the PowerShell script does the same thing in a single file. Total download is about 143 MB across both tools.

• Confirm all five starter files exist by running:

ls starter/

You should see your fetch script (fetch.sh or fetch.ps1), reference-table.md, series-outage.txt, series-slowleak.txt, and worked-example.md.

• Stage everything and make the initial commit by running:

macOS

git add -A && git commit -m "init: starter folder, .gitattributes, task checklist"

Windows

git add -A ; git commit -m "init: starter folder, .gitattributes, task checklist"

• Verify the commit by running:

git log --oneline

You should see a single commit with the message init: starter folder, .gitattributes, task checklist.

Nothing to commit?

Make sure you saved every file before running git add -A. Unsaved files in your text editor will not be picked up by git.

Help me debug my initial git commit.

Download and verify both CLIs

Sloth generates Prometheus recording and alert rules from a single spec file. promtool runs unit tests against those rules at simulated evaluation times. Both are standalone binaries with no installer.

• Run the fetch script for your platform by running one of these commands:

macOS

bash starter/fetch.sh

The script detects whether you are on an ARM or Intel Mac and downloads the correct binaries. On Linux, use the same command.

Windows

.\starter\fetch.ps1

If PowerShell blocks the script, run this command first, then re-run the fetch command above:

Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass

Prefer to download manually?

Sloth v0.16.0: https://github.com/slok/sloth/releases/tag/v0.16.0. Download the binary for your platform, rename it to sloth (or sloth.exe on Windows), and place it on PATH.

Prometheus 3.13.0: https://github.com/prometheus/prometheus/releases/tag/v3.13.0. Download the archive (.tar.gz on macOS/Linux, .zip on Windows), extract promtool, and place it on PATH.

• Move the binaries onto your PATH so later steps can call them without a path prefix:

macOS

sudo mv sloth promtool /usr/local/bin/

On Linux, use the same command.

Windows

Add the project folder to your session PATH by running:

$env:Path = "$PWD;$env:Path"

This lasts for the current PowerShell session. If you close and reopen PowerShell, run this line again from the error-budget-proof folder.

Before you run the version checks, what version numbers do you expect to see from each tool?

• Verify Sloth by running:

sloth version

✔️ I see v0.16.0

Sloth is ready. Move on to the promtool check below.

ⓧ I see a different version

Your PATH may be picking up a different Sloth binary. Re-download the v0.16.0 binary from the release page and make sure it is the one on your PATH.

Help me get the right Sloth version.

ⓧ Command not found

The binary is not on your PATH. On macOS or Linux, confirm you ran the sudo mv command above. On Windows, confirm you ran the $env:Path line in the same PowerShell session.

If you downloaded manually, double-check that the binary is renamed to exactly sloth (not sloth-darwin-arm64) and that the directory you placed it in is on your PATH.

Help me get sloth on my PATH.

• Verify promtool by running:

promtool --version

✔️ I see 3.13.0

Both CLIs are verified. Move on to tick the setup box below.

ⓧ I see a different version

Your PATH may be picking up a different promtool binary. Re-download the Prometheus 3.13.0 archive from the release page, extract promtool, and make sure it is the one on your PATH.

The version floor matters. On an older archive the burn-rate assertions fail on a floating-point tail rather than on a wrong threshold, which is the most confusing way a correct answer can look wrong.

Help me get the right promtool version.

ⓧ Command not found

The binary is not on your PATH. On macOS or Linux, confirm you ran the sudo mv command above. On Windows, confirm you ran the $env:Path line in the same PowerShell session.

If you extracted manually, confirm the file is named exactly promtool (or promtool.exe on Windows) and that its directory is on your PATH.

Help me get promtool on my PATH.

• Open TASKS.md in your text editor and find this line:

- [ ] Setup: init repo, download and verify both CLIs

• Change [ ] to [x] so the line looks like this:

- [x] Setup: init repo, download and verify both CLIs

• Save TASKS.md.

Your repo is initialized, the starter folder is committed, and both CLIs are verified. Next up, you will prompt your AI to draft the design decisions and argue with every denominator.

AI-Draft the Design and Argue Every Denominator

Your repo is initialized and both CLIs are verified. Now it is time to design the system before writing a single rule.

Sloth can generate alert rules from a spec, but the generated thresholds are only as trustworthy as the indicator beneath them.

In this step, your AI drafts three design decisions and three candidate service level indicators. Your job is to reject every denominator you cannot defend.

In this step, get ready to:

• Prompt your AI to draft three MADR v4 decision records and three candidate SLIs.
• Review each SLI denominator, reject at least one, and write a fourth decision record yourself.
• Draft the requirements brief, component view, and execution plan, then commit.

Prompt the AI for three decision records and three candidate SLIs

Each design decision will be captured as a formal MADR v4 record: a title, context and problem statement, considered options, and a decision outcome with consequences. Your AI drafts the first three records and proposes three candidate SLIs for you to evaluate.

• Create the directory for your decision records by running:

mkdir -p docs/decisions

On Windows PowerShell

Drop the -p flag. PowerShell creates parent directories automatically, so mkdir docs/decisions is all you need.

• Switch to your AI chat client.
• Copy and paste this prompt into your AI chat client:

You are helping me design an error budget alerting system for a service called bordercontrol.

Context:
- An SLI (Service Level Indicator) is good events divided by valid events, where a valid event is one the service was meant to serve.
- An error budget is 1 minus the SLO, the unreliability the service is allowed.
- Burn rate is how fast the error budget is consumed, where 1 means it runs out exactly at the window edge.
- MWMB (Multi-Window, Multi-Burn-rate) alerting pairs a long window that confirms a burn is real with a short window that confirms it is still happening.
- The two-person on-call roster constraint: the roster is too small to absorb a bad night.

Requests:
1. Draft three MADR v4 decision records as separate files saved as docs/decisions/ADR-0001.md, docs/decisions/ADR-0002.md, and docs/decisions/ADR-0003.md. Each record should have a title, context and problem statement, considered options, and a decision outcome with consequences.
2. Propose three candidate SLIs in a single file saved as docs/sli-review.md. Each SLI should define error_query and total_query with the denominator argued: what counts as a valid event and what is excluded (health checks, cancelled requests, synthetic probes).

What should you expect back?

Your AI should return two things. Check that both are present before moving on.

• Three MADR v4 decision records, each with a title, context and problem statement, considered options, and a decision outcome with a consequences subsection.
• Three candidate SLIs, each defining error_query and total_query with a written argument for the denominator: what counts as a valid event and what is explicitly excluded (health checks, cancelled requests, synthetic probes).

💡 What should the three decision records cover?

Each record follows the MADR v4 template: title, context and problem statement, considered options, and a decision outcome with a consequences subsection.

• ADR-0001: proxy request metrics vs application-level instrumentation. Tradeoff: an error returning HTTP 200 stays invisible. Reversal trigger: first invisible-200 incident.
• ADR-0002: generated rules via Sloth vs hand-written PromQL. Tradeoff: the generator owns the rule shape. Reversal trigger: first rule Sloth cannot express.
• ADR-0003: simulated-time proof via promtool vs live-window observation. Tradeoff: nothing was scraped, but the container engine is deleted, dropping setup from rung 3 to rung 1. Reversal trigger: regulatory requirement for live-window observation.

• Save each decision record as ADR-0001.md, ADR-0002.md, and ADR-0003.md inside docs/decisions/.
• Save the three candidate SLIs into a new file at docs/sli-review.md.

• Confirm your files exist by running:

ls docs/decisions/

You should see ADR-0001.md, ADR-0002.md, and ADR-0003.md listed.

Directory not found or files missing?

• Make sure you ran mkdir from inside your error-budget-proof/ folder, not from your Desktop.
• Check that each file was saved to docs/decisions/ and not to the repo root.

Why are my decision record files not showing up?

Review the SLIs, reject at least one, and write ADR-0004

The denominator of each SLI defines what counts as a valid event. This is the judgment this role is paid for.

• Open docs/sli-review.md in your text editor.
• For each candidate SLI, examine the total_query denominator: does it include health-check traffic that inflates the total? Does it exclude legitimate error types the service should count?
• Write a rejection for at least one SLI, naming exactly why the denominator is wrong.
• Mark the surviving SLI as your chosen indicator.
• Save docs/sli-review.md.

Why does the denominator matter so much?

The denominator determines what the indicator measures. A total_query that includes synthetic health-check traffic inflates the denominator and hides real user errors behind passing probes.

A denominator that excludes cancelled requests might look generous until a bug cancels requests the service was meant to serve. Every exclusion is a claim about what the service owes its users.

• Create docs/decisions/ADR-0004.md in your text editor.
• Write this record yourself. Do not have the AI draft it.

What goes in ADR-0004?

The decision: native Sloth prometheus/v1 format vs OpenSLO.

• Decision outcome: native prometheus/v1 format. It is the recommended, most feature-complete path with SLI plugin support and full label control.
• Tradeoff: vendor lock-in to Sloth's spec shape.
• Reversal trigger: the team adopts a multi-vendor SLO platform that requires OpenSLO as input.

• Confirm all four records exist by running:

ls docs/decisions/

You should see ADR-0001.md through ADR-0004.md.

Draft the remaining docs, commit, and verify

The design phase produces a few more supporting documents before you lock it down with a commit. The component view is the blueprint that the execution plan is drawn from.

• Switch to your AI chat client.
• Copy and paste this prompt into your AI chat client:

Using the same bordercontrol error budget alerting context from our earlier conversation:

Requests:
1. Draft a requirements brief saved as docs/requirements-brief.md. Summarise the error budget proof project: the service under observation, the SLI chosen, the SLO target, the alerting approach (MWMB), and the two-person on-call constraint.
2. Draft a component view saved as docs/component-view.md. Show the data flow through the system: Sloth spec goes into sloth generate, which produces rules YAML, which goes into promtool test rules, which outputs pass or fail.

What should you expect back?

Your AI should return two files. Check that both are present and saved to the correct locations before moving on.

• A requirements brief saved as docs/requirements-brief.md. It should summarise the service under observation, the chosen SLI, the SLO target, the MWMB alerting approach, and the two-person on-call constraint.
• A component view saved as docs/component-view.md. It should show the data flow: Sloth spec goes into sloth generate, which produces rules YAML, which goes into promtool test rules, which outputs pass or fail.

💡 What does the component view show?

The data flow is a single pipeline: Sloth spec goes into sloth generate, which produces rules YAML, which goes into promtool test rules, which outputs pass or fail. The diagram precedes the plan: if they disagree about which pieces exist, fix the one that is wrong.

• Confirm both files were saved by running:

ls docs/

You should see component-view.md, decisions/, requirements-brief.md, and sli-review.md.

• Write docs/plan.md drawn from the component view: five steps budgeted 5, 5, 5, 5, and 10 minutes, summing to 30 minutes of build.
• In TASKS.md, tick the design checkbox by changing [ ] to [x] on the Design line.

Before you check the log, how many commits do you expect to see at this point?

• Commit your design work by running:

macOS

git add -A && git commit -m "design: four ADRs, SLI review, requirements brief, component view, plan"

Windows

git add -A ; git commit -m "design: four ADRs, SLI review, requirements brief, component view, plan"

• Verify the commit by running:

git log --oneline

You should see two commits: the initial commit from Step 1 and the design commit you just made. The most recent commit message reads design: four ADRs, SLI review, requirements brief, component view, plan.

Commit failed or nothing to commit?

• If git says nothing to commit, confirm your files are inside the error-budget-proof/ repo folder, not in a parent or sibling directory.
• If you see a line-ending warning, the .gitattributes file from Step 1 handles this. Confirm it contains * text=auto eol=lf.

Why is my git commit failing?

Your design is locked in: four decision records, a reviewed SLI, and a plan drawn from the component view. Next up, you will write the Sloth spec, generate the rules, and hand-derive every threshold to check them.

Write the Sloth Spec, Generate the Rules, and Hand-Derive Every Threshold

Your design is complete. Four decisions are recorded, a chosen SLI has its denominator argued, and the five-step plan is committed. Now the build begins.

Sloth encodes your SLI and SLO into a single source file, then generates recording rules, metadata rules, and both MWMB alert rules from it. The generated output is the claim. Your hand-worked derivations are the check. Until you place them side by side, you have not verified anything.

In this step, get ready to:

• Write a Sloth spec encoding the approved SLI and SLO with the denominator argued in a comment.
• Generate the MWMB alert rules and inspect the output.
• Hand-derive four burn-rate thresholds and compare them to the generated values.

Write the Sloth spec

The Sloth spec is a single YAML file that declares your service, its SLO objective, and the queries that define good versus total events. The key judgment call lives in total_query: what counts as a valid event, and what gets excluded.

• Create a specs/ folder by running this command:

mkdir specs

• Create a file called slo.yml inside specs/ and add the service definition and SLI events:

version: "prometheus/v1"
service: "bordercontrol"
labels:
  owner: "on-call-team"
  roster_size: "2"
slos:
  - name: "http-availability"
    objective: 99.9
    description: "Proxy-metered HTTP availability for bordercontrol."
    labels:
      category: availability
    sli:
      events:
        error_query: sum(rate(http_request_duration_seconds_count{job="bordercontrol",code=~"(5..|429)"}[{{.window}}]))
        # Denominator: all HTTP responses with a real status code.
        # Excludes health-check traffic (code="0" or empty from
        # synthetic probes) because those inflate the total and
        # hide real user errors. Accepts the tradeoff that an
        # application error returning HTTP 200 stays invisible
        # to this indicator. See ADR-0001.
        total_query: sum(rate(http_request_duration_seconds_count{job="bordercontrol",code!="",code!~"^0.*"}[{{.window}}]))

What does this spec define?

version: "prometheus/v1" tells Sloth to use its native, most feature-complete format. This is the format you chose in ADR-0004.

objective: 99.9 sets the SLO to 99.9% availability over the default 30-day window. The error budget is 1 - 0.999 = 0.001, or 43 minutes and 12 seconds.

error_query counts 5xx and 429 responses. total_query counts all responses with a real status code, excluding health-check traffic. The YAML comment above total_query argues that denominator and references ADR-0001. {{.window}} is a placeholder that Sloth replaces with the actual time window at generation.

• Save specs/slo.yml.
• Confirm the file exists by running:

ls specs/

You should see slo.yml in the output.

• Return to specs/slo.yml and add the alerting configuration below the total_query line, at the same indent level as sli::

    alerting:
      name: "BordercontrolHighErrorRate"
      labels:
        category: "availability"
      annotations:
        summary: "High error rate on bordercontrol HTTP responses"
      page_alert:
        labels:
          severity: "page"
          routing_key: "on-call-team"
      ticket_alert:
        labels:
          severity: "ticket"
          slack_channel: "#alerts-bordercontrol"

What does the alerting block do?

name: "BordercontrolHighErrorRate" is shared by both alert rules. Sloth adds a sloth_severity label to distinguish them.

page_alert fires for fast burns (14.4x and 6x) and routes to on-call-team. ticket_alert fires for slow burns (3x and 1x) and routes to #alerts-bordercontrol. Two alert rules, one shared name, separated by a label.

• Save specs/slo.yml.

Sloth rejecting the spec?

CRLF line endings cause parsing errors on Windows. The .gitattributes file in your repo pins all tracked files to LF, but files created outside git may still have CRLF. Confirm your editor is saving with LF line endings.

Check that {{.window}} uses double curly braces, not single. Sloth needs Go template syntax to substitute the window value.

Help me debug a Sloth spec parsing error.

✔️ Awesome, I've got everything!

Great. Double check you have saved specs/slo.yml.

ⓧ I'd like to double check the full code

Your specs/slo.yml has two sections. Scroll up and confirm both are present in order:

• The service definition and SLI events, starting with version: "prometheus/v1" and ending with the total_query line.
• The alerting configuration, starting with alerting: and ending with slack_channel: "#alerts-bordercontrol".

The combined file should have 35 lines. The alerting: key sits at the same indentation level as sli: (four spaces in from the list marker).

Generate the MWMB alert rules

The spec is the input. sloth generate reads it and produces recording rules, metadata rules, and both MWMB alert rules in a single output file.

• Create the rules/ folder by running:

mkdir rules

Before you run the generator, predict: with a 99.9% SLO and page plus ticket tiers, how many alert rules will Sloth produce, and what should distinguish them?

• Generate the rules by running:

sloth generate -i specs/slo.yml -o rules/slo-rules.yml

• Open rules/slo-rules.yml in your text editor.

You should see two alert rules sharing the name BordercontrolHighErrorRate. One has sloth_severity: page and the other has sloth_severity: ticket.

The page alert combines a 5m and 1h window at burn rate 14.4 with a 30m and 6h window at burn rate 6. The ticket alert combines a 2h and 1d window at burn rate 3 with a 6h and 3d window at burn rate 1.

What about the error budget value?

The error budget for a 99.9% SLO is 1 - 0.999 = 0.001. In the generated file, this renders as 0.0009999999999999432 due to float64 representation.

This is not a bug. It is within one ULP (unit in the last place) of the exact value. The promtool test in the next step uses fuzzy_compare: true to handle this.

🙋‍♀️ Seeing an error from sloth generate?

If Sloth says the spec is invalid, check that your YAML indentation uses spaces only, not tabs. Every nested level is two spaces.

Confirm {{.window}} appears in both error_query and total_query with double curly braces.

Help me fix a sloth generate error.

Hand-derive every threshold and commit

The generator produced thresholds. You do not yet know if they are correct. The worked example in starter/worked-example.md shows how to derive one row. Now you will derive all four and place each beside the value the generator emitted.

• Create docs/threshold-derivations.md with the following content:

# Threshold Derivations

SLO: 99.9% over 30 days
Error budget: 1 - 0.999 = 0.001
Budget in minutes: 30 * 24 * 60 * 0.001 = 43.2 minutes
Period in hours: 30 * 24 = 720 hours

Each row: burn_rate = budget_share * (period / window)

| Budget share | Window  | Burn rate (derived)   | Burn rate (generated) | Notification |
|------------- |-------- |---------------------- |---------------------- |------------- |
| 2% (0.02)    | 1 hour  | 0.02 * 720/1 = 14.4   | 14.4                  | Page         |
| 5% (0.05)    | 6 hours | 0.05 * 720/6 = 6.0    | 6                     | Page         |
| 10% (0.10)   | 1 day   | 0.10 * 720/24 = 3.0   | 3                     | Ticket       |
| 10% (0.10)   | 3 days  | 0.10 * 720/72 = 1.0   | 1                     | Ticket       |

Multi-window pairs (long + short):
| Burn rate | Long window | Short window |
|---------- |------------ |------------- |
| 14.4      | 1 hour      | 5 minutes    |
| 6         | 6 hours     | 30 minutes   |
| 3         | 1 day       | 2 hours      |
| 1         | 3 days      | 6 hours      |

Twelve generated values checked: 4 burn-rate factors + 8 windows.
Count of contradictions: ___ / 12 (fill after comparing to generated output)

What is this derivation doing?

Each row applies the formula: burn_rate = budget_share * (period / window). The first row asks: how fast does the budget burn if 2% disappears in one hour? That is 0.02 * 720 = 14.4.

The "Burn rate (generated)" column is what you read from rules/slo-rules.yml. The "Burn rate (derived)" column is what you calculated. If any derived value disagrees with the generated value, you have found a contradiction.

• Save docs/threshold-derivations.md.
• Copy the full contents of both docs/threshold-derivations.md and rules/slo-rules.yml.
• Switch to your AI chat client, paste both files, and add this prompt:

Compare the derivation table in docs/threshold-derivations.md against the generated alert rules in rules/slo-rules.yml. Confirm all 12 values match (4 burn-rate factors + 8 windows). Report any contradictions.

• Update the contradiction count at the bottom of docs/threshold-derivations.md with the number your AI reported.
• Save docs/threshold-derivations.md.

• Tick the Generate box in TASKS.md by changing - [ ] Generate: to - [x] Generate:.
• Commit your work by running:

macOS

git add -A && git commit -m "generate: Sloth spec, generated rules, threshold derivations"

Windows

git add -A ; git commit -m "generate: Sloth spec, generated rules, threshold derivations"

Before you run this, predict: will git accept the commit cleanly, or will you see a problem? If any file has unstaged changes or a line-ending mismatch, git will tell you.

• Run the commit command above.
• Verify the commit landed by running:

git log --oneline -1

You should see a commit with the message generate: Sloth spec, generated rules, threshold derivations.

Commit rejected or unexpected diff?

If git status shows files with line-ending changes you did not make, your editor may be overriding the .gitattributes LF setting. Change your editor's default line ending to LF, re-save the affected files, and commit again.

Help me fix a git commit issue in my error budget project.

Your rules are generated and the arithmetic is verified. The claim has a check beside it. Next, you will write promtool tests that prove these rules actually fire where the math says they should.

Write promtool Tests and Prove the Rules Fire

You have generated rules and hand-derived every threshold. The derivation table and the generator agree on four burn rates and eight windows.

But agreement on paper is a claim, not a proof. Until you inject counter series at known error rates and assert the alerts fire at the right simulated times, nobody can point at the output and say "the arithmetic holds." In this step you write that proof, and the whole suite answers in under five seconds.

In this step, get ready to:

• Write the outage scenario test (10% error ratio, burn rate 100).
• Write the slow-leak scenario test (0.2% error ratio, burn rate 2).
• Run the test suite and confirm both scenarios pass.

Create the test file and feed the outage scenario

The promtool test runner reads a YAML file that points at the generated rules, defines synthetic counter series, and asserts which alerts fire at which simulated evaluation times. The file header also enables fuzzy_compare, a Prometheus 3.13.0 enhancement that allows the float64 thresholds to match the hand-derived values without failing on the last mantissa bit.

• Create tests/slo_test.yml in your project folder and paste the file header along with the Scenario 1 input series:

rule_files:
  - ../rules/slo-rules.yml

evaluation_interval: 1m

fuzzy_compare: true

tests:
  # Scenario 1: 10% sustained error ratio (burn rate 100).
  # 900 good + 100 failing per minute.
  # At eval_time 1h both page and ticket fire.
  # A severe outage trips both; the page simply arrives first.
  - interval: 1m
    input_series:
      - series: 'http_request_duration_seconds_count{job="bordercontrol",code="200"}'
        values: '0+900x120'
      - series: 'http_request_duration_seconds_count{job="bordercontrol",code="500"}'
        values: '0+100x120'

What does this code do?

rule_files tells promtool to load the generated rules from ../rules/slo-rules.yml. The path is relative because the test file lives in tests/ and the rules live in rules/ at the project root.

fuzzy_compare: true weakens floating-point comparisons so differences in the last bit of the mantissa do not fail the test. Without it, the generated error budget value 0.0009999999999999432 could mismatch against the hand-derived 0.001.

The two input_series entries come from starter/series-outage.txt: 900 good requests and 100 failing requests per minute, sustained for 120 evaluation steps. That is a 10% error ratio, giving a burn rate of 0.1 / 0.001 = 100.

The input series are the scenario. Now you need the assertions that prove the alerts fire where the arithmetic says they should.

• In tests/slo_test.yml, add the following directly below the last values: '0+100x120' line, at the same indentation level as input_series:

    # Same alertname fires twice (page and ticket). One eval must expect both.
    alert_rule_test:
      - eval_time: 60m
        alertname: BordercontrolHighErrorRate
        exp_alerts:
          - exp_labels:
              alertname: BordercontrolHighErrorRate
              category: availability
              owner: on-call-team
              roster_size: "2"
              routing_key: on-call-team
              severity: page
              sloth_id: bordercontrol-http-availability
              sloth_service: bordercontrol
              sloth_severity: page
              sloth_slo: http-availability
            exp_annotations:
              summary: High error rate on bordercontrol HTTP responses
              title: (page) bordercontrol http-availability SLO error budget burn rate is too fast.
          - exp_labels:
              alertname: BordercontrolHighErrorRate
              category: availability
              owner: on-call-team
              roster_size: "2"
              severity: ticket
              slack_channel: "#alerts-bordercontrol"
              sloth_id: bordercontrol-http-availability
              sloth_service: bordercontrol
              sloth_severity: ticket
              sloth_slo: http-availability
            exp_annotations:
              summary: High error rate on bordercontrol HTTP responses
              title: (ticket) bordercontrol http-availability SLO error budget burn rate is too fast.
    promql_expr_test:
      - expr: max(slo:sli_error:ratio_rate5m{sloth_id="bordercontrol-http-availability"}) / 0.001
        eval_time: 60m
        exp_samples:
          - value: 100

What do these assertions prove?

The two alert_rule_test entries both check eval_time: 60m. The first asserts the page alert fires with sloth_severity: page. The second asserts the ticket alert fires alongside it with sloth_severity: ticket. This is a claim about order, not exclusion: a severe outage trips both, and the page simply arrives first.

The promql_expr_test divides the 5-minute error ratio recording rule by the error budget (0.001) and asserts the result equals 100. This reproduces the hand-derived burn rate from the generated metric itself.

• Save tests/slo_test.yml.
• Run the test suite with Scenario 1 alone by running this command:

promtool test rules tests/slo_test.yml

You should see output ending with SUCCESS. The 10% outage scenario passes: both page and ticket fire at 1 hour.

Seeing a float mismatch or FAILED?

• Confirm fuzzy_compare: true is at the top level of your test file, not nested inside a test group. Without it, the float64 error budget causes threshold mismatches.
• Check the indentation of alert_rule_test and promql_expr_test. Both must sit at the same level as input_series (four spaces in from the list marker).
• Verify the label values (routing_key, slack_channel) match exactly what the generated rules/slo-rules.yml contains. Open both files side by side to compare.

Help me debug a promtool test failure in my SLO unit tests

Add the slow-leak scenario and run the full suite

Scenario 1 proved a severe outage trips both alerts. Scenario 2 is the negative test and the lesson in one assertion: a burn slow enough to take weeks raises a ticket and wakes nobody.

• In tests/slo_test.yml, add the Scenario 2 test group below the Scenario 1 promql_expr_test block, with a blank line between them:

  # Scenario 2: 0.2% sustained error ratio (burn rate 2).
  # 998 good + 2 failing per minute.
  # At eval_time 6h the ticket fires alone; the page has zero instances.
  # Prove that by expecting exactly one alert (ticket). A second page instance would fail this list.
  # A burn slow enough to take weeks raises a ticket and wakes nobody.
  - interval: 1m
    input_series:
      - series: 'http_request_duration_seconds_count{job="bordercontrol",code="200"}'
        values: '0+998x420'
      - series: 'http_request_duration_seconds_count{job="bordercontrol",code="500"}'
        values: '0+2x420'
    alert_rule_test:
      - eval_time: 360m
        alertname: BordercontrolHighErrorRate
        exp_alerts:
          - exp_labels:
              alertname: BordercontrolHighErrorRate
              category: availability
              owner: on-call-team
              roster_size: "2"
              severity: ticket
              slack_channel: "#alerts-bordercontrol"
              sloth_id: bordercontrol-http-availability
              sloth_service: bordercontrol
              sloth_severity: ticket
              sloth_slo: http-availability
            exp_annotations:
              summary: High error rate on bordercontrol HTTP responses
              title: (ticket) bordercontrol http-availability SLO error budget burn rate is too fast.
    promql_expr_test:
      - expr: max(slo:sli_error:ratio_rate5m{sloth_id="bordercontrol-http-availability"}) / 0.001
        eval_time: 360m
        exp_samples:
          - value: 2

What does Scenario 2 prove?

The input series come from starter/series-slowleak.txt: 998 good and 2 failing requests per minute. That is a 0.2% error ratio, giving a burn rate of 0.002 / 0.001 = 2. The series run for 420 steps (7 hours of simulated time) so the 6-hour evaluation point has data.

At eval_time: 360m (6 hours), the assertion expects exactly one alert instance: the ticket. If a page alert also fired, promtool would fail this list because the actual alerts would not match the expected set. The promql_expr_test confirms burn rate equals 2. Together these prove the separation: a slow burn that would take weeks to exhaust the budget raises a ticket and wakes nobody.

• Save tests/slo_test.yml.

Before you run the full suite, predict: will both scenarios pass? Scenario 1 passed on its own. Do you expect Scenario 2 to change anything about Scenario 1's result?

• Run the full test suite by running this command:

promtool test rules tests/slo_test.yml

You should see SUCCESS. Both test groups pass in under five seconds. The 10% outage fires both page and ticket at 1 hour. The 0.2% slow leak fires ticket alone at 6 hours with page at zero instances. Two expression assertions reproduce burn rates 100 and 2.

Scenario 2 failing while Scenario 1 passes?

• Make sure the Scenario 2 test group starts at the same indentation level as Scenario 1 (two spaces before the - list marker). YAML lists are whitespace-sensitive.
• Check that the series values use 0+998x420 and 0+2x420 (420 steps, not 120). The 6-hour evaluation point needs at least 360 minutes of data.
• Confirm eval_time: 360m (not 60m) appears in all three Scenario 2 assertions.

Help me debug a promtool Scenario 2 failure for the slow-leak test

✔️ Awesome, I've got everything!

Great. Confirm you have saved tests/slo_test.yml and that the last run printed SUCCESS.

ⓧ I'd like to double check the full code

The full file is longer than 30 lines, so scroll up through your tests/slo_test.yml and confirm each section is present, in order:

• File header: rule_files pointing to ../rules/slo-rules.yml, evaluation_interval: 1m, fuzzy_compare: true.
• Scenario 1: input_series with 0+900x120 and 0+100x120, two alert_rule_test entries at eval_time: 60m (page and ticket), and promql_expr_test asserting value 100.
• Scenario 2: input_series with 0+998x420 and 0+2x420, alert_rule_test at eval_time: 360m (page empty, ticket fires), and promql_expr_test asserting value 2.

Commit the proof and tick the checklist

• Open TASKS.md and tick the Test box by changing - [ ] to - [x] on the Test line.
• Save TASKS.md.
• Commit everything by running these commands:

macOS

git add -A && git commit -m "proof: promtool tests pass, burn rates 100 and 2 proven"

Windows

git add -A ; git commit -m "proof: promtool tests pass, burn rates 100 and 2 proven"

• Confirm the commit landed by running:

git log --oneline -1

You should see a commit with the message proof: promtool tests pass, burn rates 100 and 2 proven.

The arithmetic is no longer a claim. Two scenarios, two burn rates, two different alert severities, all proven from one command in under five seconds. Next up, you will build the stakeholder readout that puts this proof in front of a reviewer.

Build the Stakeholder Readout, Write the README, and Close the Repo

Your rules are proven and every threshold is derived. A reviewer who arrives tomorrow, though, sees YAML files and terminal output that nobody outside the project can follow.

The readout brings all of it into one place. In this step, you will build a self-contained HTML page that a stakeholder can open by double-clicking, write the README so the next person can run everything from scratch, and close the repo with a clean tree and a tag.

In this step, get ready to:

• Build the stakeholder readout as a fully self-contained HTML page.
• Write the README with the version floor, verification commands, and platform note.
• Close the repo with every box ticked, a clean working tree, and a v1.0.0 tag.

Have the AI draft the stakeholder readout

The proof lives in test output and derivation tables scattered across your repo. A stakeholder needs all of it on one page they can open without installing anything.

• Switch to your AI chat client.
• Copy and paste this prompt into your AI chat client:

Draft a file called readout.html with the following requirements:

1. Fully self-contained: every style in a <style> block or inline. No external CSS, JavaScript, or images.
2. An SLO section stating 99.9% HTTP availability over a rolling 30-day window with an error budget of 43 minutes 12 seconds.
3. A four-row burn rate derivation table with columns for budget share, window, burn rate, and notification type (14.4 page, 6 page, 3 ticket, 1 ticket).
4. A test result summary: the 10% error scenario fires both page and ticket at 1 hour. The 0.2% error scenario fires ticket alone at 6 hours with page at zero instances.
5. One-sentence summaries of all four decision records (ADR-0001 through ADR-0004) with their reversal triggers.
6. A scope statement explaining what the proof shows and what it does not show. Include the text: "This proves the rule logic and the arithmetic behind it. It does not prove the production pipeline, because nothing was scraped and no server ran."

What should you expect back?

Your AI should return a single HTML file. Check that it includes all six elements before saving.

• All styles inside a <style> block or inline, with zero external references.
• The SLO section, the four-row burn-rate table, the test result summary, the four ADR summaries with reversal triggers, and the scope statement.

💡 Why fully self-contained?

The readout must work on any machine without a network connection. That means no <link> to a CDN, no <script src="...">, and no <img> that fetches from a URL.

If someone copies the file to a USB stick and opens it on an air-gapped laptop, it should render perfectly.

• Save the AI's output as readout.html in your project root.
• Double-click readout.html to open it in your browser.

You should see the SLO, the four-row derivation table, the proof results with both test scenarios, all four decision summaries with reversal triggers, and the scope statement.

• Open DevTools by pressing F12 (Windows/Linux) or Cmd+Option+I (macOS).
• Click the Network tab, then refresh the page.
• Confirm zero network requests appear in the list.

Page looks broken or styles are missing?

• Make sure the <style> block is inside the <head>. A style block outside the head can render inconsistently across browsers.
• Check that no <link> or <script src> tags reference external URLs.
• If the Network tab shows outbound requests, ask the AI to revise the file with all styles inline.
• Why isn't my readout.html rendering correctly?

Write the README and tick the last box

The README tells the next person everything they need to run the project and understand why each decision was made. Without it, your verified artifacts sit in a folder that only you can navigate.

• Switch to your AI chat client.
• Copy and paste this prompt into your AI chat client:

Draft a file saved as README.md for the error-budget-proof project. Include these five sections:

1. A one-sentence summary describing the project.
2. A version floor section containing this exact statement: "promtool from Prometheus 3.13.0 or newer; on an older archive the burn-rate assertions fail on a floating-point tail."
3. The two verification commands:
   sloth generate -i specs/slo.yml -o rules/slo-rules.yml
   promtool test rules tests/slo_test.yml
4. A platform note containing this exact statement: "Authored on Windows 11 with WSL2; macOS and Linux reasoned from the same published binaries."
5. A directory tree showing the full project structure (all folders and files).

What should you expect back?

Your AI should return a single markdown file saved as README.md in the project root. Check that all five sections are present before saving.

• A one-sentence project summary.
• The version floor statement naming Prometheus 3.13.0.
• Both verification commands (sloth generate and promtool test rules).
• The platform note about Windows 11 with WSL2.
• A directory tree of the full project structure.

💡 Why state the version floor?

On an older promtool the burn-rate assertions fail on a floating-point tail rather than on a wrong threshold. That is the most confusing way a correct answer can look wrong.

Stating the floor in the README saves the next person from debugging a phantom mismatch.

• Save README.md.

Now tick the last checkbox in your task checklist. In TASKS.md, find this line:

- [ ] Readout: stakeholder HTML, README, all boxes ticked, working tree clean, tagged v1.0.0

• Change - [ ] to - [x] so the line looks like this:

- [x] Readout: stakeholder HTML, README, all boxes ticked, working tree clean, tagged v1.0.0

• Save TASKS.md.
• Scroll through TASKS.md and confirm every box is now ticked.

Close the repo

Completion is not "it runs." Every box is ticked, the working tree is clean, and the last commit is tagged so the exact version is recoverable.

Before you commit, what do you think git status will show right now?

• Check the state of your working tree by running:

git status

You should see readout.html and README.md as new files, and TASKS.md as modified.

• Stage and commit everything by running these commands:

macOS

git add -A && git commit -m "close: readout, README, all tasks complete"

Windows

git add -A ; git commit -m "close: readout, README, all tasks complete"

What does this commit message tell the next person?

The close: prefix signals this is the final commit. The message names exactly what landed: the readout, the README, and the completion of all tasks.

• Tag the commit so the exact version is recoverable by running:

git tag v1.0.0

What do you expect to see when you check the tag list and the working tree status?

• Verify the tag exists by running:

git tag

• Confirm the working tree is clean by running:

git status

git tag shows v1.0.0. git status shows nothing to commit and a clean working tree. The exact version is now recoverable.

Working tree not clean?

• Check if you have unsaved changes in your editor. Save all files, then run git add -A and git commit --amend --no-edit to fold the changes into your last commit.
• Run git diff to see exactly what changed.
• Why does git status show uncommitted changes after my final commit?

Secret mission

Write the Teach-Back, the After-Action Review, and Score the AI

The validated system, the readout, and the proof are done. Two signature artifacts remain: a teach-back page that explains burn rate to someone who has never carried a pager, and an after-action review written while the frustration is still fresh. You also score the AI on how many of its drafts you had to reject.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely on your machine with no cloud resources or paid services, so there are no ongoing costs. Decide whether to keep your project folder and CLI tools or remove them entirely.

Resources you used:

• Local project folder error-budget-proof/ on your Desktop.
• Sloth v0.16.0 binary on your PATH.
• promtool binary from Prometheus 3.13.0 on your PATH.

Keep everything running

No action needed. Choose this if you want to keep experimenting with new SLO specs, adding test scenarios, or revisiting your decision records.

• Your project folder stays on your Desktop with all artifacts intact.
• The sloth and promtool binaries remain on your PATH for future use.

Pause - I'll come back to this later

There are no running processes, servers, or cloud resources to pause. Everything lives as files on disk.

• Your project folder and both CLI binaries stay where they are until you decide to remove them.
• To pick up where you left off, open a terminal in your error-budget-proof folder and run your verification commands.

Delete - I don't want to use this again

Remove all project files and the downloaded CLI binaries.

• Delete the project folder. On macOS or Linux, run:

rm -rf ~/Desktop/error-budget-proof

• On Windows, open PowerShell and run:

Remove-Item -Recurse -Force ~\Desktop\error-budget-proof

• Remove the sloth and promtool binaries you placed on your PATH. On macOS or Linux, find them and delete:

rm "$(which sloth)" "$(which promtool)"

• On Windows, locate the binaries first:

where.exe sloth.exe
where.exe promtool.exe

• Delete both .exe files from the folder shown in the output.

Nice Work!

Nice Work!

You just built, proved, and documented an error budget alerting system from first principles. Every threshold is derived, every rule is tested, every decision is recorded, and the whole thing runs in under five seconds from one command on a clean copy.

You've learned how to:

• Build a validated working system from a single Sloth spec, one generated rule file, and one promtool test suite that proves MWMB alert rules fire where the arithmetic says they should.
• Hand-derive four burn-rate thresholds (14.4, 6, 3, 1) and prove alert order: a 10% error ratio fires both page and ticket at 1 hour, while a 0.2% error ratio fires ticket alone at 6 hours and wakes nobody.
• Write four MADR v4 decision records, build a stakeholder readout that opens by double-clicking with no network dependency, and state what the proof shows and does not show.
• Secret Mission: Write the teach-back page, the after-action review before closing the folder, and score the AI on indicator rejections and threshold contradictions.

Ready to quiz yourself?