Build a Two-Host SLO Lab

Introduction

30 Second Summary

A shipment update can decide whether a logistics team reassures a customer or starts chasing a missing delivery. A slow or broken tracking request turns that routine check into a business problem.

In this project, you will turn an existing two-host Red Hat Enterprise Linux 9 lab into a reliability system for TrackWise. You will test the error budget with controlled faults before preserving the release in Git.

What You'll Build

Once the lab is complete, a TrackWise fault travels visibly from the customer request to a release decision backed by committed evidence.

By the end of this project, you'll have:

• A written reliability contract that lets you explain why a 99.5% customer SLA sits outside a stricter 99.9% internal SLO.
• A two-host customer journey you can call at GET /v1/track while a remote probe checks the same path.
• A repeatable chaos scorecard that converts a controlled outage into an evidence-backed release policy decision.
• Secret Mission: Score and commit a slow-response incident that raises latency while leaving availability close to healthy.

What do I need for this lab?

This project starts from an existing Git repository on infra01 inside your local lab at no added cloud cost.

You need SSH access with sudo on both hosts. The hosts must resolve each other over the local network.

Before We Start

Before any hands-on work, this step defines the measurable reliability contract you are building for TrackWise's GET /v1/track customer journey. The 99.5% monthly customer SLA is the external promise, while the stricter 99.9% rolling 30-day internal availability SLO provides a safety margin that exposes the availability limits of this single-instance lab.

Prepare the Two-Host Reliability Lab

A reliability experiment only proves something when every service has a clear host. Each port needs a defined purpose. Every network path needs a known source.

You will establish that foundation across infra01 and control01. You will also preserve the starting Git state before later steps change the repository.

In this step, get ready to:

• Record the repository baseline and two-host port contract.
• Verify the required RHEL tools and pinned monitoring binaries.
• Install the monitoring services and restrict the customer host ports.

Record the baseline and port contract

The repository baseline separates changes you inherited from changes you make during this project. The port contract gives each process one predictable network address.

• Use your existing SSH details to sign in to infra01 with sudo access.
• Change to the existing Git repository on infra01 using its established path.
• Inspect the starting branch and working tree by running this command:

git status

What does this command establish?

The output names the current branch. It also lists staged files plus unstaged files plus untracked files.

This snapshot becomes your reference point for every milestone commit.

• Record the branch shown by Git as your starting Git branch.
• Record the working tree summary as your starting Git status.
• Use a second terminal session to sign in to control01 with sudo access.

The two-host port contract

The customer workload belongs on control01. The monitoring control plane belongs on infra01.

• On control01, the planned TrackWise Spring Boot service uses TCP port 8080.
• On control01, node_exporter uses TCP port 9100.
• On infra01, Cockpit uses TCP port 9090.
• On infra01, Prometheus binds to 127.0.0.1:9091.
• On infra01, Alertmanager uses TCP port 9093.
• On infra01, Blackbox Exporter uses TCP port 9115.
• On infra01, Grafana uses TCP port 3000.
• On infra01, node_exporter uses TCP port 9100.

Verify the required tooling

Version checks turn environmental assumptions into evidence. This lab requires Java 21 plus Python 3.9 for its application and scoring paths.

The monitoring control plane uses pinned releases of Prometheus, Alertmanager, Blackbox Exporter, and node_exporter. Pinned versions keep the service flags consistent with the supplied unit files.

• Check the development tools and monitoring binaries on infra01 by running:

javac -version
mvn --version
git --version
make -v
curl --version
python3.9 --version
prometheus --version
alertmanager --version
blackbox_exporter --version
node_exporter --version

What should these checks prove?

• The Java compiler reports major version 21.
• Python reports version 3.9.
• Prometheus reports version 3.15.1.
• Alertmanager reports version 0.34.2.
• Blackbox Exporter reports version 0.29.1.
• node_exporter reports version 1.12.1.
• Maven plus Git plus GNU Make plus curl each print version information.

✔️ I see the required versions

Your Red Hat Enterprise Linux toolchain is ready. Keep both SSH sessions open for the service and firewall work.

• Confirm that the Prometheus family binaries came from the official pinned Linux AMD64 archives.
• Continue to the service installation substep.

ⓧ I see an older version

An older Java or Python runtime can break later build and scoring commands. Replace the affected package before continuing.

• Upgrade the required RHEL packages on infra01 by running:

sudo dnf install java-21-openjdk-devel maven git make curl python3.9 firewalld

What does this installation cover?

DNF installs the Java development kit plus the build tools plus the network client plus Python. It also ensures that firewalld is available for the host boundary.

• Download the pinned Prometheus family archives from the official Prometheus download page.
• Verify each archive against its published SHA256 checksum.
• Replace only the affected binaries under /usr/local/bin.
• Use Prometheus 3.15.1 with checksum 6a726885328ad9efebbf36e876ce2fb6225468a71fd1e7ce3ba666d5f4d97a84.
• Use Alertmanager 0.34.2 with checksum 17ee14acd0f6ba6e25a436ff223d9c17ddd5cf484f543ad12f0932eaa89ba186.
• Use Blackbox Exporter 0.29.1 with checksum b82b81839e6317e84fdd367f46d7ddd3b773fa09cbe2e3d5633a59171d741e76.
• Use node_exporter 1.12.1 with checksum b51d8a76aa2a9156a55d501aca6276fae09e262259a5e4e831d2c2222f084e63.

ⓧ A command is missing

A missing command means the corresponding package or binary is absent from the host. Install the RHEL packages first.

• Install the required RHEL packages on infra01 by running:

sudo dnf install java-21-openjdk-devel maven git make curl python3.9 firewalld

What does DNF add?

This installs the application build tools plus Python plus the host firewall tooling. DNF resolves their RHEL package dependencies.

• Download prometheus-3.15.1.linux-amd64.tar.gz from the official Prometheus download page.
• Download alertmanager-0.34.2.linux-amd64.tar.gz from the official Prometheus download page.
• Download blackbox_exporter-0.29.1.linux-amd64.tar.gz from the official Prometheus download page.
• Download node_exporter-1.12.1.linux-amd64.tar.gz from the official Prometheus download page.
• Verify every archive against the checksum listed in the preceding version check.
• Extract each archive on infra01.
• Install the extracted binaries under /usr/local/bin.

• Switch back to the control01 SSH session.
• Check for the pinned host exporter by running:

node_exporter --version

What should the exporter report?

A prepared control01 host reports node_exporter version 1.12.1. The service listens on TCP port 9100 when it runs.

A tool still reports the wrong version

• Check whether the shell is finding an older binary earlier in its executable search path.
• Confirm that the replacement binary exists under /usr/local/bin.
• Still stuck? Help me diagnose conflicting tool versions on my RHEL hosts.

Install services and secure the network

A systemd unit records the exact binary plus its configuration path plus its restart behavior. These units keep the monitoring control plane consistent after a host reboot.

• On infra01, create /etc/systemd/system/prometheus.service with your terminal editor.
• Paste this exact unit into the file:

[Unit]
Description=Prometheus monitoring server
After=network.target

[Service]
Type=simple
User=prometheus
ExecStart=/usr/local/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus --storage.tsdb.retention.time=35d --web.listen-address=127.0.0.1:9091
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

What does the Prometheus unit control?

• The service runs as the existing prometheus user.
• The listener binds to 127.0.0.1:9091.
• The time-series database stays under /var/lib/prometheus.
• The retention setting keeps 35d of data for rolling 30-day calculations.

• Save /etc/systemd/system/prometheus.service.
• Create /etc/systemd/system/alertmanager.service with your terminal editor.
• Paste this exact unit into the file:

[Unit]
Description=Prometheus Alertmanager
After=network.target

[Service]
Type=simple
User=prometheus
ExecStart=/usr/local/bin/alertmanager --config.file=/etc/alertmanager/alertmanager.yml --storage.path=/var/lib/alertmanager --cluster.listen-address=
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

What does the Alertmanager unit control?

• The process loads its routing configuration from /etc/alertmanager/alertmanager.yml.
• Alert lifecycle state stays under /var/lib/alertmanager.
• The empty cluster listener disables high-availability clustering for this single monitoring host.

• Save /etc/systemd/system/alertmanager.service.
• Create /etc/systemd/system/blackbox_exporter.service with your terminal editor.
• Paste this exact unit into the file:

[Unit]
Description=Prometheus Blackbox Exporter
After=network.target

[Service]
Type=simple
User=prometheus
ExecStart=/usr/local/bin/blackbox_exporter --config.file=/etc/blackbox_exporter/blackbox.yml
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

What does the Blackbox Exporter unit control?

The exporter loads probe modules from /etc/blackbox_exporter/blackbox.yml. Its HTTP listener accepts probe requests on TCP port 9115.

• Save /etc/systemd/system/blackbox_exporter.service.

✔️ Awesome, I have all three units

Your three monitoring units now use the required service users plus paths plus listeners. Confirm that every file is saved before continuing.

ⓧ I'd like to double check the full code

Compare each saved unit with these complete reference files.

[Unit]
Description=Prometheus monitoring server
After=network.target

[Service]
Type=simple
User=prometheus
ExecStart=/usr/local/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus --storage.tsdb.retention.time=35d --web.listen-address=127.0.0.1:9091
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

[Unit]
Description=Prometheus Alertmanager
After=network.target

[Service]
Type=simple
User=prometheus
ExecStart=/usr/local/bin/alertmanager --config.file=/etc/alertmanager/alertmanager.yml --storage.path=/var/lib/alertmanager --cluster.listen-address=
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

[Unit]
Description=Prometheus Blackbox Exporter
After=network.target

[Service]
Type=simple
User=prometheus
ExecStart=/usr/local/bin/blackbox_exporter --config.file=/etc/blackbox_exporter/blackbox.yml
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

• Reload the systemd manager configuration on infra01 by running:

sudo systemctl daemon-reload

Why reload systemd?

The reload makes systemd read the new unit files from disk. Starting a new service before this reload can leave systemd unaware of the file.

• Enable the three services at boot by running:

sudo systemctl enable --now prometheus.service
sudo systemctl enable --now alertmanager.service
sudo systemctl enable --now blackbox_exporter.service

What does enabling the services do?

The commands register every service for future boots. The --now option also starts each process in the current session.

The network boundary on control01 protects the future customer API plus host metrics. Only the private IPv4 address of infra01 receives access.

• Switch back to the control01 SSH session.
• Record the private IPv4 address of infra01 as infra01 private IPv4 address.
• Record the active firewalld zone on control01 as your active firewalld zone.
• Permit the TrackWise API port from infra01 by running:

sudo firewall-cmd --permanent --zone=[[FIREWALL_ZONE="your active firewalld zone"]] --add-rich-rule='rule family="ipv4" source address="[[INFRA01_PRIVATE_IPV4="infra01 private IPv4 address"]]/32" port port="8080" protocol="tcp" accept'

What does this rule allow?

The permanent rich rule accepts TCP traffic to port 8080 when the source matches the private address of infra01.

• Permit the node_exporter port from infra01 by running:

sudo firewall-cmd --permanent --zone=[[FIREWALL_ZONE="your active firewalld zone"]] --add-rich-rule='rule family="ipv4" source address="[[INFRA01_PRIVATE_IPV4="infra01 private IPv4 address"]]/32" port port="9100" protocol="tcp" accept'

Why restrict the metrics port?

Host metrics can reveal operating system details. The source restriction keeps port 9100 available to the monitoring host without exposing it to every network peer.

• Apply the permanent firewall rules by running:

sudo firewall-cmd --reload

What does the reload change?

The reload activates the permanent rule set. The restrictions now apply to new inbound connections on control01.

• Confirm the permanent rich rules by running:

sudo firewall-cmd --permanent --list-rich-rules

What should the firewall show?

You should see one source-restricted rule for TCP port 8080. You should also see one source-restricted rule for TCP port 9100.

The services or firewall rules failed

• Confirm that every unit references a binary under /usr/local/bin.
• Confirm that the prometheus service account can read each configuration file.
• Confirm that the active firewall zone matches your active firewalld zone.
• Still stuck? Help me troubleshoot my monitoring services and rich rules.

Checkpoint: exact host contracts

Use these reference statements to confirm the final host layout.

The planned TrackWise Spring Boot service listens on TCP 8080 and node_exporter listens on TCP 9100; permanent firewalld rich rules permit these ports only from infra01's private IPv4 address.

Cockpit listens on TCP 9090, Prometheus listens on 127.0.0.1:9091, Alertmanager listens on TCP 9093, Blackbox Exporter listens on TCP 9115, Grafana listens on TCP 3000, and node_exporter listens on TCP 9100.

Together these contracts keep customer traffic on control01. Monitoring remains anchored on infra01.

Before the final check, predict which result the Blackbox probe returns while the TrackWise service is still undeployed.

• Switch back to the existing repository session on infra01.
• Verify the repository plus Prometheus plus Alertmanager plus the probe endpoint by running:

git status
promtool check config /etc/prometheus/prometheus.yml
amtool alert
curl -sL "http://localhost:9115/probe?target=control01:8080/v1/track&module=http_2xx&debug=true"

What should the final check show?

• Git reports the same known baseline you recorded before setup.
• Prometheus reports that /etc/prometheus/prometheus.yml is valid.
• Alertmanager returns an alert table.
• Blackbox Exporter prints probe diagnostics plus metrics.

A connection failure to control01:8080 is the intended result before deployment. Step 4 removes that failure by placing the TrackWise journey on the port.

Your two-host boundary is now explicit. The repository baseline is preserved. Next up, you will turn the reliability goals into a written TrackWise contract.

Write the Reliability Contract

Your two-host lab now has a verified network path. Your Git repository also has a known baseline.

Metrics cannot decide whether TrackWise is acceptable without a customer-facing SLA. A stricter internal SLO gives engineers room to act before the error budget reaches its limit.

In this step, get ready to:

• Publish the repository overview.
• Define the TrackWise reliability terms.
• Commit the documentation milestone.

Document the lab and repository

The README.md file gives reviewers a map of the lab before they inspect its implementation. The .gitignore file keeps generated files out of future commits.

• In your SSH editor on infra01, create README.md inside the repository where you recorded the baseline.
• Add the project overview by pasting this first section:

# TrackWise SLO and Error Budget Lab

A two-host RHEL 9 reliability lab for a fictional B2B shipment-tracking API. The project separates a 99.5% customer SLA from a stricter 99.9% internal SLO, measures one customer journey, alerts on error-budget burn, and scores controlled incidents.

## Architecture

- `control01`: Spring Boot TrackWise API on port 8080 and node_exporter on 9100.
- `infra01`: Prometheus on 9091, Alertmanager on 9093, Blackbox Exporter on 9115, Grafana on 3000, and node_exporter on 9100.
- Customer journey: `GET /v1/track`.
- Lab fault control: `GET /lab/fault?mode=none|error|slow`.

## Reliability contract

- Customer SLA: 99.5% successful eligible requests per calendar month.
- Internal availability SLO: 99.9% over a rolling 30-day window.
- Internal latency SLO: p95 at or below 300 ms.
- Full terms: `docs/trackwise-reliability-contract.md`.

What Does This Section Establish?

• The architecture section assigns the customer workload to control01.
• The monitoring responsibility belongs to infra01.
• The GET /v1/track journey becomes the customer request path used throughout the lab.
• The GET /lab/fault?mode=none|error|slow endpoint keeps fault control separate from customer traffic.

• Save README.md in your SSH editor.
• Confirm the editor shows the Architecture section above the Reliability contract section.

Missing the First README Section?

• Check that the filename is exactly README.md.
• Compare the host names against control01 and infra01.
• Still stuck? Help me check the first section of my TrackWise README.

• Place your cursor on a new line below the Full terms entry.
• Complete the README by pasting this remaining section:

## Validate

```bash
promtool check config /etc/prometheus/prometheus.yml
promtool check rules /etc/prometheus/slo-rules.yml
curl http://control01:8080/v1/track
make budget
```

## Score an incident

```bash
make chaos-score
make chaos-score FAULT=slow FAULT_SECONDS=360
```

Evidence is written under `evidence/`. The core lab release is tagged `v1.0.0`.

## Important limitation

This is a single-instance learning environment. Restarts, patching, host failure, and experiments consume the internal error budget. The project measures that limitation rather than presenting the lab as production high availability.

What Does the README Prove?

• The Validate section records the checks used by the completed lab.
• The Score an incident section records both supported chaos runs.
• The evidence path makes experiment results easy to review.
• The v1.0.0 target gives the finished core lab a named release point.

• Save the completed README.md.
• Confirm the final section describes the single-instance limitation.

README Sections Out of Order?

• Place Validate directly after the Reliability contract section.
• Place Important limitation at the bottom of the file.
• Need another pair of eyes? Help me compare my complete TrackWise README.

✔️ Awesome, I've got everything!

Your README now maps the lab from customer journey to release target.

ⓧ I'd like to double check the full code

• Compare your complete README.md with this reference:

# TrackWise SLO and Error Budget Lab

A two-host RHEL 9 reliability lab for a fictional B2B shipment-tracking API. The project separates a 99.5% customer SLA from a stricter 99.9% internal SLO, measures one customer journey, alerts on error-budget burn, and scores controlled incidents.

## Architecture

- `control01`: Spring Boot TrackWise API on port 8080 and node_exporter on 9100.
- `infra01`: Prometheus on 9091, Alertmanager on 9093, Blackbox Exporter on 9115, Grafana on 3000, and node_exporter on 9100.
- Customer journey: `GET /v1/track`.
- Lab fault control: `GET /lab/fault?mode=none|error|slow`.

## Reliability contract

- Customer SLA: 99.5% successful eligible requests per calendar month.
- Internal availability SLO: 99.9% over a rolling 30-day window.
- Internal latency SLO: p95 at or below 300 ms.
- Full terms: `docs/trackwise-reliability-contract.md`.

## Validate

```bash
promtool check config /etc/prometheus/prometheus.yml
promtool check rules /etc/prometheus/slo-rules.yml
curl http://control01:8080/v1/track
make budget
```

## Score an incident

```bash
make chaos-score
make chaos-score FAULT=slow FAULT_SECONDS=360
```

Evidence is written under `evidence/`. The core lab release is tagged `v1.0.0`.

## Important limitation

This is a single-instance learning environment. Restarts, patching, host failure, and experiments consume the internal error budget. The project measures that limitation rather than presenting the lab as production high availability.

How to Use This Reference

Compare the headings first. Check each command only after the section order matches.

• Check that Git can see the new README by running this command:

git status

What Does This Command Show?

Git compares the repository with its recorded baseline. The new README appears as an untracked file until you stage it.

Good progress. You should see README.md under the untracked files.

README Missing From Git Status?

• Check that you saved the file inside the repository where you recorded the baseline.
• Check that the filename does not contain an extra extension.
• Still missing? Help me find why Git cannot see README.md.

Repository ignore rules protect future commits from build output. They also exclude local editor settings from the project history.

• In your SSH editor on infra01, create .gitignore beside README.md.
• Add the repository ignore rules by pasting this content:

target/
*.class
*.jar
*.log
.env
.idea/
.vscode/
.DS_Store
Thumbs.db
*.tar.gz
*.rpm
evidence/*.tmp

What Do These Rules Protect?

• Build outputs stay outside the repository history.
• Local editor settings stay specific to each contributor.
• Downloaded packages stay outside future documentation commits.
• Temporary evidence files stay separate from final experiment records.

• Save .gitignore.
• Confirm the file ends with evidence/*.tmp.

✔️ Awesome, I've got everything!

Your repository now has ignore rules for generated files.

ⓧ I'd like to double check the full code

• Compare your complete .gitignore with this reference:

target/
*.class
*.jar
*.log
.env
.idea/
.vscode/
.DS_Store
Thumbs.db
*.tar.gz
*.rpm
evidence/*.tmp

How to Check the Rules

Check every pattern on its own line. Preserve the leading dots on the editor folders.

• Confirm Git can see both repository files by running this command:

git status

What Changed in Git?

The status output now includes both new repository files. They remain untracked until the documentation set is complete.

You should see README.md plus .gitignore in the untracked files.

Ignore File Missing From Git Status?

• Check that the filename begins with a dot.
• Check that .gitignore sits beside README.md.
• Need help? Help me fix my TrackWise ignore file.

Define the customer and engineering terms

The customer contract measures successful eligible requests over each calendar month. The engineering objective uses a rolling 30-day window to create a safety margin.

The contract also records exclusions before an incident occurs. Its release states turn remaining budget into a concrete engineering decision.

• Create the docs directory by running this command:

mkdir -p docs

What Does This Command Do?

The command creates the docs directory for project documentation. The option keeps the command safe when that directory already exists.

• Confirm the directory exists by running this command:

ls

What Should the Listing Contain?

The directory listing should include docs. This gives the reliability contract a documented location.

You should see docs beside the repository files.

Docs Directory Missing?

• Check that your SSH session is still inside the repository.
• Check that the directory name uses lowercase letters.
• Still stuck? Help me create the docs directory in my repository.

• In your SSH editor on infra01, create docs/trackwise-reliability-contract.md.
• Add the customer-facing terms by pasting this first section:

# TrackWise API Reliability Contract

## Product and customer journey

TrackWise is a fictional B2B shipment-tracking API used by logistics customers.
The covered customer journey is `GET /v1/track`, which returns the current state of a demonstration shipment.

## Customer-facing SLA

- Availability commitment: 99.5% successful eligible requests per calendar month.
- Eligible request: a valid `GET /v1/track` request that reaches the TrackWise service.
- Successful request: an eligible request whose response is not HTTP 5xx.
- Excluded requests: customer-caused HTTP 4xx responses, approved load tests, and traffic outside `/v1/track`.
- Scheduled maintenance exclusion: up to four hours per calendar month when customers receive at least 72 hours notice.
- Emergency maintenance and unannounced maintenance are not excluded.

## Service credits

- 99.0% to less than 99.5% monthly availability: 10% of the monthly service fee.
- 95.0% to less than 99.0% monthly availability: 25% of the monthly service fee.
- Less than 95.0% monthly availability: 50% of the monthly service fee.
- Credits are capped at 50% of the monthly service fee and require a customer claim within 30 days.

How Do the Customer Terms Work?

• The covered request is the same shipment-tracking journey documented in the README.
• HTTP 5xx responses count against customer availability.
• The exclusions define traffic outside the customer commitment.
• The credit tiers make a missed commitment carry an explicit consequence.

• Save docs/trackwise-reliability-contract.md in your SSH editor.
• Confirm the customer-facing section ends with the credit claim period.

Customer Terms Incomplete?

• Check that the customer availability commitment is 99.5%.
• Check that the credit tiers are 10%, 25%, and 50%.
• Need help comparing the terms? Help me check the customer-facing contract.

• Place your cursor on a new line below the service credit section.
• Complete the contract by pasting the internal objectives plus policy:

## Internal engineering objectives

- Availability SLO: 99.9% successful eligible requests over a rolling 30-day window.
- Latency SLO: p95 server latency at or below 300 ms over a rolling 30-day window.
- Probe objective: the external `/v1/track` probe succeeds from `infra01`; it corroborates reachability but is not added to the request-based availability ratio.
- Availability error budget: 0.1% of eligible requests over the rolling 30-day window.

The internal SLO is intentionally stricter than the customer SLA. It creates a safety margin, but it is not a claim that this single-instance lab provides production high availability. Restarts, patching, host failure, and experiments spend the internal budget and reveal where redundancy would be required.

## Error budget policy

- NORMAL: more than 50% of the budget remains. Feature and experiment work may continue.
- CAUTION: more than 25% and at most 50% remains. Review risk and avoid unrelated high-risk changes.
- FREEZE: more than 0% and at most 25% remains. Stop feature releases and prioritize reliability work.
- EXHAUSTED: no budget remains. Allow only security fixes, incident recovery, and changes that directly improve reliability.

Any single incident that consumes at least 20% of the internal budget requires a short post-incident review with one concrete reliability action.

How Does the Internal Policy Work?

• The 99.9% internal objective leaves a 0.1% request error budget.
• The latency objective measures p95 server latency against 300 ms.
• The external probe corroborates reachability without changing the request-based ratio.
• The four policy states connect budget remaining to release behavior.

• Save the completed docs/trackwise-reliability-contract.md.
• Confirm the file ends with the post-incident review rule.

✔️ Awesome, I've got everything!

The TrackWise contract now separates the customer promise from the engineering target.

ⓧ I'd like to double check the full code

• Compare your complete docs/trackwise-reliability-contract.md with this reference:

# TrackWise API Reliability Contract

## Product and customer journey

TrackWise is a fictional B2B shipment-tracking API used by logistics customers.
The covered customer journey is `GET /v1/track`, which returns the current state of a demonstration shipment.

## Customer-facing SLA

- Availability commitment: 99.5% successful eligible requests per calendar month.
- Eligible request: a valid `GET /v1/track` request that reaches the TrackWise service.
- Successful request: an eligible request whose response is not HTTP 5xx.
- Excluded requests: customer-caused HTTP 4xx responses, approved load tests, and traffic outside `/v1/track`.
- Scheduled maintenance exclusion: up to four hours per calendar month when customers receive at least 72 hours notice.
- Emergency maintenance and unannounced maintenance are not excluded.

## Service credits

- 99.0% to less than 99.5% monthly availability: 10% of the monthly service fee.
- 95.0% to less than 99.0% monthly availability: 25% of the monthly service fee.
- Less than 95.0% monthly availability: 50% of the monthly service fee.
- Credits are capped at 50% of the monthly service fee and require a customer claim within 30 days.

## Internal engineering objectives

- Availability SLO: 99.9% successful eligible requests over a rolling 30-day window.
- Latency SLO: p95 server latency at or below 300 ms over a rolling 30-day window.
- Probe objective: the external `/v1/track` probe succeeds from `infra01`; it corroborates reachability but is not added to the request-based availability ratio.
- Availability error budget: 0.1% of eligible requests over the rolling 30-day window.

The internal SLO is intentionally stricter than the customer SLA. It creates a safety margin, but it is not a claim that this single-instance lab provides production high availability. Restarts, patching, host failure, and experiments spend the internal budget and reveal where redundancy would be required.

## Error budget policy

- NORMAL: more than 50% of the budget remains. Feature and experiment work may continue.
- CAUTION: more than 25% and at most 50% remains. Review risk and avoid unrelated high-risk changes.
- FREEZE: more than 0% and at most 25% remains. Stop feature releases and prioritize reliability work.
- EXHAUSTED: no budget remains. Allow only security fixes, incident recovery, and changes that directly improve reliability.

Any single incident that consumes at least 20% of the internal budget requires a short post-incident review with one concrete reliability action.

How to Compare the Contract

Check the customer terms first. Compare the internal objectives only after those terms match.

Commit the documentation milestone

A focused documentation commit keeps the reliability agreement reviewable before implementation changes arrive. Its message identifies the contract as a distinct project milestone.

• Stage the README plus ignore rules plus contract by running this command:

git add README.md .gitignore docs/trackwise-reliability-contract.md

What Does Staging Do?

Git copies the current versions of the three files into the index. The next commit records that staged documentation set.

• Confirm the three files are staged by running this command:

git status

What Should Be Staged?

• The staged changes should include README.md.
• The staged changes should include .gitignore.
• The staged changes should include docs/trackwise-reliability-contract.md.

You should see all three files under the changes ready for commit.

Missing a Staged File?

• Check that each file is saved before staging.
• Check that the contract uses the exact docs/trackwise-reliability-contract.md path.
• Still missing a file? Help me diagnose my staged documentation set.

This commit records documentation only. You can review or revert it independently from the monitoring implementation.

• Create the documentation milestone by running this command:

git commit -m "docs: define TrackWise reliability contract"

What Does This Commit Record?

The commit stores the staged README plus ignore rules plus reliability contract. Its message makes the documentation milestone easy to identify in the history.

• Return to README.md in your SSH editor.
• Confirm the Architecture section names both hosts.
• Switch to docs/trackwise-reliability-contract.md in your SSH editor.
• Confirm the Error budget policy section contains all four release states.

Before you inspect the history, which commit message do you expect at the top?

• Verify the documentation commit by running this command:

git log --oneline

What Does the History Confirm?

The concise history shows each commit hash beside its message. The newest entry proves the contract has a durable review point.

You should see docs: define TrackWise reliability contract at the top of the log. That documentation milestone is now preserved.

Commit Missing From the Log?

• Check the earlier commit output for an identity or repository error.
• Check that the commit message matches docs: define TrackWise reliability contract.
• Need help reading the history? Help me diagnose my missing documentation commit.

Your repository now explains what reliable service means before it measures anything. Next, you will prove why the legacy chaos routes cannot test the customer journey correctly.

Prove the Legacy SLI Blind Spot

Your TrackWise contract now defines which requests count toward the customer promise. The next question is whether the legacy chaos experiment tests that same request population.

A service-level indicator can only judge the labels included in its query. The legacy application sends normal traffic to / while its fault target sends failures to /error. You will measure both paths before deciding whether the customer-journey SLI detects the incident.

In this step, get ready to:

• Inspect the exported HTTP request timer family with its URI and status labels.
• Generate baseline traffic before running the legacy error experiment.
• Preserve the blind-spot evidence in a reviewable Git commit.

Inspect the legacy request metrics

The legacy Spring Boot application publishes its Micrometer request timer through /actuator/prometheus. The Prometheus export keeps each URI and response status in a separate label set.

• Switch back to the infra01 terminal from the previous step.
• Inspect the legacy metrics endpoint by running this command:

curl --fail --silent --show-error http://localhost:8080/actuator/prometheus

What does this scrape show?

• The HTTP timer appears as an http_server_requests_seconds_* family or an http_server_requests_duration_seconds_* family.
• The uri label identifies the route that received each request.
• The status label separates successful responses from HTTP failures.

• Locate the HTTP request timer family in the terminal output.
• Confirm its samples include the uri label.
• Confirm the same samples include the status label.
• Record the family you found as your HTTP request timer family.

Cannot find the request timer?

• Confirm the legacy application is still running on port 8080.
• Send one normal request to the application before checking the metrics endpoint again.
• Still stuck? Help me identify the Micrometer HTTP request timer in this Spring Boot metrics output.

Run the legacy chaos experiment

A useful comparison needs normal requests before the fault traffic begins. The baseline gives the customer path its own successful series.

Each traffic run lasts three minutes with the current Makefile defaults. The steady request stream is deliberate because it gives the counters enough activity to compare.

• Generate the normal traffic baseline by running this target:

make chaos-baseline

What does the baseline do?

The target repeatedly requests the legacy customer path at /. Those requests build a successful URI series before the experiment introduces failures.

• Inspect the updated metrics after the baseline by running this command:

curl --fail --silent --show-error http://localhost:8080/actuator/prometheus

What changed after the baseline?

The counter for uri="/" now contains successful requests. This series represents the legacy customer path that a journey-scoped SLI would measure.

Before the fault traffic starts, consider this question: will a journey-scoped SLI register failures sent to a different URI.

• Run the legacy error experiment by running this target:

make chaos-errors

What does the legacy target do?

The target alternates normal requests to / with failing requests to /error. Both paths reach the same application while Micrometer records them under different URI labels.

• Inspect the metrics after the fault traffic by running this command:

curl --fail --silent --show-error http://localhost:8080/actuator/prometheus

What does the comparison prove?

• The successful customer-path series keeps uri="/".
• The HTTP 500 series carries uri="/error".
• A query scoped to the customer URI excludes the failures because the two label sets represent different request populations.

The shortfall is now visible. The application is producing failures while the customer-path series remains falsely healthy.

Do you only see one URI series?

• Confirm the legacy chaos target completed its full request run.
• Confirm the application remained active while the error traffic ran.
• Still stuck? Help me compare the separate URI and status series from my legacy chaos experiment.

Commit the blind-spot evidence

Evidence turns this experiment into a reviewable engineering finding. The cumulative scrape contains the successful baseline series plus the separate HTTP 500 series.

• Preserve the cumulative metrics under evidence/ by running these commands:

mkdir -p evidence
curl --fail --silent --show-error http://localhost:8080/actuator/prometheus > evidence/legacy-sli-blind-spot.txt

What does this evidence contain?

The scrape preserves the timer family with its uri and status labels. Its cumulative counters include the baseline traffic plus the legacy error traffic because the service stayed running between both tests.

The separate series support one conclusion: /error does not test an SLI scoped to the customer path.

✔️ My evidence shows both series

Your evidence file preserves successful customer-path traffic plus HTTP 500 traffic on the separate error URI. The blind spot is ready to commit.

ⓧ I'd like to double check the evidence

Compare the saved scrape against these checkpoints before committing it.

• Confirm the file contains the HTTP request timer family you recorded earlier.
• Confirm a successful series contains uri="/".
• Confirm an HTTP 500 series contains uri="/error".
• Confirm the separate labels demonstrate why customer-path queries exclude the error traffic.

A Git milestone keeps this known limitation separate from the implementation that fixes it later. The reliability documentation remains unchanged in this commit.

• Stage the evidence by running these commands:

git add evidence
git commit -m "test: capture legacy SLI blind spot"

What did this commit preserve?

The commit records evidence that normal customer traffic and legacy failure traffic use separate URI label series. Reviewers can now see why the old chaos target is insufficient for a journey-scoped reliability test.

Before the checkpoint, predict whether Git will report one new evidence commit with a clean working tree.

• Verify the repository checkpoint by running these commands:

git status
git log --oneline

What should you see?

Git reports a clean working tree. The latest history entry shows test: capture legacy SLI blind spot above the existing documentation commit.

Is the evidence missing from the commit?

• Confirm evidence/legacy-sli-blind-spot.txt exists inside the repository.
• Confirm the evidence path is not covered by the temporary-file ignore rule.
• Still stuck? Help me verify why my legacy SLI evidence was not included in the Git commit.

You have pinned down the legacy blind spot with exported metrics and a reviewable commit. Next, you will place fault injection behind the covered customer journey so every failure reaches the same SLI.

Build Journey SLIs and External Probing

Your committed Git evidence proved that faults on /error leave a journey-scoped SLI looking healthy. The experiment needs every customer attempt to use the same path before its measurements can be trusted.

You will move fault injection behind one Spring Boot journey. Micrometer records the server view. Prometheus stores the measurements. Blackbox Exporter checks the same journey from outside the application.

In this step, get ready to:

• Move error and latency faults behind /v1/track.
• Configure server-side SLIs plus an external journey probe.
• Validate the monitoring files before committing the implementation.

Put faults behind the customer journey

The activeFault field controls how /v1/track responds. Every normal request plus every fault now shares one Micrometer URI label.

• Return to ChaosController.java in the repository on infra01.
• Replace the legacy customer handler with these fields plus the track() method:

    private volatile String activeFault = "none";
    private volatile long slowMillis = 600;

    @GetMapping("/v1/track")
    public String track() throws InterruptedException {
        if ("error".equals(activeFault)) {
            throw new ResponseStatusException(
                    HttpStatus.INTERNAL_SERVER_ERROR,
                    "simulated TrackWise journey failure");
        }
        if ("slow".equals(activeFault)) {
            Thread.sleep(slowMillis);
        }
        return "{\"trackingId\":\"DEMO-001\",\"status\":\"IN_TRANSIT\"}\n";
    }

How does the journey behave?

• The activeFault field starts at none, so the journey returns the demonstration shipment.
• The error mode raises HTTP 500 from the customer journey itself.
• The slow mode delays the same journey by the value stored in slowMillis.

• Add the fault-control method below track():

    @GetMapping("/lab/fault")
    public String setFault(
            @RequestParam(defaultValue = "none") String mode,
            @RequestParam(defaultValue = "600") long ms) {

        if (!"none".equalsIgnoreCase(mode)
                && !"error".equalsIgnoreCase(mode)
                && !"slow".equalsIgnoreCase(mode)) {
            throw new ResponseStatusException(
                    HttpStatus.BAD_REQUEST,
                    "mode must be none, error, or slow");
        }

        activeFault = mode.toLowerCase();
        slowMillis = Math.min(Math.max(ms, 0), 10000);
        return "fault=" + activeFault + " slowMillis=" + slowMillis + "\n";
    }

What does the fault control do?

• The mode parameter accepts none, error, or slow.
• Unsupported modes return a bad-request response before the active state changes.
• The delay is constrained to a range from 0 through 10000 milliseconds.

• Replace the contents of application.properties with this configuration:

server.address=0.0.0.0
management.endpoints.web.exposure.include=health,prometheus
management.metrics.distribution.slo.http.server.requests=300ms

What does the application configuration do?

• The server listens on every host interface so the restricted connection from infra01 can reach it.
• The management configuration exposes the health endpoint plus the Prometheus scrape endpoint.
• The request timer publishes a 300ms histogram boundary for the latency objective.

• Save ChaosController.java.
• Save application.properties.
• Deploy the updated application to control01 by running:

make install-app

What does this target deploy?

The existing project target packages the TrackWise application. It installs the service through the repository's prepared deployment workflow.

Before you test the normal mode, what shipment response do you expect from the customer journey?

• Request the TrackWise journey from infra01 by running:

curl http://control01:8080/v1/track

What should you see?

You should see {"trackingId":"DEMO-001","status":"IN_TRANSIT"}. The response proves that the application is reachable through the restricted two-host path.

• Switch the journey to error mode by running:

curl "http://control01:8080/lab/fault?mode=error&ms=600"

What changed?

The response reports fault=error with a slowMillis value of 600. Future calls to the same customer path now fail.

• Request the customer journey again by running:

curl http://control01:8080/v1/track

What does the failure prove?

You should receive an HTTP 500 response from /v1/track. Micrometer can now count the fault against the same URI label used by normal customer attempts.

• Switch the journey to slow mode by running:

curl "http://control01:8080/lab/fault?mode=slow&ms=600"

What changed this time?

The fault state now reports fault=slow with a 600 millisecond delay. The journey remains successful while its latency rises above the internal objective.

• Request the delayed customer journey by running:

curl http://control01:8080/v1/track

What should you notice?

You should still receive the shipment response. The pause before the response proves that latency faults can affect the customer path without creating HTTP 500 errors.

• Restore the normal journey mode by running:

curl "http://control01:8080/lab/fault?mode=none&ms=600"

What does recovery look like?

The response reports fault=none. New journey requests return immediately with the demonstration shipment status.

Journey request still failing?

• Check that activeFault is initialized to none.
• Check that the firewall rule on control01 still permits port 8080 from the private IPv4 address of infra01.
• Use Help me diagnose why the TrackWise journey is unreachable or returning the wrong fault mode..

Configure the journey monitoring stack

Server metrics explain how the application handled requests. An external probe adds an independent view of whether infra01 can complete the same customer journey.

• Create monitoring/blackbox.yml with this module:

modules:
  http_2xx:
    prober: http

What does the probe module do?

The http_2xx module uses the HTTP prober. Blackbox Exporter reports whether the target returns a successful response.

• Create monitoring/alertmanager.yml with this routing configuration:

route:
  receiver: lab-console
  group_by: [alertname, service, severity]
  group_wait: 0s
  group_interval: 1m
  repeat_interval: 4h

receivers:
  - name: lab-console

How are alerts grouped?

The Alertmanager route groups notifications by alert name. It also groups them by service plus severity.

The lab-console receiver keeps alert lifecycle data local to the lab. This makes each alert available for timestamp inspection.

• Create the first section of monitoring/prometheus.yml with these global settings plus scrape jobs:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/slo-rules.yml

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - localhost:9093

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9091"]

  - job_name: node-infra
    static_configs:
      - targets: ["localhost:9100"]

  - job_name: node-control
    static_configs:
      - targets: ["control01:9100"]

What does this section monitor?

Prometheus evaluates rules every 15s. It sends active alerts to Alertmanager on port 9093.

The first three jobs monitor Prometheus plus both host exporters. This separates infrastructure reachability from application behavior.

• Append the application scrape plus journey probe jobs to monitoring/prometheus.yml:

  - job_name: observer
    metrics_path: /actuator/prometheus
    static_configs:
      - targets: ["control01:8080"]

  - job_name: journey-probe
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - http://control01:8080/v1/track
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115

How does the external probe work?

The observer job collects Micrometer metrics from control01. The journey-probe job sends /v1/track to Blackbox Exporter on port 9115.

Relabeling preserves the customer URL as the probe target. It directs the actual scrape request to the local exporter.

• Create the first recording-rule section in monitoring/slo-rules.yml:

groups:
  - name: trackwise-sli
    interval: 15s
    rules:
      - record: trackwise:requests:rate5m
        expr: |
          sum(rate(http_server_requests_seconds_count{job="observer",uri="/v1/track"}[5m]))
          or
          sum(rate(http_server_requests_duration_seconds_count{job="observer",uri="/v1/track"}[5m]))
      - record: trackwise:errors:rate5m
        expr: |
          sum(rate(http_server_requests_seconds_count{job="observer",uri="/v1/track",status=~"5.."}[5m]))
          or
          sum(rate(http_server_requests_duration_seconds_count{job="observer",uri="/v1/track",status=~"5.."}[5m]))
      - record: trackwise:availability_error_ratio:rate5m
        expr: trackwise:errors:rate5m / trackwise:requests:rate5m
      - record: trackwise:requests:rate1h
        expr: sum(rate(http_server_requests_seconds_count{job="observer",uri="/v1/track"}[1h])) or sum(rate(http_server_requests_duration_seconds_count{job="observer",uri="/v1/track"}[1h]))
      - record: trackwise:errors:rate1h
        expr: sum(rate(http_server_requests_seconds_count{job="observer",uri="/v1/track",status=~"5.."}[1h])) or sum(rate(http_server_requests_duration_seconds_count{job="observer",uri="/v1/track",status=~"5.."}[1h]))
      - record: trackwise:availability_error_ratio:rate1h
        expr: trackwise:errors:rate1h / trackwise:requests:rate1h

Why record requests and errors separately?

The request rules count every attempt on /v1/track. The error rules select HTTP 5xx responses from the same URI population.

Dividing errors by requests creates the availability error ratio. The alternate metric names support the timer forms that the inspected actuator endpoint may expose.

• Append the middle-window recording rules to monitoring/slo-rules.yml:

      - record: trackwise:requests:rate30m
        expr: sum(rate(http_server_requests_seconds_count{job="observer",uri="/v1/track"}[30m])) or sum(rate(http_server_requests_duration_seconds_count{job="observer",uri="/v1/track"}[30m]))
      - record: trackwise:errors:rate30m
        expr: sum(rate(http_server_requests_seconds_count{job="observer",uri="/v1/track",status=~"5.."}[30m])) or sum(rate(http_server_requests_duration_seconds_count{job="observer",uri="/v1/track",status=~"5.."}[30m]))
      - record: trackwise:availability_error_ratio:rate30m
        expr: trackwise:errors:rate30m / trackwise:requests:rate30m
      - record: trackwise:requests:rate6h
        expr: sum(rate(http_server_requests_seconds_count{job="observer",uri="/v1/track"}[6h])) or sum(rate(http_server_requests_duration_seconds_count{job="observer",uri="/v1/track"}[6h]))
      - record: trackwise:errors:rate6h
        expr: sum(rate(http_server_requests_seconds_count{job="observer",uri="/v1/track",status=~"5.."}[6h])) or sum(rate(http_server_requests_duration_seconds_count{job="observer",uri="/v1/track",status=~"5.."}[6h]))
      - record: trackwise:availability_error_ratio:rate6h
        expr: trackwise:errors:rate6h / trackwise:requests:rate6h

Why add middle windows?

The 30m plus 6h windows distinguish short spikes from sustained consumption. Alert rules can require both views before firing.

• Confirm that Git sees the monitoring edits by running:

git status

What should Git report?

You should see modified or untracked paths under monitoring/ plus src/. This is the visible checkpoint before you finish the longer rule file.

• Append the long-window plus journey-output rules to monitoring/slo-rules.yml:

      - record: trackwise:requests:rate3d
        expr: sum(rate(http_server_requests_seconds_count{job="observer",uri="/v1/track"}[3d])) or sum(rate(http_server_requests_duration_seconds_count{job="observer",uri="/v1/track"}[3d]))
      - record: trackwise:errors:rate3d
        expr: sum(rate(http_server_requests_seconds_count{job="observer",uri="/v1/track",status=~"5.."}[3d])) or sum(rate(http_server_requests_duration_seconds_count{job="observer",uri="/v1/track",status=~"5.."}[3d]))
      - record: trackwise:availability_error_ratio:rate3d
        expr: trackwise:errors:rate3d / trackwise:requests:rate3d
      - record: trackwise:requests:rate30d
        expr: sum(rate(http_server_requests_seconds_count{job="observer",uri="/v1/track"}[30d])) or sum(rate(http_server_requests_duration_seconds_count{job="observer",uri="/v1/track"}[30d]))
      - record: trackwise:errors:rate30d
        expr: sum(rate(http_server_requests_seconds_count{job="observer",uri="/v1/track",status=~"5.."}[30d])) or sum(rate(http_server_requests_duration_seconds_count{job="observer",uri="/v1/track",status=~"5.."}[30d]))
      - record: trackwise:availability_error_ratio:rate30d
        expr: trackwise:errors:rate30d / trackwise:requests:rate30d
      - record: trackwise:journey_latency_p95_seconds:rate5m
        expr: histogram_quantile(0.95, sum by (le) (rate(http_server_requests_seconds_bucket{job="observer",uri="/v1/track"}[5m]) or rate(http_server_requests_duration_seconds_bucket{job="observer",uri="/v1/track"}[5m])))
      - record: trackwise:probe_success:latest
        expr: probe_success{job="journey-probe"}

What do the final SLI rules add?

The 3d plus 30d ratios support slower budget decisions. The p95 rule measures server latency from histogram buckets.

The probe rule records the latest external success value. It remains separate from the request-based availability ratio.

• Append the burn-rate alert group to monitoring/slo-rules.yml:

  - name: trackwise-burn-alerts
    interval: 15s
    rules:
      - alert: TrackWiseAvailabilityBudgetBurnPage
        expr: (trackwise:availability_error_ratio:rate1h > (14.4 * 0.001) and trackwise:availability_error_ratio:rate5m > (14.4 * 0.001)) or (trackwise:availability_error_ratio:rate6h > (6 * 0.001) and trackwise:availability_error_ratio:rate30m > (6 * 0.001))
        labels:
          service: trackwise
          severity: page
        annotations:
          summary: TrackWise is rapidly consuming its 99.9% availability budget
          runbook: docs/trackwise-reliability-contract.md
      - alert: TrackWiseAvailabilityBudgetBurnTicket
        expr: trackwise:availability_error_ratio:rate3d > 0.001 and trackwise:availability_error_ratio:rate6h > 0.001
        labels:
          service: trackwise
          severity: ticket
        annotations:
          summary: TrackWise is steadily consuming its 99.9% availability budget
          runbook: docs/trackwise-reliability-contract.md

How do the burn alerts work?

The page alert combines the 14.4 pair with the 6 pair. Each pair requires its long window plus its short window to exceed the threshold.

The ticket alert watches slower consumption across 3d plus 6h windows. Both alerts point reviewers to the committed reliability contract.

• Create systemd/prometheus.service as the repository copy of the prepared Prometheus unit:

[Unit]
Description=Prometheus monitoring server
After=network.target

[Service]
Type=simple
User=prometheus
ExecStart=/usr/local/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus --storage.tsdb.retention.time=35d --web.listen-address=127.0.0.1:9091
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

What does the Prometheus unit preserve?

The systemd unit keeps Prometheus on local port 9091. It retains 35d of metrics for the rolling objective.

• Create systemd/alertmanager.service as the repository copy of the prepared Alertmanager unit:

[Unit]
Description=Prometheus Alertmanager
After=network.target

[Service]
Type=simple
User=prometheus
ExecStart=/usr/local/bin/alertmanager --config.file=/etc/alertmanager/alertmanager.yml --storage.path=/var/lib/alertmanager --cluster.listen-address=
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

What does the Alertmanager unit preserve?

The unit reads the lab routing file from /etc/alertmanager/alertmanager.yml. Its empty cluster address keeps this single-node lab outside high-availability mode.

• Confirm that Git sees both repository unit files by running:

git status

What should Git show now?

You should see new paths under systemd/ alongside the application plus monitoring changes. The working tree now reflects each implementation area planned for the commit.

• Create systemd/blackbox_exporter.service as the repository copy of the prepared probe unit:

[Unit]
Description=Prometheus Blackbox Exporter
After=network.target

[Service]
Type=simple
User=prometheus
ExecStart=/usr/local/bin/blackbox_exporter --config.file=/etc/blackbox_exporter/blackbox.yml
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

What does the probe unit preserve?

The unit starts Blackbox Exporter with the repository's probe configuration installed under /etc/blackbox_exporter/. Prometheus reaches the exporter through its local listener.

• Save every file under monitoring/.
• Save every file under systemd/.
• Install the monitoring configuration on infra01 by running:

make install-monitoring

What does this target install?

The existing project target installs the Prometheus configuration. It also installs the SLO rules plus the Alertmanager and Blackbox Exporter configurations.

Before you validate the files, do you expect both checks to succeed after copying the supplied configuration?

• Validate the installed Prometheus configuration by running:

sudo /usr/local/bin/promtool check config /etc/prometheus/prometheus.yml

What confirms the configuration is valid?

Promtool should report that the Prometheus configuration is valid. A successful check proves that the scrape jobs plus Alertmanager wiring can be parsed.

• Validate the installed SLO rules by running:

sudo /usr/local/bin/promtool check rules /etc/prometheus/slo-rules.yml

What confirms the rules are valid?

Promtool should report success for /etc/prometheus/slo-rules.yml. This confirms that the recording rules plus burn-rate alerts use valid rule syntax.

Promtool validation failed?

• Check the indentation beneath scrape_configs plus both rule groups.
• Check that the Prometheus file points to /etc/prometheus/slo-rules.yml.
• Use Help me fix my TrackWise Prometheus or SLO rule validation error..

✔️ Awesome, I've got everything!

Great. Both Promtool checks pass. Save the application plus monitoring files before the final fault test.

ⓧ I'd like to double check the full code

Compare your application controller against this complete file.

package com.nextwork.observer;

import org.springframework.http.HttpStatus;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.server.ResponseStatusException;

@RestController
public class ChaosController {

    private volatile String activeFault = "none";
    private volatile long slowMillis = 600;

    @GetMapping("/v1/track")
    public String track() throws InterruptedException {
        if ("error".equals(activeFault)) {
            throw new ResponseStatusException(
                    HttpStatus.INTERNAL_SERVER_ERROR,
                    "simulated TrackWise journey failure");
        }
        if ("slow".equals(activeFault)) {
            Thread.sleep(slowMillis);
        }
        return "{\"trackingId\":\"DEMO-001\",\"status\":\"IN_TRANSIT\"}\n";
    }

    @GetMapping("/lab/fault")
    public String setFault(
            @RequestParam(defaultValue = "none") String mode,
            @RequestParam(defaultValue = "600") long ms) {

        if (!"none".equalsIgnoreCase(mode)
                && !"error".equalsIgnoreCase(mode)
                && !"slow".equalsIgnoreCase(mode)) {
            throw new ResponseStatusException(
                    HttpStatus.BAD_REQUEST,
                    "mode must be none, error, or slow");
        }

        activeFault = mode.toLowerCase();
        slowMillis = Math.min(Math.max(ms, 0), 10000);
        return "fault=" + activeFault + " slowMillis=" + slowMillis + "\n";
    }

    @GetMapping("/slow")
    public String slow(@RequestParam(defaultValue = "500") long ms) throws InterruptedException {
        Thread.sleep(Math.min(ms, 10000));
        return "slept " + ms + "ms\n";
    }

    @GetMapping("/error")
    public String error() {
        throw new ResponseStatusException(
                HttpStatus.INTERNAL_SERVER_ERROR,
                "simulated failure");
    }

    @GetMapping("/simulate-cpu")
    public String cpuEndpoint() {
        long endTime = System.currentTimeMillis() + 10000;
        while (System.currentTimeMillis() < endTime) {
            // Bounded CPU fault for the existing lab.
        }
        return "CPU Burned";
    }
}

What should match?

Confirm the controller contains the shared journey handler plus the global fault-control method. The three legacy lab handlers remain available.

Compare your application properties against this complete file.

server.address=0.0.0.0
management.endpoints.web.exposure.include=health,prometheus
management.metrics.distribution.slo.http.server.requests=300ms

What should match?

Confirm all three properties appear exactly once. The histogram boundary must remain 300ms.

Compare your monitoring files against these complete references.

modules:
  http_2xx:
    prober: http

What should match?

Confirm the file defines one http_2xx HTTP probe module.

route:
  receiver: lab-console
  group_by: [alertname, service, severity]
  group_wait: 0s
  group_interval: 1m
  repeat_interval: 4h

receivers:
  - name: lab-console

What should match?

Confirm the route plus receiver both use lab-console.

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/slo-rules.yml

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - localhost:9093

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9091"]

  - job_name: node-infra
    static_configs:
      - targets: ["localhost:9100"]

  - job_name: node-control
    static_configs:
      - targets: ["control01:9100"]

  - job_name: observer
    metrics_path: /actuator/prometheus
    static_configs:
      - targets: ["control01:8080"]

  - job_name: journey-probe
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - http://control01:8080/v1/track
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115

What should match?

Confirm all five scrape jobs are present. The probe target must remain the TrackWise customer journey.

groups:
  - name: trackwise-sli
    interval: 15s
    rules:
      - record: trackwise:requests:rate5m
        expr: |
          sum(rate(http_server_requests_seconds_count{job="observer",uri="/v1/track"}[5m]))
          or
          sum(rate(http_server_requests_duration_seconds_count{job="observer",uri="/v1/track"}[5m]))
      - record: trackwise:errors:rate5m
        expr: |
          sum(rate(http_server_requests_seconds_count{job="observer",uri="/v1/track",status=~"5.."}[5m]))
          or
          sum(rate(http_server_requests_duration_seconds_count{job="observer",uri="/v1/track",status=~"5.."}[5m]))
      - record: trackwise:availability_error_ratio:rate5m
        expr: trackwise:errors:rate5m / trackwise:requests:rate5m
      - record: trackwise:requests:rate1h
        expr: sum(rate(http_server_requests_seconds_count{job="observer",uri="/v1/track"}[1h])) or sum(rate(http_server_requests_duration_seconds_count{job="observer",uri="/v1/track"}[1h]))
      - record: trackwise:errors:rate1h
        expr: sum(rate(http_server_requests_seconds_count{job="observer",uri="/v1/track",status=~"5.."}[1h])) or sum(rate(http_server_requests_duration_seconds_count{job="observer",uri="/v1/track",status=~"5.."}[1h]))
      - record: trackwise:availability_error_ratio:rate1h
        expr: trackwise:errors:rate1h / trackwise:requests:rate1h
      - record: trackwise:requests:rate30m
        expr: sum(rate(http_server_requests_seconds_count{job="observer",uri="/v1/track"}[30m])) or sum(rate(http_server_requests_duration_seconds_count{job="observer",uri="/v1/track"}[30m]))
      - record: trackwise:errors:rate30m
        expr: sum(rate(http_server_requests_seconds_count{job="observer",uri="/v1/track",status=~"5.."}[30m])) or sum(rate(http_server_requests_duration_seconds_count{job="observer",uri="/v1/track",status=~"5.."}[30m]))
      - record: trackwise:availability_error_ratio:rate30m
        expr: trackwise:errors:rate30m / trackwise:requests:rate30m
      - record: trackwise:requests:rate6h
        expr: sum(rate(http_server_requests_seconds_count{job="observer",uri="/v1/track"}[6h])) or sum(rate(http_server_requests_duration_seconds_count{job="observer",uri="/v1/track"}[6h]))
      - record: trackwise:errors:rate6h
        expr: sum(rate(http_server_requests_seconds_count{job="observer",uri="/v1/track",status=~"5.."}[6h])) or sum(rate(http_server_requests_duration_seconds_count{job="observer",uri="/v1/track",status=~"5.."}[6h]))
      - record: trackwise:availability_error_ratio:rate6h
        expr: trackwise:errors:rate6h / trackwise:requests:rate6h
      - record: trackwise:requests:rate3d
        expr: sum(rate(http_server_requests_seconds_count{job="observer",uri="/v1/track"}[3d])) or sum(rate(http_server_requests_duration_seconds_count{job="observer",uri="/v1/track"}[3d]))
      - record: trackwise:errors:rate3d
        expr: sum(rate(http_server_requests_seconds_count{job="observer",uri="/v1/track",status=~"5.."}[3d])) or sum(rate(http_server_requests_duration_seconds_count{job="observer",uri="/v1/track",status=~"5.."}[3d]))
      - record: trackwise:availability_error_ratio:rate3d
        expr: trackwise:errors:rate3d / trackwise:requests:rate3d
      - record: trackwise:requests:rate30d
        expr: sum(rate(http_server_requests_seconds_count{job="observer",uri="/v1/track"}[30d])) or sum(rate(http_server_requests_duration_seconds_count{job="observer",uri="/v1/track"}[30d]))
      - record: trackwise:errors:rate30d
        expr: sum(rate(http_server_requests_seconds_count{job="observer",uri="/v1/track",status=~"5.."}[30d])) or sum(rate(http_server_requests_duration_seconds_count{job="observer",uri="/v1/track",status=~"5.."}[30d]))
      - record: trackwise:availability_error_ratio:rate30d
        expr: trackwise:errors:rate30d / trackwise:requests:rate30d
      - record: trackwise:journey_latency_p95_seconds:rate5m
        expr: histogram_quantile(0.95, sum by (le) (rate(http_server_requests_seconds_bucket{job="observer",uri="/v1/track"}[5m]) or rate(http_server_requests_duration_seconds_bucket{job="observer",uri="/v1/track"}[5m])))
      - record: trackwise:probe_success:latest
        expr: probe_success{job="journey-probe"}

  - name: trackwise-burn-alerts
    interval: 15s
    rules:
      - alert: TrackWiseAvailabilityBudgetBurnPage
        expr: (trackwise:availability_error_ratio:rate1h > (14.4 * 0.001) and trackwise:availability_error_ratio:rate5m > (14.4 * 0.001)) or (trackwise:availability_error_ratio:rate6h > (6 * 0.001) and trackwise:availability_error_ratio:rate30m > (6 * 0.001))
        labels:
          service: trackwise
          severity: page
        annotations:
          summary: TrackWise is rapidly consuming its 99.9% availability budget
          runbook: docs/trackwise-reliability-contract.md
      - alert: TrackWiseAvailabilityBudgetBurnTicket
        expr: trackwise:availability_error_ratio:rate3d > 0.001 and trackwise:availability_error_ratio:rate6h > 0.001
        labels:
          service: trackwise
          severity: ticket
        annotations:
          summary: TrackWise is steadily consuming its 99.9% availability budget
          runbook: docs/trackwise-reliability-contract.md

What should match?

Confirm every recording rule scopes requests to /v1/track. Confirm the page plus ticket alerts appear in the second rule group.

Compare your repository unit files against these complete references.

[Unit]
Description=Prometheus monitoring server
After=network.target

[Service]
Type=simple
User=prometheus
ExecStart=/usr/local/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus --storage.tsdb.retention.time=35d --web.listen-address=127.0.0.1:9091
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

What should match?

Confirm Prometheus listens on 127.0.0.1:9091 with 35d retention.

[Unit]
Description=Prometheus Alertmanager
After=network.target

[Service]
Type=simple
User=prometheus
ExecStart=/usr/local/bin/alertmanager --config.file=/etc/alertmanager/alertmanager.yml --storage.path=/var/lib/alertmanager --cluster.listen-address=
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

What should match?

Confirm Alertmanager uses the expected configuration path plus local state directory.

[Unit]
Description=Prometheus Blackbox Exporter
After=network.target

[Service]
Type=simple
User=prometheus
ExecStart=/usr/local/bin/blackbox_exporter --config.file=/etc/blackbox_exporter/blackbox.yml
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

What should match?

Confirm Blackbox Exporter reads /etc/blackbox_exporter/blackbox.yml.

Prove the shared journey now fails visibly

The final check recreates the failure that the legacy experiment missed. This time the application metric plus the external probe observe the same broken customer path.

• Switch the journey to error mode by running:

curl "http://control01:8080/lab/fault?mode=error&ms=600"

What state is active?

The response should report fault=error. Every new request to /v1/track now contributes an HTTP 500 to the journey series.

Before you send the next request, do you expect the server-side ratio plus the external probe to stay healthy?

• Send a failing customer request by running:

curl http://control01:8080/v1/track

What should the request reveal?

You should receive HTTP 500 from the customer journey. This is the intended result because the fault now lives behind /v1/track.

• Repeat the customer request several times to add failing samples to the journey timer.
• Wait for the next 15s scrape interval.
• Query the five-minute availability error ratio by running:

curl 'http://localhost:9091/api/v1/query?query=trackwise%3Aavailability_error_ratio%3Arate5m'

What should the ratio show?

The query should return a value above zero after Prometheus has collected the failing samples. The server-side SLI now sees the same fault that the customer receives.

• Query the external probe result by running:

curl 'http://localhost:9091/api/v1/query?query=probe_success%7Bjob%3D%22journey-probe%22%7D'

What should the probe show?

The returned value should be 0 for the journey-probe job. Blackbox Exporter independently confirms that infra01 cannot complete the journey successfully.

• Restore the journey to normal mode by running:

curl "http://control01:8080/lab/fault?mode=none&ms=600"

What does this reset protect?

The reset returns new customer requests to normal behavior. It leaves the lab ready for the scored incident in the next step.

• Stage the implementation files by running:

git add src monitoring systemd

What enters the commit?

Git stages the application changes plus every monitoring configuration and repository unit file. The existing documentation plus blind-spot evidence remain unchanged.

• Commit the journey monitoring implementation by running:

git commit -m "feat: add journey SLO monitoring"

What milestone does this record?

The commit records the transition from URI-separated chaos to a shared customer journey. It also records the server-side SLIs plus external probe configuration.

Before the final check, do you expect Git to report any uncommitted implementation files?

• Verify the repository state by running:

git status

What should Git report?

Git should report a clean working tree. That confirms the application plus monitoring implementation are preserved in feat: add journey SLO monitoring.

That closes the blind spot. Your customer journey now produces server-side availability plus latency measurements while an external probe checks the same path. Next up, you will turn those signals into timestamped alerts plus an error-budget release decision.

Alert, Score, and Gate Releases

Your journey now reports availability, latency, and probe health through Prometheus. Those measurements already expose the effect of a controlled TrackWise fault.

Measurements become operational when rapid error budget burn produces a timestamped alert. In this step, Alertmanager records the alert lifecycle while a Python scorer turns the incident into evidence and a Git release decision.

In this step, get ready to:

• Confirm the burn-rate alerts are loaded and routed to Alertmanager.
• Score a controlled outage against the availability budget.
• Commit the evidence and create an annotated release tag.

Confirm alert routing and timestamps

The page alert combines two long-window and short-window pairs. Alertmanager preserves the firing time in startsAt, which gives the scorer a stable point for calculating time to detect.

• Return to the terminal session on infra01.
• Confirm Prometheus has loaded the recording rules and alert rules by running:

curl http://localhost:9091/api/v1/rules

What does this request check?

• The response lists the rules currently loaded by Prometheus.
• The alert entries show whether each expression is healthy.

• Find TrackWiseAvailabilityBudgetBurnPage in the response.
• Find TrackWiseAvailabilityBudgetBurnTicket in the response.
• Confirm both alert entries report a healthy rule state.
• Check Alertmanager's current alert table by running:

curl http://localhost:9093/api/v2/alerts

What does the alert response show?

• The endpoint returns alert objects currently known to Alertmanager.
• An active alert includes its labels and lifecycle timestamps.
• An empty array means no alert is firing before the controlled outage.

Build and run the SLO scorer

The scorer controls the existing /lab/fault endpoint while sending requests through /v1/track. It samples Prometheus before and after the incident while querying Alertmanager during the fault.

• Create scripts/slo_lab.py inside the existing repository with your editor.
• Add the imports and service settings from this first section:

#!/usr/bin/env python3.9

import json
import os
import time
from datetime import datetime, timezone
from pathlib import Path
from urllib.error import HTTPError, URLError
from urllib.parse import urlencode
from urllib.request import urlopen

PROM_URL = os.environ.get("PROM_URL", "http://localhost:9091")
ALERTMANAGER_URL = os.environ.get("ALERTMANAGER_URL", "http://localhost:9093")
APP_URL = os.environ.get("APP_URL", "http://control01:8080")
SLO_ERROR_RATIO = 0.001
SECONDS_IN_30_DAYS = 30 * 24 * 60 * 60
PAGE_ALERT = "TrackWiseAvailabilityBudgetBurnPage"


def utc_now():
    return datetime.now(timezone.utc)


def iso(value):
    return value.isoformat().replace("+00:00", "Z")


def get_json(url, timeout=5):
    with urlopen(url, timeout=timeout) as response:
        return json.load(response)

How is the scorer configured?

• The three service URLs point to the existing Prometheus, Alertmanager, and TrackWise endpoints.
• The SLO_ERROR_RATIO value represents the 0.1% error budget created by the 99.9% availability objective.
• The time helpers create UTC timestamps that remain comparable across both hosts.

• Add the Prometheus query and fault-control helpers below get_json() by copying:

def prom_value(expression, default=0.0):
    query = urlencode({"query": expression})
    payload = get_json(f"{PROM_URL}/api/v1/query?{query}")
    results = payload.get("data", {}).get("result", [])
    return float(results[0]["value"][1]) if results else default


def set_fault(mode, milliseconds=600):
    query = urlencode({"mode": mode, "ms": milliseconds})
    with urlopen(f"{APP_URL}/lab/fault?{query}", timeout=5) as response:
        return response.read().decode("utf-8").strip()


def journey_succeeds(timeout=5):
    try:
        with urlopen(f"{APP_URL}/v1/track", timeout=timeout) as response:
            return 200 <= response.status < 500
    except (HTTPError, URLError, TimeoutError):
        return False


def active_page_alert():
    query = urlencode({"active": "true", "silenced": "false", "inhibited": "false", "unprocessed": "false", "filter": f'alertname="{PAGE_ALERT}"'})
    alerts = get_json(f"{ALERTMANAGER_URL}/api/v2/alerts?{query}")
    return alerts[0] if alerts else None

How does the scorer observe the lab?

• The Prometheus helper returns the first value from an instant query.
• The fault helper changes the global mode without changing the customer URI.
• The alert helper filters the Alertmanager API for the active page alert.

• Add the timestamp parser and release-policy functions below active_page_alert() by copying:

def parse_rfc3339(value):
    return datetime.fromisoformat(value.replace("Z", "+00:00"))


def policy_state(remaining_ratio):
    if remaining_ratio > 0.50:
        return "NORMAL"
    if remaining_ratio > 0.25:
        return "CAUTION"
    if remaining_ratio > 0:
        return "FREEZE"
    return "EXHAUSTED"


def budget():
    error_ratio = prom_value("trackwise:availability_error_ratio:rate30d")
    burn_rate = error_ratio / SLO_ERROR_RATIO
    remaining_ratio = max(0.0, 1.0 - burn_rate)
    print(f"window_error_ratio={error_ratio:.6f}")
    print(f"availability={100.0 * (1.0 - error_ratio):.5f}%")
    print(f"allowed_error_ratio={SLO_ERROR_RATIO:.6f}")
    print(f"burn_rate={burn_rate:.2f}x")
    print(f"budget_remaining={100.0 * remaining_ratio:.2f}%")
    print(f"policy_state={policy_state(remaining_ratio)}")

How does the release gate work?

• The budget function compares the rolling 30-day error ratio with the allowed 0.1% ratio.
• The remaining fraction selects NORMAL, CAUTION, FREEZE, or EXHAUSTED.
• The printed fields make the gate suitable for a local release check.

• Start the controlled-incident function below budget() by copying:

def chaos():
    fault = os.environ.get("FAULT", "error")
    duration = int(os.environ.get("FAULT_SECONDS", "360"))
    rate = float(os.environ.get("RATE", "5"))
    milliseconds = int(os.environ.get("MS", "600"))
    before_ratio = prom_value("trackwise:availability_error_ratio:rate30d")
    request_rate = prom_value("trackwise:requests:rate30d", default=rate)
    forecast_total = max(request_rate * SECONDS_IN_30_DAYS, rate * duration)
    predicted_errors = rate * duration if fault == "error" else 0.0
    predicted_budget_percent = predicted_errors / max(forecast_total * SLO_ERROR_RATIO, 1.0) * 100.0
    t0 = utc_now()
    set_fault(fault, milliseconds)
    alert_start = None
    request_count = 0
    failed_count = 0
    interval = 1.0 / rate
    deadline = time.monotonic() + duration

How is the incident predicted?

• The function reads the fault type, duration, request rate, and delay from its environment.
• The forecast estimates total 30-day traffic from the observed request rate.
• The predicted budget cost uses the planned failed requests before the fault begins.

• Continue chaos() with the request loop and recovery measurements by copying:

    while time.monotonic() < deadline:
        loop_start = time.monotonic()
        request_count += 1
        if not journey_succeeds(timeout=max(5, milliseconds / 1000 + 2)):
            failed_count += 1
        if alert_start is None:
            alert = active_page_alert()
            if alert:
                alert_start = parse_rfc3339(alert["startsAt"])
        time.sleep(max(0.0, interval - (time.monotonic() - loop_start)))
    fault_end = utc_now()
    set_fault("none", milliseconds)
    recovery_start = time.monotonic()
    while not journey_succeeds():
        time.sleep(0.25)
    service_recovered = utc_now()
    service_ttr = time.monotonic() - recovery_start
    reset_deadline = time.monotonic() + 420
    while time.monotonic() < reset_deadline and active_page_alert():
        time.sleep(5)
    alert_reset = utc_now()
    time.sleep(30)
    after_ratio = prom_value("trackwise:availability_error_ratio:rate30d")
    actual_budget_percent = max(0.0, after_ratio - before_ratio) / SLO_ERROR_RATIO * 100.0
    ttd = (alert_start - t0).total_seconds() if alert_start else None

What does the incident loop measure?

• Each request contributes to the observed request and failure counts.
• The first active page alert supplies the startsAt value used for time to detect.
• The recovery loops measure customer recovery separately from alert reset.

• Add the scorecard fields inside chaos() by copying:

    result = {
        "fault": fault,
        "fault_milliseconds": milliseconds,
        "planned_duration_seconds": duration,
        "planned_request_rate": rate,
        "predicted_failed_requests": predicted_errors,
        "predicted_budget_consumed_percent": predicted_budget_percent,
        "t0": iso(t0),
        "alert_startsAt": iso(alert_start) if alert_start else None,
        "time_to_detect_seconds": ttd,
        "fault_removed_at": iso(fault_end),
        "service_recovered_at": iso(service_recovered),
        "time_to_service_recovery_seconds": service_ttr,
        "alert_reset_observed_at": iso(alert_reset),
        "requests_sent": request_count,
        "failed_requests_observed": failed_count,
        "error_ratio_before": before_ratio,
        "error_ratio_after": after_ratio,
        "actual_budget_consumed_percent": actual_budget_percent,
        "prediction_error_percent_points": actual_budget_percent - predicted_budget_percent,
    }

What goes into the scorecard?

• The scorecard preserves the planned fault settings and predicted budget cost.
• The lifecycle fields distinguish detection, fault removal, customer recovery, and alert reset.
• The final fields compare predicted budget use with the observed ratio movement.

• Finish the scorer with evidence persistence and command dispatch by copying:

    Path("evidence").mkdir(exist_ok=True)
    output = Path("evidence") / f"slo-chaos-{t0.strftime('%Y%m%dT%H%M%SZ')}.json"
    output.write_text(json.dumps(result, indent=2) + "\n", encoding="utf-8")
    print(json.dumps(result, indent=2))
    print(f"evidence_file={output}")


if __name__ == "__main__":
    command = "budget" if len(os.sys.argv) < 2 else os.sys.argv[1]
    if command == "budget":
        budget()
    elif command == "chaos":
        chaos()
    else:
        raise SystemExit("usage: slo_lab.py [budget|chaos]")

How is the result preserved?

• The scorer creates the evidence directory when needed.
• Each run receives a UTC timestamp in its JSON filename.
• The final command switch exposes separate budget and chaos modes.

• Append the SLO validation and scoring targets to the existing Makefile by copying:

PROM_URL ?= http://localhost:9091
ALERTMANAGER_URL ?= http://localhost:9093
APP_URL ?= http://control01:8080
FAULT ?= error
FAULT_SECONDS ?= 360
RATE ?= 5

validate-slo:
	sudo /usr/local/bin/promtool check config /etc/prometheus/prometheus.yml
	sudo /usr/local/bin/promtool check rules /etc/prometheus/slo-rules.yml

budget:
	PROM_URL="$(PROM_URL)" python3.9 scripts/slo_lab.py budget

chaos-score:
	PROM_URL="$(PROM_URL)" ALERTMANAGER_URL="$(ALERTMANAGER_URL)" APP_URL="$(APP_URL)" FAULT="$(FAULT)" FAULT_SECONDS="$(FAULT_SECONDS)" RATE="$(RATE)" MS="$(MS)" python3.9 scripts/slo_lab.py chaos

What do the Make targets provide?

• The variables provide defaults for the monitoring and application endpoints.
• The budget target prints the current release-policy decision.
• The chaos-score target passes the controlled-incident settings into the scorer.

✔️ Awesome, I've got everything!

• Save scripts/slo_lab.py.
• Save the Makefile.

ⓧ I'd like to double check the full code

Compare scripts/slo_lab.py with this complete file.

#!/usr/bin/env python3.9

import json
import os
import time
from datetime import datetime, timezone
from pathlib import Path
from urllib.error import HTTPError, URLError
from urllib.parse import urlencode
from urllib.request import urlopen

PROM_URL = os.environ.get("PROM_URL", "http://localhost:9091")
ALERTMANAGER_URL = os.environ.get("ALERTMANAGER_URL", "http://localhost:9093")
APP_URL = os.environ.get("APP_URL", "http://control01:8080")
SLO_ERROR_RATIO = 0.001
SECONDS_IN_30_DAYS = 30 * 24 * 60 * 60
PAGE_ALERT = "TrackWiseAvailabilityBudgetBurnPage"


def utc_now():
    return datetime.now(timezone.utc)


def iso(value):
    return value.isoformat().replace("+00:00", "Z")


def get_json(url, timeout=5):
    with urlopen(url, timeout=timeout) as response:
        return json.load(response)


def prom_value(expression, default=0.0):
    query = urlencode({"query": expression})
    payload = get_json(f"{PROM_URL}/api/v1/query?{query}")
    results = payload.get("data", {}).get("result", [])
    return float(results[0]["value"][1]) if results else default


def set_fault(mode, milliseconds=600):
    query = urlencode({"mode": mode, "ms": milliseconds})
    with urlopen(f"{APP_URL}/lab/fault?{query}", timeout=5) as response:
        return response.read().decode("utf-8").strip()


def journey_succeeds(timeout=5):
    try:
        with urlopen(f"{APP_URL}/v1/track", timeout=timeout) as response:
            return 200 <= response.status < 500
    except (HTTPError, URLError, TimeoutError):
        return False


def active_page_alert():
    query = urlencode({"active": "true", "silenced": "false", "inhibited": "false", "unprocessed": "false", "filter": f'alertname="{PAGE_ALERT}"'})
    alerts = get_json(f"{ALERTMANAGER_URL}/api/v2/alerts?{query}")
    return alerts[0] if alerts else None


def parse_rfc3339(value):
    return datetime.fromisoformat(value.replace("Z", "+00:00"))


def policy_state(remaining_ratio):
    if remaining_ratio > 0.50:
        return "NORMAL"
    if remaining_ratio > 0.25:
        return "CAUTION"
    if remaining_ratio > 0:
        return "FREEZE"
    return "EXHAUSTED"


def budget():
    error_ratio = prom_value("trackwise:availability_error_ratio:rate30d")
    burn_rate = error_ratio / SLO_ERROR_RATIO
    remaining_ratio = max(0.0, 1.0 - burn_rate)
    print(f"window_error_ratio={error_ratio:.6f}")
    print(f"availability={100.0 * (1.0 - error_ratio):.5f}%")
    print(f"allowed_error_ratio={SLO_ERROR_RATIO:.6f}")
    print(f"burn_rate={burn_rate:.2f}x")
    print(f"budget_remaining={100.0 * remaining_ratio:.2f}%")
    print(f"policy_state={policy_state(remaining_ratio)}")


def chaos():
    fault = os.environ.get("FAULT", "error")
    duration = int(os.environ.get("FAULT_SECONDS", "360"))
    rate = float(os.environ.get("RATE", "5"))
    milliseconds = int(os.environ.get("MS", "600"))
    before_ratio = prom_value("trackwise:availability_error_ratio:rate30d")
    request_rate = prom_value("trackwise:requests:rate30d", default=rate)
    forecast_total = max(request_rate * SECONDS_IN_30_DAYS, rate * duration)
    predicted_errors = rate * duration if fault == "error" else 0.0
    predicted_budget_percent = predicted_errors / max(forecast_total * SLO_ERROR_RATIO, 1.0) * 100.0
    t0 = utc_now()
    set_fault(fault, milliseconds)
    alert_start = None
    request_count = 0
    failed_count = 0
    interval = 1.0 / rate
    deadline = time.monotonic() + duration
    while time.monotonic() < deadline:
        loop_start = time.monotonic()
        request_count += 1
        if not journey_succeeds(timeout=max(5, milliseconds / 1000 + 2)):
            failed_count += 1
        if alert_start is None:
            alert = active_page_alert()
            if alert:
                alert_start = parse_rfc3339(alert["startsAt"])
        time.sleep(max(0.0, interval - (time.monotonic() - loop_start)))
    fault_end = utc_now()
    set_fault("none", milliseconds)
    recovery_start = time.monotonic()
    while not journey_succeeds():
        time.sleep(0.25)
    service_recovered = utc_now()
    service_ttr = time.monotonic() - recovery_start
    reset_deadline = time.monotonic() + 420
    while time.monotonic() < reset_deadline and active_page_alert():
        time.sleep(5)
    alert_reset = utc_now()
    time.sleep(30)
    after_ratio = prom_value("trackwise:availability_error_ratio:rate30d")
    actual_budget_percent = max(0.0, after_ratio - before_ratio) / SLO_ERROR_RATIO * 100.0
    ttd = (alert_start - t0).total_seconds() if alert_start else None
    result = {
        "fault": fault,
        "fault_milliseconds": milliseconds,
        "planned_duration_seconds": duration,
        "planned_request_rate": rate,
        "predicted_failed_requests": predicted_errors,
        "predicted_budget_consumed_percent": predicted_budget_percent,
        "t0": iso(t0),
        "alert_startsAt": iso(alert_start) if alert_start else None,
        "time_to_detect_seconds": ttd,
        "fault_removed_at": iso(fault_end),
        "service_recovered_at": iso(service_recovered),
        "time_to_service_recovery_seconds": service_ttr,
        "alert_reset_observed_at": iso(alert_reset),
        "requests_sent": request_count,
        "failed_requests_observed": failed_count,
        "error_ratio_before": before_ratio,
        "error_ratio_after": after_ratio,
        "actual_budget_consumed_percent": actual_budget_percent,
        "prediction_error_percent_points": actual_budget_percent - predicted_budget_percent,
    }
    Path("evidence").mkdir(exist_ok=True)
    output = Path("evidence") / f"slo-chaos-{t0.strftime('%Y%m%dT%H%M%SZ')}.json"
    output.write_text(json.dumps(result, indent=2) + "\n", encoding="utf-8")
    print(json.dumps(result, indent=2))
    print(f"evidence_file={output}")


if __name__ == "__main__":
    command = "budget" if len(os.sys.argv) < 2 else os.sys.argv[1]
    if command == "budget":
        budget()
    elif command == "chaos":
        chaos()
    else:
        raise SystemExit("usage: slo_lab.py [budget|chaos]")

Compare the complete Makefile with this reference.

SHELL := /usr/bin/bash

.PHONY: all build run smoke install-app install-monitoring validate-monitoring status incident-down incident-up evidence pause resume uninstall clean
.PHONY: chaos-help chaos-check chaos-baseline chaos-latency chaos-errors host-cpu-stress
.PHONY: validate-slo budget chaos-score

all: build

build:
	mvn package

run:
	mvn spring-boot:run

smoke:
	curl --fail --silent --show-error http://localhost:8080/
	@printf '\n'
	curl --fail --silent --show-error http://localhost:8080/actuator/health
	@printf '\n'
	curl --fail --silent --show-error http://localhost:8080/actuator/prometheus > /dev/null
	@echo "Prometheus endpoint returned successfully"

install-app: build
	sudo bash scripts/install-app.sh

install-monitoring:
	sudo bash scripts/install-monitoring.sh

validate-monitoring:
	sudo /usr/local/bin/promtool check config /etc/prometheus/prometheus.yml
	sudo /usr/local/bin/promtool check rules /etc/prometheus/alerts.yml

status:
	systemctl is-active observer.service
	systemctl is-active prometheus.service
	systemctl is-active node_exporter.service
	systemctl is-active grafana-server.service

incident-down:
	sudo systemctl stop observer.service

incident-up:
	sudo systemctl start observer.service

evidence:
	mkdir -p evidence
	systemctl status observer.service > evidence/observer-status.txt || true
	journalctl -u observer.service -b > evidence/observer-journal.txt
	curl --fail --silent --show-error http://localhost:8080/actuator/health > evidence/observer-health.json || printf '{"status":"unreachable"}\n' > evidence/observer-health.json

pause:
	sudo systemctl stop observer.service prometheus.service node_exporter.service grafana-server.service

resume:
	sudo systemctl start node_exporter.service prometheus.service grafana-server.service observer.service

uninstall:
	sudo bash scripts/uninstall.sh

clean:
	mvn clean

BASE ?= http://localhost:8080
DURATION ?= 180
MS ?= 300

chaos-help:
	@echo "make chaos-baseline [DURATION=120] normal traffic only"
	@echo "make chaos-latency [MS=300 DURATION=180] slow requests"
	@echo "make chaos-errors [DURATION=180] 50% failing requests"

chaos-check:
	@curl -sf $(BASE)/actuator/health > /dev/null || { echo "observer is not healthy, stopping"; exit 1; }

chaos-baseline: chaos-check
	@echo "[$$(date +%T)] BASELINE start, $(DURATION)s"
	@timeout --foreground $(DURATION) bash -c 'while true; do curl -s -o /dev/null $(BASE)/; sleep 0.2; done' || true
	@echo "[$$(date +%T)] BASELINE end"

chaos-latency: chaos-check
	@echo "[$$(date +%T)] LATENCY start: /slow?ms=$(MS), $(DURATION)s"
	@timeout --foreground $(DURATION) bash -c 'while true; do curl -s -o /dev/null "$(BASE)/slow?ms=$(MS)" & sleep 0.2; done' || true
	@echo "[$$(date +%T)] LATENCY end"

chaos-errors: chaos-check
	@echo "[$$(date +%T)] ERRORS start: 50% failing, $(DURATION)s"
	@timeout --foreground $(DURATION) bash -c 'while true; do curl -s -o /dev/null $(BASE)/; curl -s -o /dev/null $(BASE)/error; sleep 0.2; done' || true
	@echo "[$$(date +%T)] ERRORS end"

CPU_DURATION ?= 30

host-cpu-stress:
	@echo "Starting CPU stress for $(CPU_DURATION)s"
	@timeout --foreground $(CPU_DURATION) bash -c 'while :; do :; done'

PROM_URL ?= http://localhost:9091
ALERTMANAGER_URL ?= http://localhost:9093
APP_URL ?= http://control01:8080
FAULT ?= error
FAULT_SECONDS ?= 360
RATE ?= 5

validate-slo:
	sudo /usr/local/bin/promtool check config /etc/prometheus/prometheus.yml
	sudo /usr/local/bin/promtool check rules /etc/prometheus/slo-rules.yml

budget:
	PROM_URL="$(PROM_URL)" python3.9 scripts/slo_lab.py budget

chaos-score:
	PROM_URL="$(PROM_URL)" ALERTMANAGER_URL="$(ALERTMANAGER_URL)" APP_URL="$(APP_URL)" FAULT="$(FAULT)" FAULT_SECONDS="$(FAULT_SECONDS)" RATE="$(RATE)" MS="$(MS)" python3.9 scripts/slo_lab.py chaos

Before you run the gate, which policy state do you expect from the current 30-day availability ratio?

• Save both files.
• Test the release-policy calculation by running:

make budget

What should the gate print?

• The output includes the measured error ratio and calculated availability.
• The burn rate determines the percentage of budget remaining.
• The final line prints one release-policy state.

You should see window_error_ratio, availability, allowed_error_ratio, burn_rate, budget_remaining, and policy_state in the output.

Budget command failing?

• Check that scripts/slo_lab.py is inside the repository's scripts folder.
• Check the indentation under the budget target in the Makefile.
• Use the first traceback line to locate any copied Python indentation issue.
• Still stuck? Help me debug my TrackWise budget target.

The controlled run lasts six minutes before recovery checks begin. The scorer may continue waiting for the page alert to reset, so keep the terminal open while it collects the complete lifecycle.

Before you start the outage, do you expect the page alert timestamp to appear before or after the fault is removed?

• Run the controlled error incident by running:

make chaos-score

What happens during the scored run?

• The scorer predicts failed requests and availability-budget consumption.
• The TrackWise journey switches to the error fault while requests continue using /v1/track.
• The scorer captures Alertmanager's startsAt timestamp before restoring normal service.
• The final JSON compares predicted budget cost with the observed 30-day ratio movement.

You should see a JSON scorecard followed by an evidence_file path under evidence/. The alert_startsAt field records when Alertmanager first received the page alert.

Scorecard missing an alert timestamp?

• Confirm TrackWiseAvailabilityBudgetBurnPage still reports a healthy loaded rule.
• Confirm the scorer can reach Alertmanager at http://localhost:9093 from infra01.
• Confirm requests reached the /v1/track journey throughout the fault window.
• Still stuck? Help me trace the missing startsAt value.

• Read the current release gate after the incident by running:

make budget

How should I read the result?

• A NORMAL result permits feature and experiment work.
• A CAUTION result calls for a risk review.
• A FREEZE or EXHAUSTED result prioritizes reliability work over feature releases.

That completes the operational loop. A customer-path failure now produces timestamped alert evidence and a release-policy decision.

Commit and tag the release

The scorecard becomes reviewable evidence once it joins the implementation in Git history. An annotated tag then marks the exact commit that represents the completed core lab.

• Stage the scorer, Makefile, and final evidence by running:

git add Makefile scripts evidence

What does this stage?

Git adds the release gate, scorer, and generated incident evidence to the index for the next commit.

• Record the scored-chaos milestone by running:

git commit -m "feat: score chaos against the error budget"

What does this commit preserve?

The commit ties the scorer implementation to the controlled-error scorecard that demonstrates it.

• Create the annotated core-lab release tag by running:

git tag -a v1.0.0 -m "TrackWise SLO lab v1.0.0"

Why use an annotated tag?

The tag stores the release name and message with a reference to the completed core-lab commit.

Before you verify the release, do you expect Git to report any uncommitted project files?

• Verify the clean release and milestone history by running:

git status
git log --oneline --decorate
git show v1.0.0

What proves the release is complete?

• The status output confirms the working tree is clean.
• The decorated log shows the documentation, blind-spot, monitoring, and scoring milestones.
• The tag view shows the TrackWise SLO lab v1.0.0 annotation above the completed release commit.

You should see nothing to commit, working tree clean in the status output. The v1.0.0 tag should decorate the scored-chaos commit.

Release verification not clean?

• Read the paths listed by git status to identify any unstaged scorecard or source file.
• Confirm the scoring commit appears before checking the tag.
• Confirm v1.0.0 points to the completed core-lab commit.
• Still stuck? Help me verify my TrackWise release history.

Strong finish. Your lab now turns a controlled customer-path outage into alert timing, recovery timing, budget cost, committed evidence, and a release gate.

Secret mission

Score a Latency Incident

A successful response can still make customers wait too long. Run a controlled slow-response incident, compare the latency and availability signals, then preserve your conclusion as reviewable evidence in Git.

Clean Up Your Resources

Clean Up Your Resources

Everything in this lab runs locally on infra01 or control01. There are no ongoing cloud costs, so choose whether to keep the lab active, pause its services, or remove its deployed resources.

Resources you used:

• The Prometheus service configuration in /etc/prometheus/prometheus.yml. The SLO rules are stored in /etc/prometheus/slo-rules.yml.
• The Alertmanager service configuration in /etc/alertmanager/alertmanager.yml. Its local state is stored under /var/lib/alertmanager.
• The Blackbox Exporter service configuration in /etc/blackbox_exporter/blackbox.yml.
• The deployed Spring Boot TrackWise service on control01. Its systemd unit is observer.service.
• The node_exporter service on control01, if Step 1 installed it for this lab.
• The permanent firewalld rich rules that restrict TCP ports 8080 and 9100 to the private IPv4 address of infra01.
• The Prometheus metric history stored under /var/lib/prometheus.

Keep everything running

No action is needed. Choose this option if you want to collect more rolling history or repeat the controlled incidents.

• The TrackWise journey stays available on control01.
• Prometheus continues scraping the server-side metrics and external journey probe.
• Alertmanager continues recording alert lifecycle timestamps.
• The permanent firewall restrictions continue limiting access to infra01.
• Your Git repository retains the evidence commits and the annotated v1.0.0 core release tag.

Pause - I'll come back to this later

Pausing frees host resources while preserving the monitoring configuration, metric history, evidence, commits, and release tag.

• Stop observer.service on control01.
• Stop node_exporter.service on control01 if this project installed it.
• Stop prometheus.service, alertmanager.service, blackbox_exporter.service, node_exporter.service, and grafana-server.service on infra01.
• Leave the configuration files and service units in place.
• Leave the permanent firewall rules in place.
• Confirm that each paused service reports a stopped state.

Your committed scorecards and latency conclusion remain ready for the next experiment.

Delete - I don't want to use this again

This cleanup removes the deployed lab components. Your committed repository, evidence, milestone history, latency-incident commit, and annotated v1.0.0 tag stay untouched.

Prometheus History Is Shared

Removing /var/lib/prometheus clears every sample in that data directory. This includes any metric history that predates the TrackWise lab.

Clean Up infra01

• Stop prometheus.service on infra01.
• Stop alertmanager.service on infra01.
• Stop blackbox_exporter.service on infra01.
• Disable alertmanager.service and blackbox_exporter.service.
• Remove the installed unit files for alertmanager.service and blackbox_exporter.service from the installation locations used earlier.
• Delete /etc/alertmanager/alertmanager.yml.
• Delete /etc/blackbox_exporter/blackbox.yml.
• Delete /etc/prometheus/slo-rules.yml.
• Remove the observer scrape job from /etc/prometheus/prometheus.yml.
• Remove the journey-probe scrape job from /etc/prometheus/prometheus.yml.
• Remove the Alertmanager target from /etc/prometheus/prometheus.yml.
• Remove the SLO rule file reference from /etc/prometheus/prometheus.yml.
• Delete /var/lib/alertmanager to remove the lab alert state.
• Delete /var/lib/prometheus to remove the local metric history.
• Start prometheus.service with the remaining pre-project monitoring configuration.
• Confirm that alertmanager.service and blackbox_exporter.service are no longer active.

Clean Up control01

• Stop observer.service on control01.
• Disable observer.service.
• Remove the deployed TrackWise service unit from the installation location used earlier.
• Remove the deployed Spring Boot application artifact from the installation location used earlier.
• Stop node_exporter.service if Step 1 installed it on control01.
• Disable node_exporter.service if Step 1 installed it on control01.
• Remove the installed node_exporter unit and binary if they were added during this project.
• Remove the permanent rich rule that allows the private IPv4 address of infra01 to reach TCP port 8080.
• Remove the permanent rich rule that allows the private IPv4 address of infra01 to reach TCP port 9100.
• Reload firewalld to apply the permanent rule removal.
• Confirm from infra01 that the removed project services are no longer reachable on control01.

The durable project record remains in your existing repository. Check that cleanup did not modify the committed evidence by running:

git status

What Does This Check?

The command compares your repository with the latest commit.

You should see nothing to commit, working tree clean.

Nice Work!

Nice Work!

Outstanding work. Your two-host TrackWise lab now turns measured customer reliability into a documented release decision.

You've learned how to:

• Define a reliability contract that separates a 99.5% customer SLA from a stricter 99.9% internal SLO.
• Expose the legacy SLI blind spot where failures on /error leave the customer journey healthy. Close that gap by measuring /v1/track with Micrometer metrics plus a Blackbox Exporter probe in Prometheus.
• Trigger multiwindow burn-rate alerts when the error budget burns too quickly. Capture detection plus recovery timing in a repeatable incident scorecard. Gate releases with four policy states: NORMAL, CAUTION, FREEZE, or EXHAUSTED. Preserve the core lab in a clean Git history with an annotated v1.0.0 tag.
• Complete the optional Secret Mission by scoring a 600 ms latency-only incident. Compare trackwise:journey_latency_p95_seconds:rate5m with probe_duration_seconds to connect server latency with probe latency. Confirm that successful slow responses leave the request-based availability error ratio near zero. Preserve that conclusion as committed evidence after the core v1.0.0 release.

Ready to quiz yourself?