Deploy, Monitor, and Break a Distributed System

Introduction

30 Second Summary

Every time you open your favorite app, there's an entire team making sure it stays online. Site Reliability Engineers are the people who define "how reliable is reliable enough" and build the systems to prove it.

In this project, you will deploy an 11-service e-commerce application to a local Kubernetes cluster, build a full observability stack, define reliability targets, and run chaos experiments to test how the system handles failure.

What You'll Build

A complete SRE reliability platform running on your local machine that combines Infrastructure as Code, observability, and chaos engineering.

By the end of this project, you'll have:

• A Terraform project that deploys an 11-microservice application and its entire monitoring stack to a local Kubernetes cluster.
• A Grafana dashboard tracking SLIs (latency, error rate, availability) against defined SLO targets, powered by Prometheus metrics and Loki logs.
• Chaos experiments that simulate real production failures and incident runbooks documenting how to detect, diagnose, and remediate each failure scenario.
• Secret Mission: An automated SLO burn-rate alerting system that detects when your error budget is being consumed too fast.

Are there any prerequisites?

You need a Mac with Apple Silicon (M-series chip) and Homebrew installed. No prior Kubernetes or Terraform experience is required.

Before We Start

Before We Start...

Install Your SRE Toolkit

Before you can deploy microservices and monitor their reliability, you need the right tools on your machine.

In this step, you'll install three core SRE tools: OrbStack (which gives you Docker, kubectl, and a local Kubernetes cluster), Terraform (for Infrastructure as Code), and Helm (for deploying packaged Kubernetes applications).

In this step, get ready to:

• Install OrbStack and verify the Kubernetes cluster is running.
• Install Terraform and Helm via Homebrew and verify versions.
• Confirm kubectl can communicate with the OrbStack Kubernetes cluster.

Install OrbStack and start the Kubernetes cluster

OrbStack is a lightweight container runtime for macOS that bundles Docker, kubectl, and a single-node Kubernetes cluster (based on k3s) into one fast package.

• Open your terminal by pressing Cmd+Space (macOS) and typing Terminal, then press Enter.
• Install OrbStack by running this command:

brew install orbstack

What does this command do?

Homebrew installs OrbStack as a macOS app that provides Docker, kubectl, and a lightweight Kubernetes cluster. No separate Docker Desktop or kubectl installation needed.

• OrbStack may ask for admin permissions to install command-line tools and configure networking. Click Allow if prompted.
• Start the Kubernetes cluster by running:

orb start k8s

• Verify the cluster is running by running:

kubectl get nodes

You should see a single node with a Ready status. This confirms your Kubernetes cluster is operational.

Don't see a Ready node?

If you get a connection refused error, wait 10-15 seconds for the cluster to finish starting, then try kubectl get nodes again.

You can check if OrbStack is running by using orb status. If the cluster is not running, run orb start k8s again.

help me troubleshoot why my OrbStack Kubernetes cluster isn't starting

Why OrbStack instead of Docker Desktop?

OrbStack starts in about 2 seconds, uses significantly less CPU and battery, and makes all Kubernetes service types (ClusterIP, NodePort, LoadBalancer) accessible directly from your Mac without port-forwarding.

This is critical for this project since you'll access Grafana, Prometheus, and the Online Boutique frontend directly from your browser.

Install Terraform and Helm via Homebrew

Terraform lets you define your infrastructure in code files rather than clicking through UIs. Helm is a package manager for Kubernetes that bundles complex applications into reusable charts.

• Add the HashiCorp tap and install Terraform by running these commands:

brew tap hashicorp/tap
brew install hashicorp/tap/terraform

What is a Homebrew tap?

A tap is a third-party repository of Homebrew formulae. HashiCorp maintains their own tap so you always get the official, latest version of Terraform directly from them.

• Verify your Terraform version by running:

terraform version

✔️ I see version 1.15 or higher

Terraform is ready to go. Move on to the Helm installation below.

ⓧ I see an older version

You have an older Terraform version installed. Upgrade it by running:

brew upgrade hashicorp/tap/terraform

• Run terraform version again to confirm you now see 1.15 or higher.

ⓧ Command not found

The install may not have completed. Re-run the tap and install commands:

brew tap hashicorp/tap
brew install hashicorp/tap/terraform

• Open a new terminal window and run terraform version again.

• Install Helm by running:

brew install helm

• Verify your Helm version by running:

helm version

✔️ I see version 3.21 or higher

Helm is ready. You now have both infrastructure tools installed.

ⓧ I see an older version

Upgrade Helm to the latest version by running:

brew upgrade helm

• Run helm version again to confirm you see 3.21 or higher.

ⓧ Command not found

Re-run the install command:

brew install helm

• Open a new terminal window and run helm version again.

Having trouble with Homebrew installs?

If Homebrew itself seems outdated, run brew update first to refresh its package index, then retry the install commands.

help me troubleshoot why Terraform or Helm won't install via Homebrew on my Mac

Verify the full toolkit works together

The final check confirms that kubectl (bundled with OrbStack) can talk to your Kubernetes cluster. This is the same connection Terraform will use in the next step.

• Check cluster connectivity by running:

kubectl cluster-info

You should see output showing the Kubernetes control plane is running at a local address. This confirms your kubeconfig (~/.kube/config) is properly configured by OrbStack.

• Confirm the system pods are healthy by running:

kubectl get pods -n kube-system

You should see several pods (like coredns and metrics-server) all in Running status.

Seeing connection refused or no resources found?

Make sure OrbStack is running. You can check with orb status. If it shows stopped, run orb start k8s again.

If kubectl reports it cannot find a valid kubeconfig, OrbStack automatically writes one to ~/.kube/config when the cluster starts. Restart OrbStack and try again.

help me troubleshoot kubectl not connecting to my OrbStack Kubernetes cluster

Your SRE toolkit is locked and loaded. Next up, you'll use Terraform to deploy an entire 11-service microservices application to your cluster.

Deploy Online Boutique with Terraform

Your cluster is running, but an empty cluster isn't useful. You need services to observe, break, and fix.

In this step, you'll use Terraform to deploy Google's Online Boutique to your local cluster. Online Boutique is an 11-microservice e-commerce app that communicates over gRPC. It gives you a realistic distributed system to practice SRE on.

In this step, get ready to:

• Create a Terraform project with Kubernetes, Helm, and kubectl providers.
• Download and deploy Online Boutique's 11 microservices.
• Verify all pods are running and access the storefront in your browser.

Create the Terraform project structure

Terraform needs provider configuration to know which APIs it's managing. You'll configure three providers: kubernetes for namespaces, helm for chart deployments (used in a later step), and kubectl for raw YAML manifests.

• In your sre-platform project folder in Google Antigravity, create a new file called main.tf.
• Add the Terraform block and provider configuration by pasting this code into main.tf:

terraform {
  required_version = ">= 1.15.0"

  required_providers {
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.17"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.36"
    }
    kubectl = {
      source  = "alekc/kubectl"
      version = ">= 2.0.0"
    }
  }
}

provider "kubernetes" {
  config_path = "~/.kube/config"
}

provider "helm" {
  kubernetes {
    config_path = "~/.kube/config"
  }
}

provider "kubectl" {
  config_path = "~/.kube/config"
}

What does this code do?

The terraform block pins the minimum Terraform version and declares three providers from the Terraform Registry.

Each provider block points at ~/.kube/config. This is the kubeconfig file that OrbStack automatically populated when you started the Kubernetes cluster. All three providers read the same config to authenticate with your local cluster.

Why three providers?

The kubernetes provider handles native resources like namespaces. The helm provider installs Helm charts (you'll use this for Prometheus and Grafana in Step 3). The kubectl provider applies raw YAML manifests, which is how you'll deploy Online Boutique without needing to convert its manifests to HCL.

• Save main.tf.

Download and deploy Online Boutique

Online Boutique publishes a single YAML file containing all 11 Deployments and Services. You'll download it into your project so Terraform can apply it.

• Create a manifests directory inside your sre-platform folder and download the manifest by running these commands in your terminal:

mkdir -p manifests
curl -o manifests/kubernetes-manifests.yaml https://raw.githubusercontent.com/GoogleCloudPlatform/microservices-demo/v0.10.5/release/kubernetes-manifests.yaml

What does this command do?

This downloads the official v0.10.5 release manifest from Google's microservices-demo repository. The file contains Deployment and Service definitions for all 11 microservices, including a load generator that simulates user traffic.

• Confirm the file was downloaded by running:

ls manifests/

• You should see kubernetes-manifests.yaml in the output.

Now you need Terraform resources that create the Kubernetes namespaces and deploy the manifest. You'll create three namespaces upfront: one for Online Boutique, one for monitoring (Step 3), and one for chaos engineering (Step 5).

• Add the namespace and deployment resources to the bottom of main.tf by pasting this code below the provider blocks:

resource "kubernetes_namespace" "online_boutique" {
  metadata {
    name = "online-boutique"
  }
}

resource "kubernetes_namespace" "monitoring" {
  metadata {
    name = "monitoring"
  }
}

resource "kubernetes_namespace" "chaos_mesh" {
  metadata {
    name = "chaos-mesh"
  }
}

resource "kubectl_manifest" "online_boutique" {
  for_each = toset(split("---", file("${path.module}/manifests/kubernetes-manifests.yaml")))
  yaml_body = each.value

  depends_on = [kubernetes_namespace.online_boutique]
}

What does this code do?

The three kubernetes_namespace resources create isolated namespaces for each major component of your SRE platform.

The kubectl_manifest resource splits the downloaded YAML file on --- delimiters (standard YAML document separators) and applies each document as a separate Kubernetes resource. The depends_on ensures the namespace exists before any resources are created inside it.

• Save main.tf.

✔️ Awesome, I've got everything!

Great. Double check you've saved main.tf before moving on.

ⓧ I'd like to double check the full code

Here is the complete main.tf at this point:

terraform {
  required_version = ">= 1.15.0"

  required_providers {
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.17"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.36"
    }
    kubectl = {
      source  = "alekc/kubectl"
      version = ">= 2.0.0"
    }
  }
}

provider "kubernetes" {
  config_path = "~/.kube/config"
}

provider "helm" {
  kubernetes {
    config_path = "~/.kube/config"
  }
}

provider "kubectl" {
  config_path = "~/.kube/config"
}

resource "kubernetes_namespace" "online_boutique" {
  metadata {
    name = "online-boutique"
  }
}

resource "kubernetes_namespace" "monitoring" {
  metadata {
    name = "monitoring"
  }
}

resource "kubernetes_namespace" "chaos_mesh" {
  metadata {
    name = "chaos-mesh"
  }
}

resource "kubectl_manifest" "online_boutique" {
  for_each = toset(split("---", file("${path.module}/manifests/kubernetes-manifests.yaml")))
  yaml_body = each.value

  depends_on = [kubernetes_namespace.online_boutique]
}

Run terraform init, plan, and apply

Terraform uses a three-step workflow: init downloads providers, plan previews changes, and apply executes them against your cluster.

• Initialize Terraform and download the required providers by running:

terraform init

• You should see output ending with Terraform has been successfully initialized! along with messages about downloading the three providers.

Seeing a provider error?

Make sure you saved main.tf before running the command. Also confirm your terminal is in the sre-platform directory where main.tf lives.

help me fix a Terraform init error in my SRE project

• Preview what Terraform will create by running:

terraform plan

• You should see a plan summary showing resources to add (3 namespaces plus the manifest resources). Review the output to confirm no errors.
• Apply the plan and deploy everything to your cluster by running:

terraform apply

• When Terraform prompts Do you want to perform these actions?, type yes and press Enter.

How long does this take?

The first apply takes 2-5 minutes. Terraform creates the namespaces immediately, then the kubectl_manifest resources start applying. Each microservice pod needs to pull its container image, which is the bulk of the wait time.

• Once Terraform finishes, verify all 11 pods are running by executing:

kubectl get pods -n online-boutique

• You should see 11 pods with a STATUS of Running. It may take a few minutes for all pods to reach the Running state as images are pulled.

Waiting for pods to start

If some pods show ContainerCreating or Pending, wait 30 seconds and run the kubectl get pods command again. OrbStack needs time to pull the container images for the first time.

• Access the Online Boutique frontend in your browser. OrbStack makes Kubernetes services accessible via their cluster.local domain. Get the frontend service URL by running:

kubectl get svc frontend-external -n online-boutique

• Open your browser and navigate to http://frontend-external.online-boutique.svc.cluster.local. You should see the Online Boutique storefront with products displayed.

How is this accessible without port forwarding?

OrbStack routes all Kubernetes cluster.local domains directly to your Mac. This means you can reach any Kubernetes service by its internal DNS name without running kubectl port-forward. This is unique to OrbStack and makes local development significantly smoother.

Can't load the storefront?

Run kubectl get pods -n online-boutique again and confirm all pods show Running. If the frontend pod is in CrashLoopBackOff, check its logs with kubectl logs -n online-boutique deployment/frontend.

help me debug why I can't access the Online Boutique frontend in my browser

Your 11-service distributed system is live. Next up, you'll add full observability with Prometheus, Grafana, and Loki so you can actually see what's happening inside it.

Deploy the Observability Stack

Your 11-service distributed system is running, but right now you're completely blind to what's happening inside it. If a service crashes or latency spikes, you won't know until something visibly breaks.

In this step, you'll deploy a full observability stack into your cluster. Prometheus collects metrics from every pod, Grafana visualizes them on dashboards, and Loki aggregates logs from every container.

In this step, get ready to:

• Deploy Prometheus and Grafana via the kube-prometheus-stack Helm chart.
• Deploy Loki and Promtail for centralized log collection.
• Access Grafana and verify both data sources are working.

Add the Prometheus and Grafana Helm release

The kube-prometheus-stack Helm chart bundles Prometheus, Grafana, Alertmanager, and pre-built dashboards into a single deployment. You'll configure it as a helm_release resource in Terraform so your entire observability stack is managed declaratively.

• Switch back to main.tf in your editor.
• Add the following helm_release resource at the bottom of the file, after the kubectl_manifest resource:

resource "helm_release" "kube_prometheus_stack" {
  name             = "prometheus"
  namespace        = "monitoring"
  repository       = "https://prometheus-community.github.io/helm-charts"
  chart            = "kube-prometheus-stack"
  create_namespace = false

  # Set the Grafana admin password
  set {
    name  = "grafana.adminPassword"
    value = "sre-admin"
  }

  # Allow Prometheus to discover all ServiceMonitors, not just Helm-managed ones
  set {
    name  = "prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues"
    value = "false"
  }

  depends_on = [kubernetes_namespace.monitoring]
}

What does this code do?

This helm_release resource tells Terraform to install the kube-prometheus-stack chart from the prometheus-community Helm repository into your monitoring namespace.

• grafana.adminPassword sets the Grafana login password to sre-admin so you can access dashboards.
• serviceMonitorSelectorNilUsesHelmValues = false tells Prometheus to scrape metrics from all ServiceMonitors across all namespaces, not just ones labeled by this Helm release.
• depends_on ensures the monitoring namespace exists before the chart is installed.

• Save main.tf.

Add the Loki and Promtail Helm release

Metrics tell you what happened, but logs tell you why. Loki stores logs from every container, and Promtail acts as the agent that ships those logs from each node into Loki.

• Add the following helm_release resource at the bottom of main.tf, after the kube_prometheus_stack resource you just added:

resource "helm_release" "loki" {
  name             = "loki"
  namespace        = "monitoring"
  repository       = "https://grafana.github.io/helm-charts"
  chart            = "loki-stack"
  create_namespace = false

  # Enable the Promtail log collector
  set {
    name  = "promtail.enabled"
    value = "true"
  }

  # Disable authentication for local development
  set {
    name  = "loki.auth_enabled"
    value = "false"
  }

  depends_on = [kubernetes_namespace.monitoring]
}

What does this code do?

This installs the loki-stack chart from Grafana's Helm repository.

• promtail.enabled = true deploys a Promtail DaemonSet on every node. Promtail tails container logs and pushes them to Loki.
• loki.auth_enabled = false disables multi-tenancy authentication since this is a local development cluster.

• Save main.tf.

✔️ Awesome, I've got everything!

Great. Double check you've saved main.tf before continuing.

ⓧ I'd like to double check the full code

Here's the full main.tf you should have at this point:

terraform {
  required_version = ">= 1.15.0"

  required_providers {
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.17"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.36"
    }
    kubectl = {
      source  = "alekc/kubectl"
      version = ">= 2.0.0"
    }
  }
}

provider "kubernetes" {
  config_path = "~/.kube/config"
}

provider "helm" {
  kubernetes {
    config_path = "~/.kube/config"
  }
}

provider "kubectl" {
  config_path = "~/.kube/config"
}

resource "kubernetes_namespace" "online_boutique" {
  metadata {
    name = "online-boutique"
  }
}

resource "kubernetes_namespace" "monitoring" {
  metadata {
    name = "monitoring"
  }
}

resource "kubernetes_namespace" "chaos_mesh" {
  metadata {
    name = "chaos-mesh"
  }
}

resource "kubectl_manifest" "online_boutique" {
  for_each = toset(split("---", file("${path.module}/manifests/kubernetes-manifests.yaml")))
  yaml_body = each.value

  depends_on = [kubernetes_namespace.online_boutique]
}

resource "helm_release" "kube_prometheus_stack" {
  name             = "prometheus"
  namespace        = "monitoring"
  repository       = "https://prometheus-community.github.io/helm-charts"
  chart            = "kube-prometheus-stack"
  create_namespace = false

  set {
    name  = "grafana.adminPassword"
    value = "sre-admin"
  }

  set {
    name  = "prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues"
    value = "false"
  }

  depends_on = [kubernetes_namespace.monitoring]
}

resource "helm_release" "loki" {
  name             = "loki"
  namespace        = "monitoring"
  repository       = "https://grafana.github.io/helm-charts"
  chart            = "loki-stack"
  create_namespace = false

  set {
    name  = "promtail.enabled"
    value = "true"
  }

  set {
    name  = "loki.auth_enabled"
    value = "false"
  }

  depends_on = [kubernetes_namespace.monitoring]
}

Apply Terraform and verify the observability stack

Now you'll deploy both Helm releases in one shot with terraform apply. This installs Prometheus, Grafana, Loki, Promtail, and all their supporting components into your cluster.

• Apply the updated Terraform configuration by running:

terraform apply -auto-approve

What should I expect?

This command will take 2-5 minutes to complete. Terraform downloads the Helm charts and deploys all the monitoring pods. You'll see output showing 2 new resources being created (the two helm_release resources).

• Verify all monitoring pods are running by running:

kubectl get pods -n monitoring

You should see pods for Prometheus, Grafana, Alertmanager, node-exporter, kube-state-metrics, Loki, and Promtail all in Running or Ready state.

Seeing pods stuck in Pending or CrashLoopBackOff?

• Run kubectl describe pod <pod-name> -n monitoring to see the specific error. Common causes include insufficient memory or image pull issues.
• Make sure OrbStack is running and has enough resources allocated (at least 4GB RAM recommended for this stack).
• If pods are in ImagePullBackOff, check your internet connection. The Helm charts pull container images from public registries.

Help me troubleshoot my monitoring pods

• Open a port-forward to Grafana by running:

kubectl port-forward svc/prometheus-grafana -n monitoring 3000:80

Why port-forward?

Grafana's Kubernetes service uses ClusterIP by default, which means it's only reachable from inside the cluster. Port-forwarding creates a tunnel from your Mac's localhost:3000 to the Grafana service's port 80 inside the cluster.

• Open your browser and go to http://localhost:3000.
• Log in with username admin and password sre-admin.

You should see the Grafana home screen with pre-built Kubernetes dashboards already loaded.

• Verify Prometheus is scraping metrics by clicking the hamburger menu icon on the left sidebar.
• Click Explore.
• In the data source dropdown at the top, confirm Prometheus is selected.
• Type up in the query field and click Run query.

You should see a list of targets with value 1, confirming Prometheus is actively scraping your cluster.

• Now add Loki as a data source. Click the gear icon in the left sidebar to open Administration.
• Click Data sources.
• Click Add data source.
• Search for and select Loki.
• In the URL field, enter http://loki:3100.
• Scroll down and click Save & test.

You should see a green banner confirming the data source is connected and labels are found.

• Verify logs are flowing by clicking Explore again.
• Switch the data source dropdown to Loki.
• Enter the query {namespace="online-boutique"} and click Run query.

You should see a stream of logs from your Online Boutique services. This confirms Promtail is collecting logs from every container and shipping them to Loki.

Not seeing any Loki logs?

• Make sure you entered the URL as http://loki:3100 (not https). Loki runs without TLS inside the cluster.
• Check that Promtail pods are running with kubectl get pods -n monitoring -l app=promtail. You should see one Promtail pod per node.
• If the data source test fails, wait another minute. Loki may still be starting up.

Help me troubleshoot my Loki data source connection

Your observability stack is live. Next up, you'll define the reliability targets that an SRE would own for this system and build a dashboard to track them.

Define SLIs and SLOs

Your observability stack is live. Prometheus is scraping metrics, Grafana is ready for dashboards, and Loki is collecting logs from every pod.

Collecting data is only the first half of the SRE job. The real value comes from defining what "reliable" means for your system. That answer lives in three concepts: SLIs (what you measure), SLOs (the targets you commit to), and error budgets (how much failure is acceptable before you act).

In this step, get ready to:

• Define three SLIs and their SLO targets for the frontend service.
• Build a Grafana dashboard with PromQL queries that visualize each SLI.
• Write a Python script that calculates SLO burn rates from live Prometheus data.

Define your SLIs and SLO targets

An SLI (Service Level Indicator) is a quantitative measurement of one aspect of your service's reliability. An SLO (Service Level Objective) is the target value you commit to for that SLI.

For the Online Boutique frontend, you will define three SLIs that cover the most critical user-facing dimensions of reliability.

• Request latency (p99) measures how long the slowest 1% of requests take. SLO target: p99 latency under 500ms.
• Error rate measures the proportion of requests that return 5xx status codes. SLO target: error rate below 1%.
• Availability measures the proportion of requests that succeed (non-5xx). SLO target: availability above 99.5%.

What is an error budget?

If your availability SLO is 99.5%, your error budget is the remaining 0.5%. That budget represents how much failure your system is allowed before you must stop feature work and focus on reliability.

Burn rate measures how fast you are consuming that budget. A burn rate of 1x means you will exhaust the budget exactly at the end of the SLO window. A burn rate of 6x means you will burn through it 6 times faster than allowed.

Create a Grafana dashboard with PromQL queries

Now you will turn those SLI definitions into live queries that pull real data from Prometheus. You will create a Grafana dashboard with one panel per SLI.

• Open a new terminal tab and forward the Prometheus service to localhost by running this command:

kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090

Why port-forward to localhost?

OrbStack makes Kubernetes .svc.cluster.local DNS names accessible from your Mac. However, the Python script you will write later uses http://localhost:9090 for simplicity. Port-forwarding maps that address to the Prometheus pod inside the cluster.

This command runs in the foreground, so keep this terminal tab open.

• Verify Prometheus is responding by opening http://localhost:9090 in your browser.
• You should see the Prometheus web UI with a query input field at the top.

Port 9090 already in use?

If you see a binding error, another process may be using port 9090. Stop that process or choose a different local port (e.g. 9091:9090). If you change the port, remember to adjust the URL in your Python script later.

Help me troubleshoot the Prometheus port-forward command failing

• Switch back to your Grafana tab in the browser (login with admin / sre-admin).
• Click Dashboards in the left-side menu.
• Click New and select New Dashboard.
• Click the blue plus icon to add a panel, then select Configure visualization.

The panel editor opens with the default Prometheus data source preselected.

• In the query editor toolbar, click the Code tab to switch to PromQL text mode.
• Paste the following query for the error rate SLI:

sum(rate(http_server_requests_seconds_count{namespace="online-boutique",status=~"5.."}[5m])) / sum(rate(http_server_requests_seconds_count{namespace="online-boutique"}[5m]))

What does this query do?

The numerator counts 5xx responses per second over 5 minutes. The denominator counts all responses. Dividing them gives you the fraction of requests that failed.

• Click Run query to execute it.
• In the panel title field on the right sidebar, enter Error Rate (SLI).
• Click Save to save the panel.

• Add a second panel by clicking the blue plus icon again, then Configure visualization.
• Switch to Code mode and paste the p99 latency query:

histogram_quantile(0.99, sum(rate(http_server_requests_seconds_bucket{namespace="online-boutique"}[5m])) by (le))

• Click Run query.
• Title this panel p99 Latency (SLI) and click Save.

• Add a third panel. Switch to Code mode and paste the availability query:

1 - (sum(rate(http_server_requests_seconds_count{namespace="online-boutique",status=~"5.."}[5m])) / sum(rate(http_server_requests_seconds_count{namespace="online-boutique"}[5m])))

• Click Run query.
• Title this panel Availability (SLI) and click Save.
• Save the dashboard by entering SLI/SLO Dashboard as the title, then click Save.

You should see three time-series panels on your dashboard. If the Online Boutique loadgenerator is running, the panels will show live data.

Panels showing no data?

If the Online Boutique has not received traffic yet, the metrics may not exist. Check that the loadgenerator pod is running with kubectl get pods -n online-boutique. It generates synthetic traffic automatically.

Also confirm that the Prometheus data source is correctly selected in each panel's query editor.

Help me troubleshoot my Grafana SLI dashboard

Write a Python SLO burn rate monitor

Dashboards give you a visual picture, but SREs also need automated monitoring that can calculate whether the error budget is being consumed too quickly. You will write a Python script that queries Prometheus directly and outputs burn rate status to your terminal.

• Create a new file called slo_monitor.py in your sre-platform/ project folder.
• Add the imports and configuration constants by pasting this code:

"""SLO Burn Rate Monitor - Queries Prometheus and calculates error budget consumption."""

import time
import urllib.request
import urllib.parse
import json
import sys

PROMETHEUS_URL = "http://prometheus-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090"
SLO_TARGET_AVAILABILITY = 0.995  # 99.5%
SLO_TARGET_LATENCY_P99_MS = 500
CHECK_INTERVAL_SECONDS = 30

# Use localhost since OrbStack makes ClusterIPs accessible from Mac
PROMETHEUS_URL = "http://localhost:9090"


def query_prometheus(query):
    """Execute a PromQL instant query and return the result."""
    url = f"{PROMETHEUS_URL}/api/v1/query"
    params = urllib.parse.urlencode({"query": query})
    full_url = f"{url}?{params}"

    try:
        with urllib.request.urlopen(full_url, timeout=10) as response:
            data = json.loads(response.read().decode())
            if data["status"] == "success" and data["data"]["result"]:
                return float(data["data"]["result"][0]["value"][1])
    except Exception as e:
        print(f"  [WARN] Query failed: {e}")
    return None

What does this code do?

The constants define your SLO targets (99.5% availability, 500ms p99 latency) and the polling interval (30 seconds). The PROMETHEUS_URL is overridden to http://localhost:9090 because the port-forward you set up earlier makes Prometheus reachable from your Mac.

The query_prometheus() function sends a PromQL query to Prometheus's HTTP API and extracts the numeric result from the JSON response.

• Add the SLI calculation functions below the query_prometheus function:

def calculate_error_rate(window="5m"):
    """Calculate the error rate over a given window."""
    query = (
        f'sum(rate(http_server_requests_seconds_count{{namespace="online-boutique",'
        f'status=~"5.."}}[{window}])) / '
        f'sum(rate(http_server_requests_seconds_count{{namespace="online-boutique"}}[{window}]))'
    )
    return query_prometheus(query)


def calculate_latency_p99(window="5m"):
    """Calculate p99 latency over a given window."""
    query = (
        f'histogram_quantile(0.99, sum(rate(http_server_requests_seconds_bucket'
        f'{{namespace="online-boutique"}}[{window}])) by (le))'
    )
    result = query_prometheus(query)
    if result is not None:
        return result * 1000  # Convert to milliseconds
    return None

What does this code do?

These are the same PromQL queries you used in Grafana, wrapped in Python functions. The window parameter lets you query over different time ranges (5 minutes for fast burn, 1 hour for slow burn).

The latency function multiplies by 1000 to convert from seconds (how Prometheus stores it) to milliseconds (how you defined the SLO target).

• Add the burn rate evaluation functions:

def calculate_burn_rate(error_rate, slo_target):
    """Calculate burn rate: how fast the error budget is being consumed."""
    if error_rate is None:
        return None
    error_budget = 1 - slo_target
    if error_budget == 0:
        return float("inf")
    return error_rate / error_budget


def get_status_color(burn_rate):
    """Return status based on burn rate thresholds."""
    if burn_rate is None:
        return "UNKNOWN", "\033[90m"
    if burn_rate < 1.0:
        return "GREEN", "\033[92m"
    elif burn_rate < 6.0:
        return "YELLOW", "\033[93m"
    else:
        return "RED", "\033[91m"

What does this code do?

The calculate_burn_rate() function divides the current error rate by the error budget (0.005 for a 99.5% SLO). A result of 1.0 means you are consuming budget at exactly the sustainable rate. Above 1.0 means trouble.

The get_status_color() function maps burn rate values to colored terminal output. GREEN means within budget, YELLOW means elevated, and RED means budget exhaustion is imminent.

• Add the main monitoring loop and entry point:

def main():
    """Main monitoring loop."""
    reset_color = "\033[0m"
    print("=" * 60)
    print("  SLO Burn Rate Monitor")
    print(f"  Target Availability: {SLO_TARGET_AVAILABILITY * 100}%")
    print(f"  Target p99 Latency: {SLO_TARGET_LATENCY_P99_MS}ms")
    print(f"  Polling every {CHECK_INTERVAL_SECONDS}s")
    print("=" * 60)

    while True:
        print(f"\n[{time.strftime('%H:%M:%S')}] Checking SLOs...")

        # Fast burn (5-minute window)
        error_rate_5m = calculate_error_rate("5m")
        burn_rate_5m = calculate_burn_rate(error_rate_5m, SLO_TARGET_AVAILABILITY)
        status_5m, color_5m = get_status_color(burn_rate_5m)

        # Slow burn (1-hour window)
        error_rate_1h = calculate_error_rate("1h")
        burn_rate_1h = calculate_burn_rate(error_rate_1h, SLO_TARGET_AVAILABILITY)
        status_1h, color_1h = get_status_color(burn_rate_1h)

        # Latency
        latency_p99 = calculate_latency_p99("5m")
        latency_status = "OK" if latency_p99 and latency_p99 < SLO_TARGET_LATENCY_P99_MS else "BREACH"

        print(f"  {color_5m}[{status_5m}]{reset_color} Fast burn (5m): "
              f"rate={burn_rate_5m:.2f}x" if burn_rate_5m else f"  [--] Fast burn: no data")
        print(f"  {color_1h}[{status_1h}]{reset_color} Slow burn (1h): "
              f"rate={burn_rate_1h:.2f}x" if burn_rate_1h else f"  [--] Slow burn: no data")
        print(f"  Latency p99: {latency_p99:.0f}ms [{latency_status}]" if latency_p99 else "  Latency: no data")

        time.sleep(CHECK_INTERVAL_SECONDS)


if __name__ == "__main__":
    main()

What does this code do?

The main() function runs an infinite loop that checks SLOs every 30 seconds. It uses two time windows following Google's SRE methodology: a 5-minute "fast burn" window that catches rapid failures, and a 1-hour "slow burn" window that filters out brief spikes.

Each iteration prints colored status lines so you can tell at a glance whether your error budget is safe.

• Save slo_monitor.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved slo_monitor.py in your sre-platform/ folder.

ⓧ I'd like to double check the full code

"""SLO Burn Rate Monitor - Queries Prometheus and calculates error budget consumption."""

import time
import urllib.request
import urllib.parse
import json
import sys

PROMETHEUS_URL = "http://prometheus-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090"
SLO_TARGET_AVAILABILITY = 0.995  # 99.5%
SLO_TARGET_LATENCY_P99_MS = 500
CHECK_INTERVAL_SECONDS = 30

# Use localhost since OrbStack makes ClusterIPs accessible from Mac
PROMETHEUS_URL = "http://localhost:9090"


def query_prometheus(query):
    """Execute a PromQL instant query and return the result."""
    url = f"{PROMETHEUS_URL}/api/v1/query"
    params = urllib.parse.urlencode({"query": query})
    full_url = f"{url}?{params}"

    try:
        with urllib.request.urlopen(full_url, timeout=10) as response:
            data = json.loads(response.read().decode())
            if data["status"] == "success" and data["data"]["result"]:
                return float(data["data"]["result"][0]["value"][1])
    except Exception as e:
        print(f"  [WARN] Query failed: {e}")
    return None


def calculate_error_rate(window="5m"):
    """Calculate the error rate over a given window."""
    query = (
        f'sum(rate(http_server_requests_seconds_count{{namespace="online-boutique",'
        f'status=~"5.."}}[{window}])) / '
        f'sum(rate(http_server_requests_seconds_count{{namespace="online-boutique"}}[{window}]))'
    )
    return query_prometheus(query)


def calculate_latency_p99(window="5m"):
    """Calculate p99 latency over a given window."""
    query = (
        f'histogram_quantile(0.99, sum(rate(http_server_requests_seconds_bucket'
        f'{{namespace="online-boutique"}}[{window}])) by (le))'
    )
    result = query_prometheus(query)
    if result is not None:
        return result * 1000  # Convert to milliseconds
    return None


def calculate_burn_rate(error_rate, slo_target):
    """Calculate burn rate: how fast the error budget is being consumed."""
    if error_rate is None:
        return None
    error_budget = 1 - slo_target
    if error_budget == 0:
        return float("inf")
    return error_rate / error_budget


def get_status_color(burn_rate):
    """Return status based on burn rate thresholds."""
    if burn_rate is None:
        return "UNKNOWN", "\033[90m"
    if burn_rate < 1.0:
        return "GREEN", "\033[92m"
    elif burn_rate < 6.0:
        return "YELLOW", "\033[93m"
    else:
        return "RED", "\033[91m"


def main():
    """Main monitoring loop."""
    reset_color = "\033[0m"
    print("=" * 60)
    print("  SLO Burn Rate Monitor")
    print(f"  Target Availability: {SLO_TARGET_AVAILABILITY * 100}%")
    print(f"  Target p99 Latency: {SLO_TARGET_LATENCY_P99_MS}ms")
    print(f"  Polling every {CHECK_INTERVAL_SECONDS}s")
    print("=" * 60)

    while True:
        print(f"\n[{time.strftime('%H:%M:%S')}] Checking SLOs...")

        # Fast burn (5-minute window)
        error_rate_5m = calculate_error_rate("5m")
        burn_rate_5m = calculate_burn_rate(error_rate_5m, SLO_TARGET_AVAILABILITY)
        status_5m, color_5m = get_status_color(burn_rate_5m)

        # Slow burn (1-hour window)
        error_rate_1h = calculate_error_rate("1h")
        burn_rate_1h = calculate_burn_rate(error_rate_1h, SLO_TARGET_AVAILABILITY)
        status_1h, color_1h = get_status_color(burn_rate_1h)

        # Latency
        latency_p99 = calculate_latency_p99("5m")
        latency_status = "OK" if latency_p99 and latency_p99 < SLO_TARGET_LATENCY_P99_MS else "BREACH"

        print(f"  {color_5m}[{status_5m}]{reset_color} Fast burn (5m): "
              f"rate={burn_rate_5m:.2f}x" if burn_rate_5m else f"  [--] Fast burn: no data")
        print(f"  {color_1h}[{status_1h}]{reset_color} Slow burn (1h): "
              f"rate={burn_rate_1h:.2f}x" if burn_rate_1h else f"  [--] Slow burn: no data")
        print(f"  Latency p99: {latency_p99:.0f}ms [{latency_status}]" if latency_p99 else "  Latency: no data")

        time.sleep(CHECK_INTERVAL_SECONDS)


if __name__ == "__main__":
    main()

• Run the monitor by executing this command in a new terminal tab (make sure you are in the sre-platform/ folder):

python3 slo_monitor.py

You should see the monitor print a header with your SLO targets, then start polling every 30 seconds with colored status output.

Seeing connection refused or no data?

Make sure the Prometheus port-forward is still running in your other terminal tab. If it was interrupted, re-run the kubectl port-forward command from earlier in this step.

If you see [--] Fast burn: no data on every poll, the Online Boutique may not have generated enough traffic yet. Wait 2-3 minutes for the loadgenerator to produce data, then check again.

Help me debug my SLO monitor connection issues

• Press Ctrl+C to stop the monitor when you are done observing the output.

Your SLIs are defined, your dashboard is live, and your burn rate monitor can tell you in real-time whether reliability is holding. Next up, you will inject chaos into the system and watch these metrics react.

Inject Chaos with Chaos Mesh

You have observability and SLO targets defined. But how do you know your system actually holds up under real failure conditions? The answer is you don't, until you break it on purpose.

In this step, you'll deploy Chaos Mesh to your cluster and run controlled chaos engineering experiments. You'll kill pods, inject network latency, and watch your SLI dashboard light up in response.

In this step, get ready to:

• Deploy Chaos Mesh to your Kubernetes cluster via Terraform.
• Run a pod-kill experiment against the recommendation service.
• Inject network latency and observe the impact on your SLO dashboard.

Deploy Chaos Mesh via Terraform

Chaos Mesh uses CRDs (Custom Resource Definitions) to define chaos experiments as native Kubernetes objects. Before you can create experiments, you need the Chaos Mesh operator running in your cluster.

OrbStack runs a k3s-based Kubernetes distribution under the hood. This means the container runtime socket lives at a non-standard path that Chaos Mesh needs to know about.

• In Google Antigravity, open main.tf.
• Scroll to the bottom of the file, below the helm_release.loki resource.
• Add the Chaos Mesh Helm release by pasting this code at the end of the file:

resource "helm_release" "chaos_mesh" {
  name             = "chaos-mesh"
  namespace        = "chaos-mesh"
  repository       = "https://charts.chaos-mesh.org"
  chart            = "chaos-mesh"
  version          = "2.8.2"
  create_namespace = false

  set {
    name  = "chaosDaemon.runtime"
    value = "containerd"
  }

  set {
    name  = "chaosDaemon.socketPath"
    value = "/run/k3s/containerd/containerd.sock"
  }

  depends_on = [kubernetes_namespace.chaos_mesh]
}

What does this code do?

This tells Terraform to install Chaos Mesh version 2.8.2 from the official Helm chart repository into the chaos-mesh namespace.

The two set blocks configure the Chaos Daemon (the component that injects faults at the container level) to use the containerd runtime and the k3s-specific socket path /run/k3s/containerd/containerd.sock. Without these, Chaos Mesh cannot interact with your containers to kill pods or inject network faults.

• Save main.tf.
• Apply the new resource by running this command in your terminal:

terraform apply -auto-approve

• Verify the Chaos Mesh pods are running by running:

kubectl get pods -n chaos-mesh -l app.kubernetes.io/instance=chaos-mesh

You should see three pods (controller-manager, chaos-daemon, and chaos-dashboard) all in Running status.

Pods stuck in CrashLoopBackOff?

If the chaos-daemon pod is crashing, the most likely cause is an incorrect socket path. Double check that chaosDaemon.socketPath is set to exactly /run/k3s/containerd/containerd.sock in your main.tf.

Also confirm your OrbStack Kubernetes cluster is running. You can check with kubectl cluster-info.

Help me debug my Chaos Mesh pods

✔️ Awesome, I've got everything!

Great. Double check you've saved main.tf and all three Chaos Mesh pods are running.

ⓧ I'd like to double check the full code

terraform {
  required_version = ">= 1.15.0"

  required_providers {
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.17"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.36"
    }
    kubectl = {
      source  = "alekc/kubectl"
      version = ">= 2.0.0"
    }
  }
}

provider "kubernetes" {
  config_path = "~/.kube/config"
}

provider "helm" {
  kubernetes {
    config_path = "~/.kube/config"
  }
}

provider "kubectl" {
  config_path = "~/.kube/config"
}

resource "kubernetes_namespace" "online_boutique" {
  metadata {
    name = "online-boutique"
  }
}

resource "kubernetes_namespace" "monitoring" {
  metadata {
    name = "monitoring"
  }
}

resource "kubernetes_namespace" "chaos_mesh" {
  metadata {
    name = "chaos-mesh"
  }
}

resource "kubectl_manifest" "online_boutique" {
  for_each = toset(split("---", file("${path.module}/manifests/kubernetes-manifests.yaml")))
  yaml_body = each.value

  depends_on = [kubernetes_namespace.online_boutique]
}

resource "helm_release" "kube_prometheus_stack" {
  name             = "prometheus"
  namespace        = "monitoring"
  repository       = "https://prometheus-community.github.io/helm-charts"
  chart            = "kube-prometheus-stack"
  create_namespace = false

  set {
    name  = "grafana.adminPassword"
    value = "sre-admin"
  }

  set {
    name  = "prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues"
    value = "false"
  }

  depends_on = [kubernetes_namespace.monitoring]
}

resource "helm_release" "loki" {
  name             = "loki"
  namespace        = "monitoring"
  repository       = "https://grafana.github.io/helm-charts"
  chart            = "loki-stack"
  create_namespace = false

  set {
    name  = "promtail.enabled"
    value = "true"
  }

  set {
    name  = "loki.auth_enabled"
    value = "false"
  }

  depends_on = [kubernetes_namespace.monitoring]
}

resource "helm_release" "chaos_mesh" {
  name             = "chaos-mesh"
  namespace        = "chaos-mesh"
  repository       = "https://charts.chaos-mesh.org"
  chart            = "chaos-mesh"
  version          = "2.8.2"
  create_namespace = false

  set {
    name  = "chaosDaemon.runtime"
    value = "containerd"
  }

  set {
    name  = "chaosDaemon.socketPath"
    value = "/run/k3s/containerd/containerd.sock"
  }

  depends_on = [kubernetes_namespace.chaos_mesh]
}

Run a pod-kill chaos experiment

Your first chaos experiment kills a specific pod on a schedule. This simulates a process crash that Kubernetes must recover from automatically. In production, this tests whether your service mesh and replica configuration can absorb a single-instance failure without user impact.

• Create the experiments directory by running:

mkdir -p chaos-experiments

• Confirm the directory exists by running:

ls

You should see chaos-experiments listed alongside your other project files.

• Create a new file called pod-kill.yaml inside the chaos-experiments directory.
• Paste the following PodChaos experiment definition into the file:

apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: recommendation-pod-kill
  namespace: online-boutique
spec:
  action: pod-kill
  mode: one
  selector:
    namespaces:
      - online-boutique
    labelSelectors:
      app: recommendationservice
  scheduler:
    cron: "*/1 * * * *"

What does this experiment do?

• kind: PodChaos tells Chaos Mesh this is a pod-level fault injection.
• action: pod-kill terminates the target pod immediately (like a process crash).
• The selector block targets only pods with the label app: recommendationservice in the online-boutique namespace.
• The scheduler block uses a cron expression to kill the pod every 1 minute, creating repeated disruptions.

• Save chaos-experiments/pod-kill.yaml.
• Apply the experiment to your cluster by running:

kubectl apply -f chaos-experiments/pod-kill.yaml

You should see podchaos.chaos-mesh.org/recommendation-pod-kill created in the output.

• Watch the recommendation service pod restart by running:

kubectl get pods -n online-boutique -l app=recommendationservice -w

Within a minute, you'll see the pod transition through Terminating and then a new pod appear in Running status. Press Ctrl+C to stop watching.

• Switch to your Grafana SLI dashboard in the browser.
• Observe the error rate panel. You should see brief spikes each time the pod is killed and traffic hits the missing service before Kubernetes restarts it.

Why target the recommendation service?

The recommendation service is a non-critical dependency of the frontend. Killing it tests graceful degradation. The frontend should still load, but the recommendations section will briefly show errors or empty results.

This is a realistic scenario. In production, non-critical services crash all the time. The question is whether your system handles it gracefully or cascades the failure.

Pod not getting killed?

Make sure the label selector app: recommendationservice matches what the Online Boutique deployment actually uses. Verify with:

kubectl get pods -n online-boutique --show-labels

Look for the app=recommendationservice label in the output. If labels don't match, update your YAML selector accordingly.

Help me debug my PodChaos experiment

✔️ Awesome, I've got everything!

Great. Make sure you've saved your file and the experiment is actively killing pods.

ⓧ I'd like to double check the full code

apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: recommendation-pod-kill
  namespace: online-boutique
spec:
  action: pod-kill
  mode: one
  selector:
    namespaces:
      - online-boutique
    labelSelectors:
      app: recommendationservice
  scheduler:
    cron: "*/1 * * * *"

Inject network latency between services

Pod kills test hard failures. Network latency tests something subtler and often more dangerous: degraded performance. A service that responds slowly can cause cascading timeouts across the entire call chain.

• Create a new file called network-latency.yaml inside the chaos-experiments directory.
• Paste the following NetworkChaos experiment definition into the file:

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: frontend-latency
  namespace: online-boutique
spec:
  action: delay
  mode: all
  selector:
    namespaces:
      - online-boutique
    labelSelectors:
      app: productcatalogservice
  delay:
    latency: "500ms"
    correlation: "100"
    jitter: "50ms"
  direction: to
  target:
    selector:
      namespaces:
        - online-boutique
      labelSelectors:
        app: frontend
    mode: all
  duration: "5m"

What does this experiment do?

• kind: NetworkChaos injects network-level faults between pods.
• action: delay adds artificial latency to network packets.
• The selector targets the productcatalogservice pods, and direction: to with the target block means the delay applies to traffic flowing from product catalog to the frontend.
• The delay is 500ms with 50ms of jitter. correlation: "100" means every single packet gets delayed (100% correlation with the previous packet's delay decision).
• duration: "5m" runs the experiment for 5 minutes and then automatically cleans up.

• Save chaos-experiments/network-latency.yaml.
• Apply the network latency experiment by running:

kubectl apply -f chaos-experiments/network-latency.yaml

You should see networkchaos.chaos-mesh.org/frontend-latency created in the output.

• Switch to your Grafana SLI dashboard in the browser.
• Watch the p99 latency panel. Within 1-2 minutes, you should see the latency spike well above 500ms as the injected delay propagates through the product catalog calls.
• Check your slo_monitor.py terminal output. The latency status should show BREACH and the burn rate should increase.

Why is latency injection more dangerous than pod kills?

A killed pod triggers an immediate failure that Kubernetes can detect and restart within seconds. A slow pod is worse. It holds connections open, causes upstream callers to wait, and often triggers cascading timeouts across multiple services.

This is why latency SLOs (p99 < 500ms) are critical. They catch the slow-bleed degradation that availability metrics alone miss.

Latency panel not spiking?

The experiment targets traffic from productcatalogservice to frontend. If you don't see a spike, confirm the experiment is active:

kubectl get networkchaos -n online-boutique

You should see frontend-latency listed. If the experiment expired (duration is 5 minutes), re-apply it.

Help me debug my NetworkChaos experiment

✔️ Awesome, I've got everything!

Great. Make sure you've observed both the latency spike in Grafana and the burn rate change in your SLO monitor.

ⓧ I'd like to double check the full code

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: frontend-latency
  namespace: online-boutique
spec:
  action: delay
  mode: all
  selector:
    namespaces:
      - online-boutique
    labelSelectors:
      app: productcatalogservice
  delay:
    latency: "500ms"
    correlation: "100"
    jitter: "50ms"
  direction: to
  target:
    selector:
      namespaces:
        - online-boutique
      labelSelectors:
        app: frontend
    mode: all
  duration: "5m"

You've just proven something critical about your system. You know exactly how it breaks and how long it takes to recover. Next up, you'll capture this knowledge permanently in incident runbooks.

Write Incident Runbooks

You've now proven your system breaks in predictable ways. You've seen pods crash and latency spike on your Grafana dashboard. But an SRE's job isn't just to break things. It's to make sure anyone on the team can detect, diagnose, and fix problems at 3am without you.

That's what runbooks are for. A runbook is a structured document that maps a specific failure scenario to the exact steps needed to resolve it. In this step, you'll write runbooks for each chaos experiment you ran, then tie everything together in a reliability matrix.

In this step, get ready to:

• Write a runbook for recommendation service pod crashes.
• Write a runbook for elevated latency on the product catalog service.
• Create a reliability matrix mapping SLOs to chaos experiments to runbooks.

Write the recommendation service runbook

The first runbook covers the failure you tested with your PodChaos experiment. When the recommendation service crashes, an on-call engineer needs to know exactly what to look for and how to fix it.

• Create the runbooks directory inside your sre-platform folder by running this command:

mkdir runbooks

• Confirm the directory exists by running:

ls

You should see runbooks listed alongside chaos-experiments, main.tf, and slo_monitor.py.

• Create a new file called recommendation-service-unavailable.md inside the runbooks folder.
• Paste the following content into the file:

# Runbook: Recommendation Service Unavailable

## Alert Trigger

- PodChaos experiment (`pod-kill.yaml`) kills the recommendationservice pod every 60 seconds
- Error rate SLI breaches the 1% SLO threshold on the Grafana dashboard
- `slo_monitor.py` reports YELLOW or RED burn rate status

## Investigation Steps

### 1. Check pod status

```bash
kubectl get pods -n online-boutique -l app=recommendationservice
```

Look for: `CrashLoopBackOff`, `Error`, or `0/1 Ready` states.

### 2. Describe the pod for events

```bash
kubectl describe pod -n online-boutique -l app=recommendationservice
```

Look for: OOMKilled events, failed liveness probes, or image pull errors in the Events section.

### 3. Check container logs

```bash
kubectl logs -n online-boutique -l app=recommendationservice --tail=50
```

Look for: stack traces, connection refused errors to upstream services, or resource exhaustion messages.

### 4. Check Grafana error rate panel

Open Grafana > SLI/SLO Dashboard > Error Rate panel. Look for a spike correlating with the pod restart timestamps.

### 5. Search Loki logs

In Grafana > Explore > Loki, run:

```
{namespace="online-boutique", app="recommendationservice"} |= "error"
```

Look for: repeated error patterns, connection timeouts, or panic messages.

## Mitigation Actions

### Immediate: Restart the deployment

```bash
kubectl rollout restart deployment/recommendationservice -n online-boutique
```

### Verify recovery

```bash
kubectl get pods -n online-boutique -l app=recommendationservice -w
```

Wait until the pod shows `1/1 Running` and the READY column is stable.

## Root Cause Categories

| Category | Indicators | Next Steps |
|----------|-----------|-------------|
| Chaos experiment active | PodChaos CR exists in namespace | Delete the PodChaos resource |
| OOM Kill | OOMKilled in pod events | Increase memory limits in deployment spec |
| Dependency failure | Connection refused in logs | Check upstream services (productcatalogservice) |
| Image corruption | ImagePullBackOff in events | Verify image digest, force re-pull |

What does this runbook cover?

This runbook follows the standard SRE format: alert trigger (how you know something is wrong), investigation steps (how to find the root cause), mitigation (how to stop the bleeding), and root cause categories (what caused it and what to do next).

Each investigation step references a specific tool you already have running. The kubectl commands check cluster state. The Grafana panel shows the SLI impact. The Loki query surfaces application-level errors.

• Save recommendation-service-unavailable.md.

Runbooks are living documents

In production, you'd update this runbook every time you learn something new from an incident. Each post-mortem adds another row to the root cause table or refines an investigation step.

Write the elevated latency runbook

The second runbook covers the NetworkChaos scenario. Latency issues are trickier than pod crashes because the service is still running. It just responds slowly enough to breach your SLO.

• Create a new file called elevated-latency-product-catalog.md inside the runbooks folder.
• Paste the following content into the file:

# Runbook: Elevated Latency on Product Catalog

## SLO Breach Detection

- Grafana SLI/SLO Dashboard: p99 latency panel exceeds 500ms threshold
- `slo_monitor.py` output shows YELLOW (burn rate 1-6x) or RED (burn rate > 6x)
- Frontend pages load noticeably slower, product listings take multiple seconds

## Investigation Steps

### 1. Confirm the latency spike in Grafana

Open Grafana > SLI/SLO Dashboard > p99 Latency panel. Note the time the spike began and its magnitude.

### 2. Test network connectivity from frontend to productcatalogservice

```bash
kubectl exec -n online-boutique deploy/frontend -- wget -qO- --timeout=5 http://productcatalogservice:3550/ 2>&1 | head -5
```

Look for: response times > 500ms or connection timeout messages.

### 3. Check for active NetworkChaos experiments

```bash
kubectl get networkchaos -n online-boutique
```

If a NetworkChaos resource named `frontend-latency` is active, it is injecting artificial delay.

### 4. Inspect the NetworkChaos spec

```bash
kubectl describe networkchaos frontend-latency -n online-boutique
```

Look for: the `delay.latency` value, the `duration` field, and whether the experiment has expired.

### 5. Check slo_monitor.py burn rate

Run `python3 slo_monitor.py` and observe the fast burn (5m) rate. A rate above 1.0x means the error budget is being consumed faster than planned.

## Mitigation Actions

### If NetworkChaos is active: Remove the experiment

```bash
kubectl delete networkchaos frontend-latency -n online-boutique
```

Latency should return to normal within 30 seconds.

### If no chaos experiment is active: Check node resources

```bash
kubectl top pods -n online-boutique --sort-by=cpu
```

Look for pods consuming excessive CPU which could cause request queuing.

## Escalation Procedures

| Severity | Condition | Action |
|----------|-----------|--------|
| P3 - Low | p99 > 500ms for < 5 minutes | Monitor, likely transient |
| P2 - Medium | p99 > 500ms for > 5 minutes | Page on-call SRE, begin investigation |
| P1 - High | p99 > 2000ms or complete timeout | Page SRE + service owner, consider rollback |

### Escalation contacts

- Primary on-call: SRE team rotation
- Service owner: Product Catalog team
- Infrastructure: Platform Engineering team

Why include escalation procedures?

Latency issues often cross team boundaries. The SRE detects the problem, but the fix might require the service owner to optimize a query or the platform team to scale infrastructure. Escalation tables make this handoff explicit so nobody wastes time guessing who to contact.

• Save elevated-latency-product-catalog.md.

Not sure what to put in a runbook?

A good test: could someone who has never seen this system before follow your runbook at 3am and resolve the issue? If a step requires tribal knowledge, expand it with the exact command and what output to expect.

Help me understand what makes a good SRE runbook.

Create the reliability matrix

The reliability matrix ties your entire SRE project together. It maps each SLO to the chaos experiment that validates it and the runbook that remediates failures. This is the single document an SRE manager reviews to understand operational coverage.

• Create a new file called reliability-matrix.md inside the runbooks folder.
• Paste the following content into the file:

# Reliability Matrix

Maps each SLO to the chaos experiment that validates resilience and the runbook that guides remediation.

## SLO Coverage

| SLO | Target | Chaos Experiment | Runbook |
|-----|--------|------------------|---------|
| Availability | > 99.5% | `chaos-experiments/pod-kill.yaml` | `runbooks/recommendation-service-unavailable.md` |
| Error Rate | < 1% | `chaos-experiments/pod-kill.yaml` | `runbooks/recommendation-service-unavailable.md` |
| p99 Latency | < 500ms | `chaos-experiments/network-latency.yaml` | `runbooks/elevated-latency-product-catalog.md` |

## Experiment Details

| Experiment | Type | Target Service | Failure Mode |
|-----------|------|----------------|--------------|
| `pod-kill.yaml` | PodChaos | recommendationservice | Pod terminated every 60s |
| `network-latency.yaml` | NetworkChaos | productcatalogservice -> frontend | 500ms delay injected |

## Monitoring Tools

| Tool | Purpose | Access |
|------|---------|--------|
| Grafana | SLI/SLO dashboards, log exploration | http://localhost:3000 (admin/sre-admin) |
| Prometheus | Metric storage and PromQL queries | http://localhost:9090 |
| Loki | Log aggregation and search | Via Grafana Explore |
| `slo_monitor.py` | Burn rate calculation and alerting | `python3 slo_monitor.py` |

## Gap Analysis

| Gap | Risk | Recommended Next Step |
|-----|------|-----------------------|
| No disk I/O chaos | Storage failures unvalidated | Add IOChaos experiment for cartservice (Redis) |
| No DNS failure test | Service discovery failures unvalidated | Add DNSChaos for inter-service resolution |
| Single-service latency only | Cascading latency untested | Add multi-hop latency injection |

What does the reliability matrix show?

The matrix answers three questions at a glance. First, which SLOs are protected by chaos experiments? Second, which runbook handles each failure scenario? Third, where are the gaps in coverage?

The gap analysis section is what separates a good SRE from a great one. It shows you're thinking proactively about risks you haven't tested yet.

• Save reliability-matrix.md.
• Confirm all three runbook files exist by running:

ls runbooks/

You should see elevated-latency-product-catalog.md, recommendation-service-unavailable.md, and reliability-matrix.md listed.

Missing files in the runbooks directory?

Make sure you created all three files inside the runbooks/ directory, not in the sre-platform/ root. Check your current directory with pwd to confirm you're in the right location.

Help me find where my runbook files ended up.

✔️ Awesome, I've got everything!

Great. Make sure all three files are saved in your runbooks/ directory.

ⓧ I'd like to double check the full code

Here are all three runbook files for comparison.

# Runbook: Recommendation Service Unavailable

## Alert Trigger

- PodChaos experiment (`pod-kill.yaml`) kills the recommendationservice pod every 60 seconds
- Error rate SLI breaches the 1% SLO threshold on the Grafana dashboard
- `slo_monitor.py` reports YELLOW or RED burn rate status

## Investigation Steps

### 1. Check pod status

```bash
kubectl get pods -n online-boutique -l app=recommendationservice
```

Look for: `CrashLoopBackOff`, `Error`, or `0/1 Ready` states.

### 2. Describe the pod for events

```bash
kubectl describe pod -n online-boutique -l app=recommendationservice
```

Look for: OOMKilled events, failed liveness probes, or image pull errors in the Events section.

### 3. Check container logs

```bash
kubectl logs -n online-boutique -l app=recommendationservice --tail=50
```

Look for: stack traces, connection refused errors to upstream services, or resource exhaustion messages.

### 4. Check Grafana error rate panel

Open Grafana > SLI/SLO Dashboard > Error Rate panel. Look for a spike correlating with the pod restart timestamps.

### 5. Search Loki logs

In Grafana > Explore > Loki, run:

```
{namespace="online-boutique", app="recommendationservice"} |= "error"
```

Look for: repeated error patterns, connection timeouts, or panic messages.

## Mitigation Actions

### Immediate: Restart the deployment

```bash
kubectl rollout restart deployment/recommendationservice -n online-boutique
```

### Verify recovery

```bash
kubectl get pods -n online-boutique -l app=recommendationservice -w
```

Wait until the pod shows `1/1 Running` and the READY column is stable.

## Root Cause Categories

| Category | Indicators | Next Steps |
|----------|-----------|-------------|
| Chaos experiment active | PodChaos CR exists in namespace | Delete the PodChaos resource |
| OOM Kill | OOMKilled in pod events | Increase memory limits in deployment spec |
| Dependency failure | Connection refused in logs | Check upstream services (productcatalogservice) |
| Image corruption | ImagePullBackOff in events | Verify image digest, force re-pull |

# Runbook: Elevated Latency on Product Catalog

## SLO Breach Detection

- Grafana SLI/SLO Dashboard: p99 latency panel exceeds 500ms threshold
- `slo_monitor.py` output shows YELLOW (burn rate 1-6x) or RED (burn rate > 6x)
- Frontend pages load noticeably slower, product listings take multiple seconds

## Investigation Steps

### 1. Confirm the latency spike in Grafana

Open Grafana > SLI/SLO Dashboard > p99 Latency panel. Note the time the spike began and its magnitude.

### 2. Test network connectivity from frontend to productcatalogservice

```bash
kubectl exec -n online-boutique deploy/frontend -- wget -qO- --timeout=5 http://productcatalogservice:3550/ 2>&1 | head -5
```

Look for: response times > 500ms or connection timeout messages.

### 3. Check for active NetworkChaos experiments

```bash
kubectl get networkchaos -n online-boutique
```

If a NetworkChaos resource named `frontend-latency` is active, it is injecting artificial delay.

### 4. Inspect the NetworkChaos spec

```bash
kubectl describe networkchaos frontend-latency -n online-boutique
```

Look for: the `delay.latency` value, the `duration` field, and whether the experiment has expired.

### 5. Check slo_monitor.py burn rate

Run `python3 slo_monitor.py` and observe the fast burn (5m) rate. A rate above 1.0x means the error budget is being consumed faster than planned.

## Mitigation Actions

### If NetworkChaos is active: Remove the experiment

```bash
kubectl delete networkchaos frontend-latency -n online-boutique
```

Latency should return to normal within 30 seconds.

### If no chaos experiment is active: Check node resources

```bash
kubectl top pods -n online-boutique --sort-by=cpu
```

Look for pods consuming excessive CPU which could cause request queuing.

## Escalation Procedures

| Severity | Condition | Action |
|----------|-----------|--------|
| P3 - Low | p99 > 500ms for < 5 minutes | Monitor, likely transient |
| P2 - Medium | p99 > 500ms for > 5 minutes | Page on-call SRE, begin investigation |
| P1 - High | p99 > 2000ms or complete timeout | Page SRE + service owner, consider rollback |

### Escalation contacts

- Primary on-call: SRE team rotation
- Service owner: Product Catalog team
- Infrastructure: Platform Engineering team

# Reliability Matrix

Maps each SLO to the chaos experiment that validates resilience and the runbook that guides remediation.

## SLO Coverage

| SLO | Target | Chaos Experiment | Runbook |
|-----|--------|------------------|---------|
| Availability | > 99.5% | `chaos-experiments/pod-kill.yaml` | `runbooks/recommendation-service-unavailable.md` |
| Error Rate | < 1% | `chaos-experiments/pod-kill.yaml` | `runbooks/recommendation-service-unavailable.md` |
| p99 Latency | < 500ms | `chaos-experiments/network-latency.yaml` | `runbooks/elevated-latency-product-catalog.md` |

## Experiment Details

| Experiment | Type | Target Service | Failure Mode |
|-----------|------|----------------|--------------|
| `pod-kill.yaml` | PodChaos | recommendationservice | Pod terminated every 60s |
| `network-latency.yaml` | NetworkChaos | productcatalogservice -> frontend | 500ms delay injected |

## Monitoring Tools

| Tool | Purpose | Access |
|------|---------|--------|
| Grafana | SLI/SLO dashboards, log exploration | http://localhost:3000 (admin/sre-admin) |
| Prometheus | Metric storage and PromQL queries | http://localhost:9090 |
| Loki | Log aggregation and search | Via Grafana Explore |
| `slo_monitor.py` | Burn rate calculation and alerting | `python3 slo_monitor.py` |

## Gap Analysis

| Gap | Risk | Recommended Next Step |
|-----|------|-----------------------|
| No disk I/O chaos | Storage failures unvalidated | Add IOChaos experiment for cartservice (Redis) |
| No DNS failure test | Service discovery failures unvalidated | Add DNSChaos for inter-service resolution |
| Single-service latency only | Cascading latency untested | Add multi-hop latency injection |

Your SRE platform is now complete. You have a deployed distributed system, full observability, defined SLOs, chaos experiments that validate resilience, and operational runbooks that document how to respond. This is the exact stack an SRE team operates in production.

Secret mission

Automated SLO Burn-Rate Alerting

Your SLO monitor shows you the current state, but a real SRE platform doesn't wait for a human to watch a terminal. Build an automated alerting system that detects when your error budget is burning too fast and fires a webhook notification, just like PagerDuty or Slack would in production.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely on your local machine using OrbStack's Kubernetes cluster. There are no cloud resources or ongoing costs to worry about.

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

• OrbStack Kubernetes cluster (running Online Boutique, Prometheus, Grafana, Loki, Chaos Mesh).
• Terraform state and configuration files in the sre-platform/ folder on your Desktop.
• Python scripts (slo_monitor.py, slo_monitor_alerting.py), chaos experiment YAMLs, and runbook Markdown files.
• Installed tools: OrbStack, Terraform, and Helm (via Homebrew).

Keep everything running

No action needed. Choose this if you're still actively experimenting with chaos engineering or want to revisit your dashboards later.

• Your OrbStack cluster continues running in the background with minimal resource usage.
• Grafana, Prometheus, and the Online Boutique frontend remain accessible at their local addresses.
• Your Terraform state is preserved, so you can modify and re-apply infrastructure changes at any time.

Pause - I'll come back to this later

Stop the Kubernetes cluster to free up CPU and memory. Your project files and Terraform state stay on disk so you can resume later.

• Stop the OrbStack Kubernetes cluster by running:
• Stop any running Python monitor scripts by pressing Ctrl+C in those terminal windows.

orb stop k8s

When you're ready to resume, start the cluster again by running:

orb start k8s

All your Kubernetes resources (deployments, services, Helm releases) will come back automatically.

Delete - I don't want to use this again

Remove all project resources and start fresh. This tears down everything you built in this project.

• Stop any running Python scripts by pressing Ctrl+C in those terminal windows.
• Navigate to your project folder and destroy all Terraform-managed resources by running:

cd ~/Desktop/sre-platform
terraform destroy -auto-approve

• Delete the project folder from your Desktop by running:

cd ~/Desktop
rm -rf sre-platform

• Stop and reset the OrbStack Kubernetes cluster by running:

orb stop k8s

• If you want to fully uninstall OrbStack, Terraform, and Helm, run:

brew uninstall helm
brew uninstall terraform
brew uninstall --zap orbstack

What does --zap do?

The --zap flag tells Homebrew to remove OrbStack's application data and configuration files in addition to the app itself. Without it, leftover data remains in ~/.orbstack.

Nice Work!

Nice Work!

Nice work! You just built a production-grade SRE reliability platform from scratch on your local machine.

You've learned how to:

• Deployed an 11-service distributed system using Terraform and Kubernetes, managing the entire infrastructure declaratively through code.
• Built a full observability stack with Prometheus, Grafana, and Loki, then defined and tracked SLIs/SLOs using PromQL and a custom Python burn-rate calculator.
• Ran controlled chaos experiments with Chaos Mesh to validate system resilience, then wrote incident runbooks documenting detection, diagnosis, and remediation for each failure scenario.
• Built an automated SLO burn-rate monitor with multi-window detection and webhook alerting that fires when error budget is consumed too quickly.

Ready to quiz yourself?