Observability and SLOs on Kubernetes

Introduction

30 Second Summary

When something breaks in production, most teams find out from angry users or a spike in support tickets. What if your system could tell you it was heading toward trouble before anyone noticed, based on math instead of gut feeling?

In this project, you will build a complete observability stack that collects metrics from your application, defines what "good enough" means as an explicit SLO, and fires alerts only when your error budget is being consumed at an unsustainable rate.

What You'll Build

You hit your app's /error endpoint, watch the error budget stat panel count down in real time on a Grafana dashboard, and explain exactly why the fast-burn alert uses two windows and what each multiplier means mathematically.

By the end of this project, you'll have:

• A Python app instrumented with Prometheus metrics that exposes request count and latency on a /metrics endpoint, scraped automatically every 15 seconds by a GitOps-managed monitoring stack.
• Two multi-window burn-rate alerts derived from first principles that fire only when error budget consumption is mathematically unsustainable. You can explain the exact threshold calculations in an interview.
• A Grafana dashboard provisioned as code showing request rate, error rate, p95 latency, and error budget remaining as a live stat panel that counts down like a fuel gauge.
• Secret Mission: Deploy Loki for centralized log collection so you can correlate log spikes with metric anomalies.

Are there any prerequisites?

This project builds on top of a running k3d cluster with ArgoCD installed and a demo app deployed via GitOps. You need to have completed Project 1 in this series before starting.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters for the platform you're assembling.

Your k3d cluster is running and ArgoCD is reconciling your demo app, but the platform is flying blind. This project gives it eyes: instrumented metrics, automated scraping, SLO-based alerts that fire only when budget burn is unsustainable, and a Grafana dashboard that shows exactly how much error budget remains.

Instrument the Demo App

Your demo app is running on the cluster, but it is completely invisible to any monitoring system. Without a /metrics endpoint, Prometheus has nothing to scrape.

In this step, you will instrument the app with prometheus-client to emit two raw signals: a request counter (broken down by status code) and a latency histogram. These are the foundation for every alert and dashboard you build in this project.

In this step, get ready to:

• Add prometheus-client and instrument the demo app with Counter and Histogram metrics.
• Rebuild the container image and deploy the new version via ArgoCD.
• Verify /metrics returns Prometheus exposition format with live metric data.

Replace the demo app with an instrumented version

You need three files in your app/ directory: a requirements file declaring the dependency, a Dockerfile to build the image, and the instrumented Python code itself.

• Create app/requirements.txt with the prometheus-client dependency by adding this content:

prometheus-client==0.25.0

• Create app/Dockerfile to build a slim Python image with the dependency pre-installed:

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py .
EXPOSE 8080
CMD ["python", "app.py"]

What does this Dockerfile do?

It starts from a minimal Python 3.12 image, installs the dependencies declared in requirements.txt first (so Docker caches that layer), then copies the app code.

The EXPOSE 8080 documents the port the app listens on, and CMD tells the container to run your Python server on startup.

• Confirm both files exist by running:

ls app/requirements.txt app/Dockerfile

You should see both file paths printed without errors.

Now create the main application file. app/app.py is the instrumented HTTP server that exposes Prometheus metrics alongside your application endpoints.

• Start app/app.py with the imports and metric definitions:

from http.server import HTTPServer, BaseHTTPRequestHandler
from prometheus_client import Counter, Histogram, generate_latest, CONTENT_TYPE_LATEST
import time
import random
import json

REQUEST_COUNT = Counter(
    'http_requests_total',
    'Total HTTP requests by method, endpoint, and status',
    ['method', 'endpoint', 'status']
)

REQUEST_LATENCY = Histogram(
    'http_request_duration_seconds',
    'HTTP request latency in seconds',
    ['method', 'endpoint'],
    buckets=[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0]
)

What do these metrics do?

REQUEST_COUNT is a Counter that only goes up. Each HTTP request increments it with labels for method, endpoint, and status code. This lets you later compute error rates with PromQL.

REQUEST_LATENCY is a Histogram that sorts each request's duration into pre-defined time buckets. Prometheus uses these buckets to compute percentiles like p95 without storing every individual value.

• Below the metric definitions, add the request handler class with the /metrics, /healthz, and / endpoints:

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        start = time.time()

        if self.path == '/metrics':
            output = generate_latest()
            self.send_response(200)
            self.send_header('Content-Type', CONTENT_TYPE_LATEST)
            self.end_headers()
            self.wfile.write(output)
            return

        if self.path == '/healthz':
            self._respond(200, {'status': 'ok'}, '/healthz', start)
            return

        if self.path == '/':
            self._respond(200, {'message': 'Hello from the demo app'}, '/', start)
            return

Why is /metrics handled differently?

The /metrics endpoint uses generate_latest() to serialize all registered metrics into Prometheus text exposition format. It bypasses the _respond helper because you do not want scrape requests counted in your application metrics.

The /healthz endpoint gives Kubernetes liveness probes something to hit. The root / endpoint represents normal application traffic.

• Continue the do_GET method by adding the simulation endpoints for slow responses and errors:

        if self.path == '/slow':
            delay = random.uniform(1.0, 3.0)
            time.sleep(delay)
            self._respond(
                200,
                {'message': 'slow response', 'delay_seconds': round(delay, 2)},
                '/slow',
                start
            )
            return

        if self.path == '/error':
            self._respond(500, {'error': 'intentional failure'}, '/error', start)
            return

        self._respond(404, {'error': 'not found'}, self.path, start)

What are these simulation endpoints for?

/slow introduces a random 1-3 second delay. This gives the latency histogram something meaningful to measure. You will use it later to test p95 dashboard panels.

/error always returns a 500 status code. This endpoint is your controlled way to burn error budget and trigger alerts in later steps.

• Finish the file by adding the _respond helper method, the log silencer, and the server entrypoint:

    def _respond(self, status_code, body, endpoint, start):
        REQUEST_COUNT.labels(
            method='GET', endpoint=endpoint, status=str(status_code)
        ).inc()
        REQUEST_LATENCY.labels(
            method='GET', endpoint=endpoint
        ).observe(time.time() - start)
        self.send_response(status_code)
        self.send_header('Content-Type', 'application/json')
        self.end_headers()
        self.wfile.write(json.dumps(body).encode())

    def log_message(self, format, *args):
        pass


if __name__ == '__main__':
    server = HTTPServer(('0.0.0.0', 8080), Handler)
    print('Demo app listening on :8080')
    server.serve_forever()

What does _respond do?

Every application endpoint routes through _respond. It increments the request counter with the correct labels, records the latency observation, then sends the JSON response.

Centralising the metric logic in one helper ensures no endpoint forgets to record its metrics. The log_message override silences the default noisy HTTP access logs so they do not clutter your pod output.

• Save app/app.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved all three files: app/requirements.txt, app/Dockerfile, and app/app.py.

ⓧ I'd like to double check the full code

Scroll up and confirm each of these chunks is present in app/app.py, in order:

• Imports and metric definitions (REQUEST_COUNT Counter, REQUEST_LATENCY Histogram).
• Handler class with do_GET handling /metrics, /healthz, and /.
• Simulation endpoints: /slow (random delay), /error (500), and the 404 fallback.
• The _respond helper, log_message silencer, and if __name__ == '__main__' entrypoint.

For app/requirements.txt, confirm a single line: prometheus-client==0.25.0.

For app/Dockerfile, confirm it builds from python:3.12-slim, installs requirements, copies app.py, exposes port 8080, and runs python app.py.

Seeing a syntax error when you save?

Make sure the indentation in _respond uses 4 spaces per level, matching the rest of the class. Python is sensitive to mixed tabs and spaces.

Check that the /slow block's multi-line self._respond( call has its closing parenthesis aligned correctly.

help me debug a syntax error in my instrumented Python app

Rebuild and deploy via ArgoCD

With the source files ready, you need to build a new container image, load it into your k3d cluster, and let ArgoCD reconcile the updated deployment.

• Build the container image by running:

docker build -t demo-app:instrumented ./app

You should see Docker pull the python:3.12-slim base image, install prometheus-client, and finish with a success message.

Build failing on COPY app.py?

Confirm app.py is in the same directory as Dockerfile (both inside app/). Docker's build context is ./app, so both files must be there.

help me fix a Docker build error for my demo app image

• Load the image into your k3d cluster by running:

k3d image import demo-app:instrumented -c [[K3D_CLUSTER="your k3d cluster name"]]

What does k3d image import do?

Your k3d cluster runs containers inside Docker. By default, those containers cannot see images on your host's Docker daemon. The k3d image import command copies the image into the cluster's internal container runtime so your pods can pull it.

• Update your deployment manifest in your GitOps repo to reference the new image tag demo-app:instrumented. Set imagePullPolicy: Never so Kubernetes uses the locally imported image instead of trying to pull from a registry.
• Commit and push the changes to your Git repository.

git add .
git commit -m "Instrument demo app with prometheus-client"
git push

• Wait for ArgoCD to detect and sync the change. Verify the new pod is running:

kubectl get pods -l app=demo-app

You should see a pod with status Running and a recent age (under a minute if the sync just completed).

Pod stuck in ImagePullBackOff?

This means Kubernetes is trying to pull from a remote registry instead of using the local image. Confirm your deployment manifest has imagePullPolicy: Never and the image tag matches exactly: demo-app:instrumented.

Also confirm you ran k3d image import targeting the correct cluster name. Run k3d cluster list to check your cluster's name.

help me fix an ImagePullBackOff error after loading a local image into k3d

Verify /metrics returns Prometheus exposition format

The app is deployed, but you have not yet confirmed it actually serves metrics. Time to hit the endpoint and see raw Prometheus data.

• Start a port-forward to reach the demo app from your local machine:

kubectl port-forward deploy/demo-app 8080:8080

Port-forward runs in the foreground

This command takes over your terminal. Open a new terminal tab or window for the next commands. When you are done testing, press Ctrl+C to stop the forward.

Before you run the next command, what do you expect curl localhost:8080/metrics to return? Think about the format Prometheus expects.

• In your new terminal, query the metrics endpoint:

curl localhost:8080/metrics

You should see Prometheus text exposition format: lines starting with # HELP and # TYPE followed by metric names like http_requests_total and http_request_duration_seconds_bucket.

• Now generate some traffic to populate the metrics. Hit all three application endpoints:

curl localhost:8080/
curl localhost:8080/slow
curl localhost:8080/error

• Query /metrics again to see the counters and histogram populated with real observations:

curl localhost:8080/metrics

You should now see http_requests_total lines with status 200 and 500, and http_request_duration_seconds_bucket lines with non-zero counts in the higher buckets from the /slow request.

Seeing 'Connection refused' on curl?

Make sure the port-forward is still running in your other terminal. If you see error: unable to forward port, the pod may have restarted. Run kubectl get pods -l app=demo-app to confirm the pod is in Running state, then re-run the port-forward command.

help me fix a connection refused error when curling my port-forwarded demo app

Your app is now emitting metrics on every request. Prometheus has a target to scrape. Next up, you will deploy the full monitoring stack and wire Prometheus to discover this endpoint automatically.

Deploy kube-prometheus-stack via ArgoCD

Your demo app exposes raw metrics on /metrics, but nothing is collecting them. Without a time-series database to scrape and store those metrics, they vanish the moment the request ends.

You need three components: Prometheus to scrape and store metrics, Grafana to visualize them, and Alertmanager to route alerts. The kube-prometheus-stack Helm chart bundles all three into a single install. Deploying it as an ArgoCD Application means the monitoring stack itself is GitOps-managed and self-heals if someone edits it manually.

In this step, get ready to:

• Create an ArgoCD Application manifest for the kube-prometheus-stack Helm chart.
• Commit the manifest and let ArgoCD deploy the monitoring stack.
• Verify Prometheus is scraping built-in targets and confirm your demo app is not yet visible.

Create the ArgoCD Application manifest

ArgoCD uses Application resources to describe what to deploy and where. You will point one at the kube-prometheus-stack Helm chart with custom values that disable components k3d cannot expose.

• Create a new file at argocd/kube-prometheus-stack.yaml in your GitOps repository.
• Paste the first section of the manifest, which defines the Application metadata and chart source:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: kube-prometheus-stack
  namespace: argocd
spec:
  project: default
  source:
    chart: kube-prometheus-stack
    repoURL: https://prometheus-community.github.io/helm-charts
    targetRevision: 88.1.3
    helm:
      valuesObject:
        kubeEtcd:
          enabled: false
        kubeProxy:
          enabled: false
        kubeScheduler:
          enabled: false
        kubeControllerManager:
          enabled: false

What does this code do?

This declares an ArgoCD Application that pulls version 88.1.3 of the kube-prometheus-stack chart from the official Helm repository.

The four enabled: false lines disable Kubernetes components whose metrics k3d does not expose (etcd, kube-proxy, kube-scheduler, kube-controller-manager). Without this, Prometheus would show permanent target-down alerts for endpoints that do not exist in your local cluster.

• Directly below the disabled components block (still indented under valuesObject), paste the Prometheus, Grafana, destination, and sync configuration:

        prometheus:
          prometheusSpec:
            serviceMonitorSelectorNilUsesHelmValues: false
            podMonitorSelectorNilUsesHelmValues: false
            ruleSelectorNilUsesHelmValues: false
        grafana:
          sidecar:
            dashboards:
              enabled: true
              searchNamespace: ALL
  destination:
    server: https://kubernetes.default.svc
    namespace: monitoring
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true

What does this section do?

The serviceMonitorSelectorNilUsesHelmValues: false line is critical. It tells the Prometheus operator to discover ServiceMonitors from all namespaces regardless of labels. Without it, Prometheus only picks up ServiceMonitors with a specific release label, silently ignoring yours.

The Grafana sidecar config with searchNamespace: ALL lets Grafana find dashboard ConfigMaps in any namespace. The sync policy creates the monitoring namespace automatically and enables self-healing.

• Save argocd/kube-prometheus-stack.yaml.

YAML indentation looks wrong?

The second block must continue at the same indentation level as the disabled components above it (8 spaces for the prometheus: key). If your editor auto-indents, double check that prometheus: lines up with kubeEtcd: above it.

help me fix my YAML indentation in kube-prometheus-stack.yaml

✔️ Awesome, I've got everything!

Great. Double check you have saved your file before committing.

ⓧ I'd like to double check the full code

Scroll up and confirm both code blocks are present in your file, in order:

• Block 1: apiVersion through the four disabled components (kubeControllerManager).
• Block 2: prometheus: spec, grafana: sidecar, destination, and syncPolicy.

The combined file should be a single valid YAML document with 38 lines total.

Commit and let ArgoCD sync the monitoring stack

ArgoCD watches your Git repo for changes. Committing the new Application manifest triggers a sync that deploys the entire monitoring stack.

• Commit and push the new file by running these commands:

git add argocd/kube-prometheus-stack.yaml
git commit -m "Add kube-prometheus-stack ArgoCD Application"
git push

• Wait for ArgoCD to detect the new manifest and begin syncing. This typically takes 30-60 seconds.
• Watch the pods come up in the monitoring namespace by running:

kubectl get pods -n monitoring -w

Images pull from the internet, so expect 2-3 minutes before all pods reach Running status. Press Ctrl+C to stop watching once you see all pods running.

What should I see?

You should see pods for prometheus, grafana, alertmanager, node-exporter, kube-state-metrics, and the prometheus-operator. All should show Running with all containers ready (e.g. 1/1 or 2/2).

Pods stuck in Pending or ImagePullBackOff?

If pods stay in Pending, your k3d cluster may not have enough memory. Check with kubectl describe pod <pod-name> -n monitoring and look for resource constraint messages.

If you see ImagePullBackOff, confirm your internet connection is active. The chart pulls images from public registries (quay.io, docker.io).

help me debug pods not starting in the monitoring namespace

Verify Prometheus targets and confirm the designed gap

Prometheus is running, but does it know about your demo app? Before you check, think about what you expect to see.

• Port-forward the Prometheus UI by running:

kubectl port-forward svc/kube-prometheus-stack-prometheus 9090:9090 -n monitoring

• Open http://localhost:9090 in your browser.
• Navigate to Status in the top menu, then click Targets.

Before you scroll through the targets list, predict: will your demo app appear here?

You should see built-in targets (kubelet, node-exporter, kube-state-metrics) all showing UP. Your demo app is NOT listed. This is expected. Prometheus does not know about your app yet because there is no ServiceMonitor telling it where to scrape.

• Open a new terminal tab (leave the port-forward running) and port-forward Grafana:

kubectl port-forward svc/kube-prometheus-stack-grafana 3000:80 -n monitoring

• Open http://localhost:3000 in your browser.
• Log in with username admin and password prom-operator.

You should see the Grafana home page with pre-built dashboards already available. This confirms the full monitoring stack is deployed and healthy.

Why is the demo app missing from targets?

The kube-prometheus-stack uses a custom resource called a ServiceMonitor to discover scrape targets. Without a ServiceMonitor selecting your app's Service, Prometheus has no idea /metrics exists on your pods. You will fix this gap in the next step.

Cannot reach localhost:9090 or localhost:3000?

Make sure the port-forward command is still running in your terminal. If it exited, re-run it.

If you see error: unable to forward port, the pod may not be ready yet. Run kubectl get pods -n monitoring and confirm all pods are Running before retrying.

help me troubleshoot port-forwarding to Prometheus or Grafana in my k3d cluster

Your monitoring stack is live and collecting cluster-level metrics. Next up, you will wire your demo app to Prometheus by creating a ServiceMonitor so those /metrics you built in Step 1 finally get scraped.

Wire ServiceMonitor and Verify Scraping

Your kube-prometheus-stack is fully deployed and built-in exporters are reporting healthy. But your demo app is conspicuously absent from the Prometheus targets list.

Prometheus uses the ServiceMonitor CRD to discover what to scrape. Without a ServiceMonitor that selects your app's Kubernetes Service, Prometheus will never find your /metrics endpoint. In this step, you'll create both resources and verify live scraping.

In this step, get ready to:

• Create a Kubernetes Service exposing the demo app on a named port.
• Create a ServiceMonitor that tells Prometheus where to scrape.
• Generate traffic and verify metrics flow with PromQL queries.

Create the Service manifest

Prometheus scrapes pods through a Kubernetes Service. Your demo app pods are running, but no Service resource exposes them yet. The Service gives Prometheus a stable target to point at.

• Create a file called manifests/service.yaml in your GitOps repo by pasting this content:

apiVersion: v1
kind: Service
metadata:
  name: demo-app
  labels:
    app: demo-app
spec:
  selector:
    app: demo-app
  ports:
    - name: http
      port: 8080
      targetPort: 8080

What does this manifest do?

• selector: app: demo-app routes traffic to any pod carrying that label.
• The port is named http. This named port is what the ServiceMonitor references to find where /metrics lives.
• Port 8080 matches the container port your demo app listens on.

• Save manifests/service.yaml.
• Confirm the file exists by running:

ls manifests/service.yaml

You should see manifests/service.yaml printed in the output.

File not found?

• Make sure you created the manifests/ directory first. Run mkdir -p manifests if it does not exist.
• Check your terminal is in the root of your GitOps repo folder.

Help me debug why my manifests directory does not exist.

✔️ Awesome, I've got everything!

Great. Make sure the file is saved before moving on.

ⓧ I'd like to double check the full code

apiVersion: v1
kind: Service
metadata:
  name: demo-app
  labels:
    app: demo-app
spec:
  selector:
    app: demo-app
  ports:
    - name: http
      port: 8080
      targetPort: 8080

Create the ServiceMonitor manifest

The ServiceMonitor is the glue between your Service and Prometheus. It tells the Prometheus Operator which Service to scrape, which port to use, and how often.

• Create a file called manifests/servicemonitor.yaml in the same manifests/ directory by pasting this content:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: demo-app
  labels:
    app: demo-app
spec:
  selector:
    matchLabels:
      app: demo-app
  endpoints:
    - port: http
      path: /metrics
      interval: 15s

What does this manifest do?

• selector.matchLabels: app: demo-app finds any Service with that label. This matches the Service you just created.
• port: http tells Prometheus to scrape the named port called http on that Service.
• interval: 15s means Prometheus hits /metrics every 15 seconds. Fast enough for real-time dashboards without overloading a small cluster.

• Save manifests/servicemonitor.yaml.
• Confirm both manifests exist by running:

ls manifests/

You should see both service.yaml and servicemonitor.yaml listed.

Only see one file?

• Double check that you saved the ServiceMonitor file with the exact filename servicemonitor.yaml (all lowercase, no spaces).
• Confirm you are in the root of your GitOps repo. Run pwd to check.

Help me find my missing manifest file.

✔️ Awesome, I've got everything!

Great. Make sure both files are saved before moving on.

ⓧ I'd like to double check the full code

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: demo-app
  labels:
    app: demo-app
spec:
  selector:
    matchLabels:
      app: demo-app
  endpoints:
    - port: http
      path: /metrics
      interval: 15s

Understand how Prometheus discovers ServiceMonitors

Before you commit, it is worth understanding the mechanism that connects a ServiceMonitor to Prometheus. This knowledge saves hours of silent debugging on production clusters.

Why do ServiceMonitors silently disappear?

The Prometheus custom resource has a field called spec.serviceMonitorSelector. This field controls which ServiceMonitors Prometheus picks up.

In a default kube-prometheus-stack installation, this selector requires ServiceMonitors to carry the label release: <helm-release-name>. If your ServiceMonitor lacks that label, Prometheus silently ignores it. There is no error. No warning. No log line. The target simply never appears.

In Step 2, you set serviceMonitorSelectorNilUsesHelmValues: false in the Helm values. This makes the selector empty (matches everything). That is why your ServiceMonitor does not need a release label.

• Verify the empty selector is in effect by running:

kubectl get prometheus -n monitoring -o jsonpath='{.items[0].spec.serviceMonitorSelector}'

You should see {} printed. This empty object means Prometheus accepts ServiceMonitors from all namespaces with any labels.

Diagnosing on other clusters

If you ever work on a cluster without this override, diagnose selector mismatches with two commands. Run kubectl get servicemonitor --show-labels to see what labels your ServiceMonitor carries. Then compare against the Prometheus CR's selector.

Not seeing an empty object?

• Confirm ArgoCD has synced the kube-prometheus-stack Application. Run kubectl get application kube-prometheus-stack -n argocd and check that the status shows Synced and Healthy.
• If the selector shows specific labels instead of {}, revisit your argocd/kube-prometheus-stack.yaml and ensure serviceMonitorSelectorNilUsesHelmValues: false is present under prometheusSpec.

Help me figure out why my serviceMonitorSelector is not empty.

Commit, sync, and verify scraping

Time to push both manifests and let ArgoCD deploy them. Once synced, Prometheus will detect the new ServiceMonitor and start scraping your app within 30 seconds.

• Commit and push the new manifests by running:

git add manifests/
git commit -m "Add Service and ServiceMonitor for demo app"
git push

• Wait about 30 seconds for ArgoCD to detect the change and sync.
• Confirm ArgoCD applied both resources by running:

kubectl get service demo-app
kubectl get servicemonitor demo-app

Both commands should return a resource. The Service shows port 8080. The ServiceMonitor shows your scrape configuration.

Resource not found?

• Run kubectl get application -n argocd to check if the ArgoCD Application that manages your manifests folder has synced.
• If you are using a separate ArgoCD Application for manifests, confirm its source path points to the manifests/ directory.
• Make sure your git push succeeded. Run git log --oneline -1 and confirm the commit message matches.

Help me figure out why ArgoCD is not syncing my manifests.

Now for the moment of truth. Before you check the Prometheus targets page, predict: do you think your demo app will appear as a target now? Why or why not?

• Switch to the Prometheus UI at localhost:9090 (your port-forward from earlier).
• Navigate to Status > Targets.

Your demo app now appears as a target with State: UP. The gap from Step 2 is closed.

What changed?

Prometheus evaluates all ServiceMonitors that match its serviceMonitorSelector (which is empty, so it matches yours). It then resolves the selector inside your ServiceMonitor to find the Service, and scrapes every pod endpoint behind it on the named port http at /metrics.

• Set up a port-forward to the demo app Service by running this in a new terminal:

kubectl port-forward svc/demo-app 8080:8080

• Generate traffic across all endpoints by running this in another terminal:

for i in $(seq 1 20); do curl -s localhost:8080/; curl -s localhost:8080/error; curl -s localhost:8080/slow; done

What does this traffic do?

• curl -s localhost:8080/ sends 20 successful requests, incrementing http_requests_total with status 200.
• curl -s localhost:8080/error sends 20 requests that return 500, burning error budget.
• curl -s localhost:8080/slow adds 1-3 seconds of latency per request, populating the histogram buckets.

• Return to the Prometheus UI query page.
• Enter this PromQL query and click Execute:

http_requests_total

You should see multiple time series returned, broken down by method, endpoint, and status labels.

• Now query the request rate:

rate(http_requests_total[1m])

• Finally, query the p95 latency:

histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[1m]))

Both queries should return non-zero values. The rate query shows requests per second. The histogram quantile shows your 95th percentile latency in seconds (likely between 1 and 3 seconds from the /slow endpoint).

Queries returning empty or no data?

• If the target shows UP but queries return nothing, wait 30-60 seconds. Prometheus needs at least two scrapes before rate() can compute a value.
• If the target is not listed at all, check that kubectl get service demo-app shows port 8080 with the name http. A missing port name causes the ServiceMonitor to fail silently.
• Make sure your traffic generation loop completed. Look for JSON response output in the terminal where you ran the for loop.

Help me figure out why my Prometheus queries are returning no data.

Your app is now fully wired into Prometheus. Every request to any endpoint gets counted, and latency is recorded in histogram buckets. Next up, you'll define what "good" looks like with an explicit SLO and build alerts that fire only when error budget consumption is unsustainable.

Define the SLO and Build Burn-Rate Alerts

Your app is emitting metrics and Prometheus is scraping them every 15 seconds. But metrics alone do not tell you whether your service is healthy or degraded.

Without an explicit definition of "good," you cannot calculate how much failure is acceptable or know when degradation crosses the line. In this step, you will define an SLI (what to measure), an SLO (the target), and an error budget (how much failure is tolerable). Then you will derive multi-window burn-rate alert thresholds from first principles and deploy everything as a PrometheusRule custom resource via ArgoCD.

In this step, get ready to:

• Define the SLI (error ratio) and SLO (99.5% availability over 30 rolling days).
• Derive multi-window multi-burn-rate alert thresholds from first principles.
• Deploy recording rules and alerts via ArgoCD and verify them in Prometheus.

Define the SLI, SLO, and recording rules

Before you can alert on anything, you need to formalize exactly what "good" means for your service. This is the three-layer model.

What are SLI, SLO, and error budget?

An SLI (Service Level Indicator) is the metric you measure. For this project, it is the proportion of HTTP requests that do NOT return a 5xx status code.

An SLO (Service Level Objective) is the target for that metric. You will target 99.5% availability over a rolling 30-day window.

The error budget is what remains: 0.5% of requests can fail before you breach the SLO. That translates to 216 minutes of total downtime per 30 days, or 5,000 errors per million requests.

Recording rules pre-compute expensive PromQL expressions on a schedule so that alert evaluations only compare two floats instead of re-running the full rate() computation every 15 seconds.

• Create the file manifests/prometheusrule.yaml and add the recording rules group by pasting the following code:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: demo-app-slo
  labels:
    app: demo-app
spec:
  groups:
    - name: slo.demo-app.recording-rules
      rules:
        - record: slo:error_ratio:rate5m
          expr: |
            sum(rate(http_requests_total{status=~"5.."}[5m]))
              / sum(rate(http_requests_total[5m]))

        - record: slo:error_ratio:rate30m
          expr: |
            sum(rate(http_requests_total{status=~"5.."}[30m]))
              / sum(rate(http_requests_total[30m]))

        - record: slo:error_ratio:rate1h
          expr: |
            sum(rate(http_requests_total{status=~"5.."}[1h]))
              / sum(rate(http_requests_total[1h]))

        - record: slo:error_ratio:rate6h
          expr: |
            sum(rate(http_requests_total{status=~"5.."}[6h]))
              / sum(rate(http_requests_total[6h]))

What do these recording rules do?

• Each rule computes the ratio of 5xx requests to total requests over a specific time window (5m, 30m, 1h, 6h).
• The status=~"5.." regex matches any 5xx status code (500, 502, 503, etc.).
• Pre-computing these four windows means the alert rules only need to compare the pre-computed values against thresholds, keeping alert evaluation cheap.

• Save manifests/prometheusrule.yaml.
• Validate the YAML syntax by running this command:

kubectl apply --dry-run=client -f manifests/prometheusrule.yaml

You should see a line containing prometheusrule.monitoring.coreos.com/demo-app-slo configured (dry run). This confirms the CRD schema is valid without actually creating the resource yet.

Seeing a validation error?

• Check your YAML indentation. Recording rules must be nested under spec.groups[0].rules. YAML is whitespace-sensitive and a single misaligned line will fail validation.
• Confirm the PrometheusRule CRD exists on your cluster by running kubectl get crd prometheusrules.monitoring.coreos.com. If it is missing, the kube-prometheus-stack may not have synced fully yet.

Help me debug a kubectl dry-run validation error for my PrometheusRule YAML.

Derive burn-rate multipliers and write the alert rules

With recording rules defined, you now need to decide when to alert. A naive threshold ("error rate > 0.5%") would fire the instant a single request fails. You need alerts that fire only when budget consumption is unsustainably fast.

What is a burn rate?

A burn rate describes how quickly you are consuming your error budget relative to its natural depletion. Burn rate 1 means you will exhaust your 30-day budget in exactly 30 days. Burn rate 14.4 means you will exhaust it in roughly 2 days.

The formula is: burn_rate = budget_fraction / window_fraction. For the fast-burn alert, you want to detect 2% of the budget consumed in 1 hour. That gives 0.02 / (1h / 720h) = 14.4. The alert threshold becomes 14.4 x 0.005 = 0.072 (a 7.2% error rate).

For the slow-burn alert, you want to detect 5% consumed in 6 hours. That gives 0.05 / (6h / 720h) = 6. The threshold becomes 6 x 0.005 = 0.03 (a 3% error rate).

Why two windows per alert?

If you use only the 1h window, a 10-minute outage pushes the hourly average above threshold. After the outage ends, the 1h average decays slowly and the alert keeps firing for nearly 50 more minutes while users are fine. The short window (5m) drops below threshold within minutes of recovery.

The AND condition means BOTH windows must exceed the threshold simultaneously. The long window is for detection. The short window is for resolution. This is the core insight from the Google SRE Workbook.

• In manifests/prometheusrule.yaml, add the alerts group below the last recording rule. Paste the fast-burn alert:

    - name: slo.demo-app.alerts
      rules:
        - alert: SLOErrorBudgetFastBurn
          expr: |
            slo:error_ratio:rate1h > 0.072
            and
            slo:error_ratio:rate5m > 0.072
          for: 2m
          labels:
            severity: page
            slo: availability
            service: demo-app
          annotations:
            summary: >-
              Fast error budget burn on demo-app (14.4x).
              At this rate, 2% of the 30-day budget is consumed per hour.
            description: >-
              The error ratio over 1h AND 5m both exceed 7.2%.
              This corresponds to burn rate 14.4x, exhausting the full
              30-day error budget in approximately 2 days.

What does this alert do?

• The expr requires both the 1h AND 5m error ratios to exceed 0.072. Both must be true for the alert to fire.
• The for: 2m clause means the condition must hold for 2 consecutive minutes before the alert transitions from pending to firing. This filters out single-scrape spikes.
• The severity: page label tells Alertmanager to route this to a paging channel (someone needs to wake up).

• Directly below the fast-burn alert (still inside the same rules list), add the slow-burn alert:

        - alert: SLOErrorBudgetSlowBurn
          expr: |
            slo:error_ratio:rate6h > 0.03
            and
            slo:error_ratio:rate30m > 0.03
          for: 15m
          labels:
            severity: ticket
            slo: availability
            service: demo-app
          annotations:
            summary: >-
              Slow error budget burn on demo-app (6x).
              At this rate, 5% of the 30-day budget is consumed per 6 hours.
            description: >-
              The error ratio over 6h AND 30m both exceed 3%.
              This corresponds to burn rate 6x, exhausting the full
              30-day error budget in approximately 5 days.

How does the slow-burn alert differ?

• The thresholds are lower (0.03 vs 0.072) because it catches a gentler but sustained degradation.
• The for: 15m clause gives a longer grace period. A brief error burst will not trigger a ticket.
• The severity: ticket label routes to a ticket queue instead of a page. This is a next-business-day problem, not a wake-up-at-3am problem.

• Save manifests/prometheusrule.yaml.

Deploy and verify the PrometheusRule

The complete PrometheusRule is ready. Commit it to your GitOps repo so ArgoCD picks it up and deploys it into the cluster.

• Stage, commit, and push the new file by running these commands:

git add manifests/prometheusrule.yaml
git commit -m "Add SLO recording rules and burn-rate alerts"
git push

• Wait approximately 30 seconds for ArgoCD to sync the new manifest.
• Confirm the PrometheusRule resource exists by running:

kubectl get prometheusrule demo-app-slo

You should see the resource listed with its name and age.

• Switch back to the Prometheus UI at localhost:9090 (port-forward from earlier).
• Navigate to Status then Rules.

You should see two rule groups: slo.demo-app.recording-rules with four recording rules, and slo.demo-app.alerts with two alert rules.

• Navigate to the Alerts page in Prometheus.

You should see SLOErrorBudgetFastBurn and SLOErrorBudgetSlowBurn both listed in an inactive (green) state.

Before you run the final query, predict: will slo:error_ratio:rate5m return a number, NaN, or no data?

• Navigate to the Graph tab and run this PromQL query:

slo:error_ratio:rate5m

You should see a value returned. If you have not sent any errors recently, it will be 0 or NaN (NaN appears when there is zero traffic in the window, making the division 0/0). Both confirm the recording rule is evaluating correctly.

Rules not appearing in Prometheus?

• Confirm ArgoCD synced successfully. Run kubectl get application -n argocd and check the sync status is Synced.
• The kube-prometheus-stack was configured with ruleSelectorNilUsesHelmValues: false in Step 2, which makes the Prometheus CR's ruleSelector match all PrometheusRules. Verify this is still in effect: kubectl get prometheus -n monitoring -o jsonpath='{.items[0].spec.ruleSelector}' should return {}.
• Check the PrometheusRule resource is in the default namespace (not monitoring). Run kubectl get prometheusrule -A to see all PrometheusRules across namespaces.

Help me debug why my PrometheusRule recording rules are not showing up in Prometheus.

✔️ Awesome, I've got everything!

Great. Double check you have saved manifests/prometheusrule.yaml and that both alerts are visible in the Prometheus Alerts page.

ⓧ I'd like to double check the full code

Your manifests/prometheusrule.yaml should contain the following sections in order. Scroll up and confirm each chunk is present:

• The file header (apiVersion, kind, metadata with name: demo-app-slo and labels.app: demo-app).
• The slo.demo-app.recording-rules group with four recording rules: slo:error_ratio:rate5m, slo:error_ratio:rate30m, slo:error_ratio:rate1h, slo:error_ratio:rate6h.
• The slo.demo-app.alerts group with SLOErrorBudgetFastBurn (threshold 0.072, for: 2m, severity: page) and SLOErrorBudgetSlowBurn (threshold 0.03, for: 15m, severity: ticket).

Your SLO is formally defined and the alerts are armed. Next up, you will build a Grafana dashboard that visualizes the error budget draining in real time.

Build the Grafana Dashboard as Code

You have metrics flowing into Prometheus, recording rules pre-computing error ratios, and burn-rate alerts ready to fire. But there is no single-pane view where you can see all of this at a glance.

A Grafana dashboard provisioned via a ConfigMap is version-controlled and survives pod restarts. It is part of your platform, not a manual creation that gets lost when someone deletes the Grafana pod.

In this step, get ready to:

• Create a Grafana dashboard ConfigMap with four monitoring panels.
• Deploy the dashboard via ArgoCD and verify it appears in Grafana automatically.
• Generate sustained error traffic and watch the error budget drain in real time.

Create the dashboard ConfigMap

The kube-prometheus-stack deploys Grafana with a sidecar container that watches for ConfigMaps carrying the label grafana_dashboard: "1". When it finds one, it loads the JSON content as a dashboard file automatically.

• Create a new file at manifests/grafana-dashboard-configmap.yaml and add the ConfigMap header plus the first panel by pasting this code:

apiVersion: v1
kind: ConfigMap
metadata:
  name: demo-app-slo-dashboard
  namespace: monitoring
  labels:
    grafana_dashboard: "1"
data:
  demo-app-slo.json: |
    {
      "annotations": {"list": []},
      "editable": false,
      "fiscalYearStartMonth": 0,
      "graphTooltip": 1,
      "links": [],
      "panels": [
        {
          "title": "Request Rate",
          "type": "timeseries",
          "gridPos": {"h": 8, "w": 12, "x": 0, "y": 0},
          "targets": [
            {
              "expr": "sum(rate(http_requests_total[5m]))",
              "legendFormat": "total req/s",
              "refId": "A"
            }
          ],
          "fieldConfig": {
            "defaults": {"unit": "reqps"}
          }
        },

What does this code do?

The metadata.labels.grafana_dashboard: "1" label is what triggers the Grafana sidecar to pick up this ConfigMap. Without it, Grafana ignores the file entirely.

The first panel queries sum(rate(http_requests_total[5m])) to show total requests per second across all endpoints and status codes. This is the throughput signal.

• Directly below the Request Rate panel (after the trailing comma), add the Error Rate panel:

        {
          "title": "Error Rate (SLI)",
          "type": "timeseries",
          "gridPos": {"h": 8, "w": 12, "x": 12, "y": 0},
          "targets": [
            {
              "expr": "slo:error_ratio:rate5m",
              "legendFormat": "5m error ratio",
              "refId": "A"
            },
            {
              "expr": "0.005",
              "legendFormat": "SLO budget (0.5%)",
              "refId": "B"
            },
            {
              "expr": "0.072",
              "legendFormat": "fast-burn threshold (7.2%)",
              "refId": "C"
            }
          ],
          "fieldConfig": {
            "defaults": {"unit": "percentunit", "min": 0}
          }
        },

What do the reference lines do?

This panel plots three series on the same graph. The slo:error_ratio:rate5m recording rule shows the live error ratio. The constant 0.005 draws a horizontal line at your SLO budget (0.5%). The constant 0.072 draws the fast-burn threshold (7.2%).

When the live line crosses the 0.072 threshold, you know the fast-burn alert condition is met for the 5m window. Both reference lines give you instant visual context without mental arithmetic.

• Below the Error Rate panel, add the P95 Latency panel:

        {
          "title": "P95 Latency",
          "type": "timeseries",
          "gridPos": {"h": 8, "w": 12, "x": 0, "y": 8},
          "targets": [
            {
              "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))",
              "legendFormat": "p95",
              "refId": "A"
            }
          ],
          "fieldConfig": {
            "defaults": {"unit": "s"}
          }
        },

What is histogram_quantile?

The histogram_quantile function estimates the 95th percentile from your histogram buckets. This means 95% of requests completed faster than the value shown. It uses the bucket boundaries you defined in app.py to interpolate the result.

• Below the P95 Latency panel, add the first half of the Error Budget Remaining stat panel:

        {
          "title": "Error Budget Remaining",
          "type": "stat",
          "gridPos": {"h": 8, "w": 12, "x": 12, "y": 8},
          "targets": [
            {
              "expr": "clamp_min(1 - (slo:error_ratio:rate1h / 0.005), 0)",
              "legendFormat": "budget remaining",
              "refId": "A"
            }
          ],
          "fieldConfig": {
            "defaults": {
              "unit": "percentunit",
              "min": 0,
              "max": 1,
              "thresholds": {
                "mode": "absolute",
                "steps": [
                  {"color": "red", "value": null},
                  {"color": "red", "value": 0},
                  {"color": "yellow", "value": 0.1},
                  {"color": "green", "value": 0.5}
                ]
              }
            }
          },

• Immediately below the fieldConfig closing brace, add the display options and close the file:

          "options": {
            "reduceOptions": {"calcs": ["lastNotNull"]},
            "textMode": "value",
            "colorMode": "background"
          }
        }
      ],
      "schemaVersion": 39,
      "tags": ["slo", "demo-app"],
      "templating": {"list": []},
      "time": {"from": "now-1h", "to": "now"},
      "title": "Demo App SLO",
      "uid": "demo-app-slo"
    }

What does the Error Budget Remaining panel show?

The expression clamp_min(1 - (slo:error_ratio:rate1h / 0.005), 0) calculates how much of your 0.5% error budget remains. When the 1h error ratio equals your full budget (0.005), the result is 0%. The clamp_min(..., 0) prevents negative values.

The thresholds colour the stat panel red below 10%, yellow below 50%, and green above 50%. This gives you an instant visual signal of budget health.

• Save manifests/grafana-dashboard-configmap.yaml.

✔️ Awesome, I've got everything!

Great. Double check you have saved the file before moving on.

ⓧ I'd like to double check the full code

Scroll up and confirm your file contains these sections in order:

• ConfigMap metadata with name: demo-app-slo-dashboard, namespace: monitoring, and grafana_dashboard: "1" label.
• Dashboard JSON opening with "panels": [ array.
• Panel 1: Request Rate querying sum(rate(http_requests_total[5m])).
• Panel 2: Error Rate (SLI) with three targets (live ratio, SLO budget line, fast-burn threshold line).
• Panel 3: P95 Latency querying histogram_quantile.
• Panel 4: Error Budget Remaining stat panel with thresholds and options.
• Dashboard closing with "uid": "demo-app-slo".

YAML indentation errors?

The JSON content inside the ConfigMap must be indented consistently under the data: key. Every line of JSON should be indented 4 spaces further than demo-app-slo.json: |. A single misaligned line will cause a YAML parse error when you apply it.

help me debug a YAML indentation error in my Grafana dashboard ConfigMap

Deploy via GitOps and verify in Grafana

The dashboard file exists locally. Now commit it so ArgoCD syncs the ConfigMap into the cluster and the Grafana sidecar picks it up.

• Add and commit the new file by running these commands:

git add manifests/grafana-dashboard-configmap.yaml
git commit -m "Add Grafana dashboard ConfigMap"
git push

• Wait about 30 seconds for ArgoCD to detect the change and sync.
• Confirm the ConfigMap was created by running:

kubectl get configmap demo-app-slo-dashboard -n monitoring

You should see the ConfigMap listed with its name and age.

• If the Grafana port-forward from earlier is no longer running, restart it:

kubectl port-forward svc/kube-prometheus-stack-grafana 3000:80 -n monitoring

• Open localhost:3000 in your browser.
• Log in with username admin and password prom-operator.
• Click the hamburger menu on the left, then click Dashboards.
• Find Demo App SLO in the list and click it.

You should see four panels arranged in a 2x2 grid. All four panels should show live data (even if the values are very small or zero from minimal traffic).

Why did the dashboard appear without manual import?

The Grafana sidecar container watches the Kubernetes API for ConfigMaps matching the grafana_dashboard: "1" label. When one appears, the sidecar mounts the JSON file into Grafana's provisioning directory. Grafana loads it on the next polling cycle.

Because you set searchNamespace: ALL in the kube-prometheus-stack values, the sidecar looks across every namespace in the cluster.

Dashboard not showing up in Grafana?

Check that the ConfigMap has the correct label by running kubectl get configmap demo-app-slo-dashboard -n monitoring --show-labels. You should see grafana_dashboard=1 in the output.

If the label is correct but the dashboard still does not appear, the Grafana pod may need to restart to pick up the new ConfigMap. Run kubectl rollout restart deployment kube-prometheus-stack-grafana -n monitoring and wait for it to come back.

help me troubleshoot why my Grafana sidecar is not picking up my dashboard ConfigMap

Stress-test the dashboard with sustained errors

The dashboard shows live data, but to see the error budget drain, you need sustained error traffic. A brief burst will not move the 1h recording rule enough to matter.

• Open a separate terminal tab for generating traffic.
• Make sure the app port-forward is active. If not, restart it:

kubectl port-forward deploy/demo-app 8080:8080

• In another terminal tab, start a sustained error loop by running:

while true; do curl -s localhost:8080/error > /dev/null; done

Before you look at the dashboard, think about this: do you expect the fast-burn alert to fire within the first minute of errors? Why or why not?

• Switch back to Grafana and watch the Demo App SLO dashboard.
• Keep the error loop running for at least 5 minutes while watching the panels.

Here is what you should observe. The Error Rate panel spikes immediately. The Error Budget Remaining stat panel counts down. But the fast-burn alert does NOT fire instantly.

Why does the alert take several minutes to fire?

The fast-burn alert requires slo:error_ratio:rate1h > 0.072. The 1h window averages the error ratio over the full hour. If you have been running errors for only 4 minutes, the 1h average is roughly 4/60 = 0.067, which is below 0.072.

At 100% error rate, you need approximately 4.5 minutes of data before the 1h average exceeds 0.072 (since 4.5/60 = 0.075 > 0.072). This is the design working correctly. Brief spikes do not page someone at 3am. Only sustained degradation triggers the alert.

• After 5 minutes, check Prometheus to see the recording rule value. In another terminal, run:

kubectl port-forward svc/kube-prometheus-stack-prometheus 9090:9090 -n monitoring

• Open localhost:9090 and query slo:error_ratio:rate1h. You should see a value above 0.072.
• Navigate to the Alerts page in Prometheus. You should see SLOErrorBudgetFastBurn in a firing or pending state.
• Stop the error loop by pressing Ctrl+C in the terminal running curl.

What happens after you stop errors?

This is where the dual-window design proves its value. The 1h window will stay above threshold for nearly 50 minutes after a 5-minute outage (it decays slowly). But the 5m window drops below 0.072 within 5 minutes of recovery.

Because the alert requires BOTH windows to exceed the threshold (the AND condition), it resolves within minutes of recovery. The short window is for resolution, not detection.

Your observability stack is fully operational. You can see request rate, error ratio, latency, and budget health in one place. The dashboard is version-controlled and will reappear automatically if the Grafana pod restarts.

Secret mission

Add Loki for Centralized Logs

Metrics tell you something is wrong. Logs tell you what is wrong. Deploy Loki in monolithic mode via ArgoCD, wire a log collector to ship pod logs, add structured JSON logging to your app, and query filtered logs in Grafana Explore.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely on your local k3d cluster, so there are no ongoing cloud costs.

The k3d cluster consumes approximately 4-6GB of RAM when the full monitoring stack is running.

Resources you used:

• k3d cluster running Kubernetes locally (contains all workloads: demo app, ArgoCD, Prometheus, Grafana, Alertmanager, Loki).
• Local project files (app/, argocd/, manifests/ directories with all application code and Kubernetes manifests).
• GitHub repository containing your GitOps configuration.

Keep everything running

No action needed. Choose this if you plan to continue to Project 3 (chaos engineering), which builds directly on the monitoring stack you just deployed.

• Leave the k3d cluster running. The monitoring stack provides the foundation for deliberately breaking things and watching alerts fire in the next project.
• Your port-forwards will disconnect when you close your terminal. Restart them when you return with kubectl port-forward commands.

Pause - I'll come back to this later

Shut down the cluster to free up RAM while preserving all state. You can resume exactly where you left off.

• Stop the k3d cluster by running:

k3d cluster stop

• This releases all memory immediately. Your cluster state is preserved on disk.
• When you are ready to resume, start the cluster again by running:

k3d cluster start

• All pods (Prometheus, Grafana, Loki, your demo app) will restart automatically. ArgoCD will reconcile any drift.

Delete - I don't want to use this again

Remove all project resources completely. You can recreate everything from your GitOps repo since all configuration is stored in Git.

• Delete the k3d cluster and all Kubernetes resources inside it by running:

k3d cluster delete

• This removes the cluster, all namespaces, all pods, all persistent volumes, and all container images loaded into k3d.
• Delete your local project files by running:

rm -rf app/ argocd/ manifests/

• Your GitHub repository still contains all manifests. You can clone it again and rebuild the entire stack from scratch if needed.

What about the GitHub repository?

The GitHub repo is free to keep and stores all your GitOps configuration. If you want to delete it, go to the repository's Settings page, scroll to the bottom, and select Delete this repository.

Nice Work!

Nice Work!

You just built a production-grade observability stack from raw metrics to burn-rate alerts, all deployed and reconciled through GitOps. Your cluster now watches itself and knows exactly how much error budget remains at any given moment.

You've learned how to:

• Instrument a Python app with prometheus-client Counter and Histogram metrics, deploy the full kube-prometheus-stack via ArgoCD, and wire Prometheus scraping through a ServiceMonitor CRD.
• Derive multi-window multi-burn-rate alert thresholds from first principles, calculating exact values from your SLO definition and deploying them as PrometheusRule custom resources that fire only when error budget consumption is unsustainable.
• Provision a Grafana dashboard as code via a ConfigMap that shows request rate, error rate, p95 latency, and error budget remaining in a single pane.
• Secret Mission: Deploy Loki in monolithic mode via ArgoCD for centralized log collection and verify structured log queries in Grafana.

Ready to quiz yourself?