Broken Release Incident Lab

Diagnose and repair Kubernetes release failures in a local two-service lab.

Introduction

30 Second Summary

When an app fails, the visible error is often only the last link in a longer chain. Guessing at the cause can turn one outage into several.

In this project, you will build a disposable two-service incident lab in a local Kubernetes cluster. You will use layered triage to separate application configuration, readiness, Service selection, and rollout failures using observable evidence.

What You'll Build

When you finish, your terminal shows a healthy frontend-to-API response restored through evidence-led troubleshooting across each broken layer.

By the end of this project, you'll have:

A disposable two-service application that you can query from the terminal to compare direct API health with the frontend response.
A layered diagnostic path that uses Pod state, container logs, readiness, and EndpointSlices to locate the failing boundary.
An evidence-based incident runbook that records each symptom, observation, root cause, fix, and verification.
Secret Mission: Stall a Deployment with an unavailable image before recovering the previous working revision through rollback.

Are there any prerequisites?

This project assumes Linux with Docker Engine running, kubectl installed, and Visual Studio Code ready. Step 1 installs kind v0.33.0 without requiring local Python.

Before We Start

Before the hands-on work begins, commit to what you are building and why observable evidence comes before every fix.

Build the Incident Environment

Production incidents punish guesswork because one bad change can hide the evidence you needed. A dedicated local Kubernetes cluster gives you a safe place to create destructive failures.

This step uses kind to run the cluster as a Docker container node. The v0.33.0 pin keeps every later incident on the same environment.

In this step, get ready to:

Identify your Linux CPU architecture.
Install kind v0.33.0 through a temporary Python container.
Create the incident-lab cluster and verify its Kubernetes context.

Install kind v0.33.0

The kind release binary differs between AMD64 and ARM64 processors. A temporary container supplies Python 3.14.7 for the download without installing Python on your Linux host.

Identify your processor architecture by running this command:

uname -m

What does this command show?

The uname -m command reports your machine architecture. Use x86_64 for the AMD64 download path or aarch64 for the ARM64 download path.

A version check separates the correct release from an older installation or a missing command.

Check your current kind version by running this command:

kind version

What does this check prove?

This command asks the installed kind binary for its release information. The lab needs v0.33.0 so the cluster setup remains consistent.

✔️ I see v0.33.0

Your kind installation already matches the pinned release. That prerequisite is ready for the disposable cluster.

ⓧ I see another version

Replacing the existing binary is controlled here. This only updates the local kind command at /usr/local/bin/kind.

Download the AMD64 binary if your architecture output was x86_64 by running:

docker run --rm --user "$(id -u):$(id -g)" --volume "$PWD:/work" --workdir /work python:3.14.7-slim-trixie python -c "import urllib.request; urllib.request.urlretrieve('https://kind.sigs.k8s.io/dl/v0.33.0/kind-linux-amd64', 'kind')"

How does the download work?

The Python container downloads the AMD64 release into your current folder as kind.
The mounted directory lets the downloaded file remain after the temporary container exits.
The --rm option removes the helper container after the download completes.

Download the ARM64 binary if your architecture output was aarch64 by running:

docker run --rm --user "$(id -u):$(id -g)" --volume "$PWD:/work" --workdir /work python:3.14.7-slim-trixie python -c "import urllib.request; urllib.request.urlretrieve('https://kind.sigs.k8s.io/dl/v0.33.0/kind-linux-arm64', 'kind')"

How does this path differ?

This command follows the same temporary-container workflow. Its download URL selects the ARM64 release for aarch64 processors.

The next command uses elevated permissions to place the binary in the system command path. Linux may ask for your account password before moving the file.

Activate the downloaded binary and verify the pinned release by running these commands:

chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind
kind version

What do these commands change?

The first command makes the downloaded file executable.
The second command places kind at /usr/local/bin/kind so your shell can find it.
The final command confirms the installed release.

You should now see kind report v0.33.0.

Download or install failed?

Confirm Docker Engine is running before retrying the matching download command. Check that you selected the command for the architecture reported by uname -m.

If the binary cannot move into /usr/local/bin, confirm your Linux account can use elevated permissions. Help me diagnose my kind installation.

ⓧ Command not found

Your Linux host does not have kind available yet. The temporary Python container keeps the host installation limited to the single kind binary.

Download the AMD64 binary if your architecture output was x86_64 by running:

docker run --rm --user "$(id -u):$(id -g)" --volume "$PWD:/work" --workdir /work python:3.14.7-slim-trixie python -c "import urllib.request; urllib.request.urlretrieve('https://kind.sigs.k8s.io/dl/v0.33.0/kind-linux-amd64', 'kind')"

How does the download work?

The Python container downloads the AMD64 release into your current folder as kind.
The mounted directory keeps the downloaded file on your Linux host.
The helper container is removed automatically after the download.

Download the ARM64 binary if your architecture output was aarch64 by running:

docker run --rm --user "$(id -u):$(id -g)" --volume "$PWD:/work" --workdir /work python:3.14.7-slim-trixie python -c "import urllib.request; urllib.request.urlretrieve('https://kind.sigs.k8s.io/dl/v0.33.0/kind-linux-arm64', 'kind')"

How does this path differ?

This command downloads the ARM64 release for aarch64 processors. It leaves the binary in your current folder.

The installation uses elevated permissions only when moving the binary into the system command path. Linux may ask for your account password at that point.

Activate the downloaded binary and verify the pinned release by running these commands:

chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind
kind version

What do these commands change?

The executable permission allows Linux to run the binary.
The move places kind at /usr/local/bin/kind.
The version check confirms the installation.

You should now see kind report v0.33.0. The cluster tool is ready without a host Python installation.

Still unable to run kind?

Confirm Docker Engine is running before retrying the matching download command. Check that your architecture matches the binary you downloaded.

If the move fails, confirm your Linux account can use elevated permissions. Help me troubleshoot the missing kind command.

Create the isolated cluster

The incident-lab cluster contains the destructive failures you create later. Its single control-plane node gives you a clean environment without touching a shared cluster.

Why kind for this lab?

Kind runs local Kubernetes nodes as Docker containers. That produces a short build and load workflow for the application image used later.

Other local cluster tools introduce extra driver or runtime choices. Kind keeps this lab focused on workload evidence.

The first control-plane image pull can take several minutes. Allow up to five minutes before treating the wait as a problem.

Create the named cluster and wait for its control plane by running this command:

kind create cluster --name incident-lab --wait 5m

What does this command create?

The command creates a local cluster named incident-lab. The wait setting gives its control plane up to five minutes to become ready.

You should see the cluster creation finish successfully. Your disposable incident boundary now exists.

Cluster creation did not finish?

Confirm Docker Engine is running with enough resources for a container node. Retry the cluster command after Docker becomes responsive.

A slow first image pull can use most of the five-minute wait. Help me diagnose the kind cluster creation failure.

Select and verify the cluster context

A Kubernetes context tells kubectl which cluster to control. Selecting kind-incident-lab prevents later incident commands from reaching another environment.

Select the generated context and inspect the reachable cluster by running these commands:

kubectl config use-context kind-incident-lab
kind get clusters
kubectl cluster-info
kubectl get nodes

What do these checks prove?

The context command directs kubectl to kind-incident-lab.
The cluster list confirms that incident-lab exists.
The cluster information confirms that the Kubernetes control plane is reachable.
The node list exposes the control-plane readiness state.

Before the final check, do you expect both the tool version and the node state to match the pinned lab environment?

Confirm the kind release and node readiness by running the final check:

kind version && kubectl get nodes

What does the final check confirm?

The first command confirms the pinned kind release. The second command proves kubectl can reach the cluster and read its node state.

You should see kind report v0.33.0. You should also see one control-plane node with Ready in its status.

Node not showing Ready?

Confirm the selected context is kind-incident-lab. Check Docker Engine if kubectl cannot reach the control plane.

If the node remains unavailable, inspect the earlier cluster creation output before recreating anything. Help me diagnose the node readiness problem.

Your disposable Kubernetes cluster is live and reachable. Next, you will deploy the two-service release that creates your first evidence boundary.

Deploy the Broken Release

Your isolated kind cluster is ready for a controlled incident. The next challenge is proving where a request succeeds before deciding where the release is broken.

You will deploy a two-service application to Kubernetes using a local Docker image. A direct API request will succeed while the frontend request fails.

In this step, get ready to:

Create the application files inside broken-release-lab.
Build the local image and deploy both services to the cluster.
Probe each request path to reveal the partial failure.

Create and containerize the lab files

The same Python application runs in frontend or API mode based on environment variables. A separate probe reports the response status without requiring Python on your Linux host.

Create the broken-release-lab directory on your Desktop by running these commands:

cd ~/Desktop
mkdir -p broken-release-lab/k8s
cd broken-release-lab
: > app.py
: > probe.py
: > Dockerfile
: > k8s/app.yaml
: > runbook.md

What do these commands do?

The first three commands create the lab directory and make it your current terminal location.
The remaining commands create the five empty project files.
The k8s directory keeps the Kubernetes manifest separate from the application code.

Open the current broken-release-lab directory in Visual Studio Code by running this command:

code .

What does this command do?

The period tells Visual Studio Code to open the current terminal directory. The Explorer sidebar gives you direct access to every lab file.

You should see app.py, probe.py, Dockerfile, runbook.md, and the k8s directory in the Explorer sidebar.

Visual Studio Code did not open?

Confirm your terminal is inside broken-release-lab. You can also open the directory through Visual Studio Code's folder picker.

Ask for help with the editor command.

Select app.py in the Explorer sidebar.
Add the imports and runtime configuration using this code:

import http.server
import json
import os
import urllib.request

MODE = os.getenv("MODE", "frontend")
PORT = int(os.getenv("PORT", "8080"))
BACKEND_URL = os.getenv("BACKEND_URL", "http://incident-api:8080")


class IncidentHandler(http.server.BaseHTTPRequestHandler):
    def write_response(self, status_code, content_type, body):
        body_bytes = body.encode("utf-8")
        self.send_response(status_code)
        self.send_header("Content-Type", f"{content_type}; charset=utf-8")
        self.send_header("Content-Length", str(len(body_bytes)))
        self.end_headers()
        self.wfile.write(body_bytes)

What does this code do?

The environment variables let one image act as either the frontend or the API.
The write_response() helper sends a status code with the response body.
The default backend address targets the internal API Service.

Append the health endpoint and API response logic below write_response() using this code:

    def do_GET(self):
        if self.path == "/healthz":
            body = json.dumps({"status": "ok", "mode": MODE})
            self.write_response(200, "application/json", body)
            return

        if MODE == "api" and self.path == "/api/message":
            body = json.dumps(
                {
                    "service": "incident-api",
                    "message": "Request crossed frontend -> Service -> API",
                }
            )
            self.write_response(200, "application/json", body)
            return

How do these routes help?

The health route gives Kubernetes a small response for readiness checks.
The API route returns the message that proves a request crossed the internal Service boundary.

Append the frontend request handler inside do_GET() using this code:

        if MODE == "frontend" and self.path == "/":
            api_url = f"{BACKEND_URL}/api/message"
            try:
                with urllib.request.urlopen(api_url, timeout=2) as response:
                    payload = json.loads(response.read().decode("utf-8"))
                    message = payload["message"]
            except (OSError, TimeoutError, json.JSONDecodeError, KeyError, TypeError) as exc:
                print(
                    f"backend_request_failed url={api_url} error={exc}",
                    flush=True,
                )
                self.write_response(
                    502,
                    "text/plain",
                    f"Frontend is running, but the API dependency failed: {exc}\n",
                )
                return

            body = f"""<!doctype html>
<html lang="en">
  <head><meta charset="utf-8"><title>Incident Lab</title></head>
  <body>
    <h1>Incident Lab Healthy</h1>
    <p>{message}</p>
  </body>
</html>
"""
            self.write_response(200, "text/html", body)
            return

What happens in the frontend?

The frontend calls the API address stored in BACKEND_URL.
A successful dependency call becomes a small HTML page.
A failed dependency call becomes an HTTP 502 response with a diagnostic log entry.

Finish app.py with the fallback route and server startup code:

        self.write_response(404, "text/plain", "Not found\n")

    def log_message(self, format, *args):
        print(
            f'access client={self.client_address[0]} message="{format % args}"',
            flush=True,
        )


if __name__ == "__main__":
    print(f"starting mode={MODE} port={PORT} backend_url={BACKEND_URL}", flush=True)
    server = http.server.ThreadingHTTPServer(("0.0.0.0", PORT), IncidentHandler)
    server.serve_forever()

How does the server start?

Unknown paths receive an HTTP 404 response.
Request logs expose the client address and HTTP message.
The final lines start the threaded server on every container network interface.

Save app.py.
Confirm the unsaved-change marker beside app.py disappears.

Having trouble assembling app.py?

Check that each new fragment stays inside the indentation shown. Python uses indentation to keep the request branches inside do_GET().

Compare your file with the full reference near the end of this step.

Select probe.py in the Explorer sidebar.
Add the response probe using this code:

import sys
import urllib.error
import urllib.request

url = sys.argv[1]

try:
    with urllib.request.urlopen(url, timeout=3) as response:
        print(f"status={response.status}")
        print(response.read().decode("utf-8"))
except urllib.error.HTTPError as exc:
    print(f"status={exc.code}")
    print(exc.read().decode("utf-8"))
    raise SystemExit(1) from exc
except OSError as exc:
    print(f"request_failed={exc}")
    raise SystemExit(1) from exc

What does the probe report?

The probe accepts a URL as its first command-line argument.
Successful responses print their status code with the response body.
HTTP errors still print their status code before the probe exits unsuccessfully.

Save probe.py.
Confirm the unsaved-change marker beside probe.py disappears.

Select Dockerfile in the Explorer sidebar.
Define the local application image using this code:

FROM python:3.14.7-slim-trixie
WORKDIR /app
COPY app.py probe.py ./
EXPOSE 8080
CMD ["python", "app.py"]

What does the Dockerfile package?

The image uses the Python 3.14.7 slim base.
Both Python files are copied into /app.
Containers start the application server on port 8080 by default.

Save Dockerfile.
Build the local image from the current directory by running this command:

docker build -t incident-app:1.0 .

What does this build create?

Docker packages the server and probe into the local image tagged incident-app:1.0. Both Kubernetes workloads can reuse that one image with different environment values.

The first build can pause while Docker downloads the Python base image. A successful build finishes with the image tagged as incident-app:1.0.

Image build failed?

Confirm Docker Engine is still running.
Confirm the terminal is inside broken-release-lab.
Confirm Dockerfile, app.py, and probe.py are saved.

Get help with the failed build.

Deploy the Kubernetes resources

A Deployment maintains each group of application Pods. A Service provides the stable internal address used to reach those Pods.

Each workload also has a readiness probe that controls whether its Pods receive Service traffic. The manifest deploys two API replicas with one frontend replica.

Expand the k8s directory in the Explorer sidebar.
Select app.yaml.
Add the first half of the API Deployment using this code:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: incident-api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: incident-api
  template:
    metadata:
      labels:
        app: incident-api
    spec:
      containers:
        - name: app
          image: incident-app:1.0
          imagePullPolicy: Never

What does this Deployment control?

The replica count asks Kubernetes to maintain two API Pods.
The selector connects the Deployment to Pods carrying the app: incident-api label.
The pull policy requires the image to exist inside the kind node.

Append the API environment and readiness configuration using this code:

          env:
            - name: MODE
              value: api
            - name: PORT
              value: "8080"
          ports:
            - name: http
              containerPort: 8080
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 1
            periodSeconds: 2
---

How does the API become ready?

The environment configures these containers to run in API mode.
The readiness probe checks /healthz on port 8080.
The document separator prepares the file for the next Kubernetes resource.

Append the API Service using this code:

apiVersion: v1
kind: Service
metadata:
  name: incident-api
spec:
  selector:
    app: incident-api
  ports:
    - name: http
      port: 8080
      targetPort: 8080
---

How does the Service find the API?

The Service selects Pods with the app: incident-api label. It sends traffic from Service port 8080 to the same container port.

Append the first half of the frontend Deployment using this code:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: incident-frontend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: incident-frontend
  template:
    metadata:
      labels:
        app: incident-frontend
    spec:
      containers:
        - name: app
          image: incident-app:1.0
          imagePullPolicy: Never

What does the frontend Deployment control?

This Deployment maintains one Pod with the app: incident-frontend label. It uses the same local image as the API.

Append the frontend environment and readiness configuration using this code:

          env:
            - name: MODE
              value: frontend
            - name: PORT
              value: "8080"
            - name: BACKEND_URL
              value: http://incident-api-wrong:8080
          ports:
            - name: http
              containerPort: 8080
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 1
            periodSeconds: 2
---

How is the frontend configured?

The environment starts this container in frontend mode.
The frontend uses BACKEND_URL whenever it handles a request for its home page.
Its own health route remains independent of the downstream API request.

Finish the manifest with the frontend Service using this code:

apiVersion: v1
kind: Service
metadata:
  name: incident-frontend
spec:
  selector:
    app: incident-frontend
  ports:
    - name: http
      port: 8080
      targetPort: 8080

How is the frontend exposed?

The frontend Service selects the frontend Pod. Port forwarding later gives your Linux host temporary access to this internal Service.

Save k8s/app.yaml.
Load the local image into the named kind cluster by running this command:

kind load docker-image incident-app:1.0 --name incident-lab

Why load the image into kind?

The kind node runs as a Docker container with its own image store. Loading incident-app:1.0 makes the local image available to the Pods.

You should see confirmation that the image was loaded into the incident-lab cluster.

Create the Kubernetes resources and monitor both rollouts by running these commands:

kubectl apply -f k8s/app.yaml
kubectl rollout status deployment/incident-api --timeout=60s
kubectl rollout status deployment/incident-frontend --timeout=60s

What happens during deployment?

The apply command creates both Deployments with both Services.
Kubernetes also manages an EndpointSlice for each selector-backed Service.
The rollout commands wait for two ready API Pods and one ready frontend Pod.

Both rollout checks should complete successfully. This proves the containers satisfy their own readiness checks.

A rollout did not complete?

Confirm the image load command targeted incident-lab.
Confirm every image value in k8s/app.yaml uses incident-app:1.0.
Confirm both readiness paths use /healthz.

Get help with the rollout evidence.

Probe both request paths

Port forwarding lets you test each internal Service from your Linux host. Comparing the two responses separates cluster-wide failure from a broken dependency path.

Start the API port-forward in one terminal by running this command:

kubectl port-forward service/incident-api 8081:8080

What does this port-forward expose?

The command maps local port 8081 to port 8080 on the API Service. Leave this process running while you use another terminal for the probe.

Before you probe the API, do you expect the direct request to reach a ready API Pod?

Run the direct API probe in another terminal using this command:

docker run --rm --network host incident-app:1.0 python probe.py http://127.0.0.1:8081/api/message

What does this probe test?

The probe container shares the Linux host network. It sends a request through the API port-forward without involving the frontend.

You will see status=200 with the message Request crossed frontend -> Service -> API. The cluster can run the API and route direct Service traffic.

Switch back to the terminal running the API port-forward.
Press Ctrl+C to stop it.
Start the frontend port-forward by running this command:

kubectl port-forward service/incident-frontend 8080:8080

What does this port-forward expose?

This command maps local port 8080 to the frontend Service. Leave it running while the next probe tests the complete request path.

Before you probe the frontend, do you think its successful readiness check guarantees that the API dependency also works?

Probe the frontend through its port-forward by running this command:

docker run --rm --network host incident-app:1.0 python probe.py http://127.0.0.1:8080/

What does this probe test?

This request reaches the frontend before the frontend attempts its internal API call. The response reveals whether the entire dependency path works.

You will see status=502 with a message explaining that the API dependency failed. This is the intended broken release.

The failure is your first evidence boundary

The API succeeds directly while the frontend fails through its dependency path. The ready Pods prove that process health alone cannot confirm the complete application path.

Select runbook.md in the Explorer sidebar.
Add the incident evidence table using this content:

# Broken Release Incident Runbook

| Incident | User symptom | Kubernetes evidence | Application evidence | Root cause | Fix | Verification |
| --- | --- | --- | --- | --- | --- | --- |
| Dependency configuration | Frontend request fails while API works directly | Frontend and API Pods are Running and Ready | Frontend logs show `backend_request_failed` for the wrong Service name | Incorrect `BACKEND_URL` | Point the frontend to `http://incident-api:8080` | Frontend probe reports `status=200` |
| Readiness | Frontend cannot reach API while API containers keep running | API Pods are Running but not Ready, and the API EndpointSlice has a blank `ENDPOINTS` column | API access logs show repeated HTTP 404 responses for `/ready` | Readiness probe targets the wrong path | Restore the probe path to `/healthz` | API endpoint addresses return and frontend reports `status=200` |
| Service selector | Frontend cannot reach API while API Pods are Ready | Service selector does not match Pod labels and the EndpointSlice has a blank `ENDPOINTS` column | API process remains healthy | Selector changed to `app: incident-api-wrong` | Restore selector `app: incident-api` | Endpoint addresses return and frontend reports `status=200` |

How will the runbook help?

Each row connects a user symptom to Kubernetes evidence and application evidence. The final columns preserve the root cause with its repair and verification.

Save runbook.md.
Compare every project file with the complete reference below.

✔️ Awesome, I've got everything!

Great work. Save every file before moving on to the diagnosis.

ⓧ I'd like to double check the full code

Dockerfile

FROM python:3.14.7-slim-trixie
WORKDIR /app
COPY app.py probe.py ./
EXPOSE 8080
CMD ["python", "app.py"]

app.py

import http.server
import json
import os
import urllib.request

MODE = os.getenv("MODE", "frontend")
PORT = int(os.getenv("PORT", "8080"))
BACKEND_URL = os.getenv("BACKEND_URL", "http://incident-api:8080")


class IncidentHandler(http.server.BaseHTTPRequestHandler):
    def write_response(self, status_code, content_type, body):
        body_bytes = body.encode("utf-8")
        self.send_response(status_code)
        self.send_header("Content-Type", f"{content_type}; charset=utf-8")
        self.send_header("Content-Length", str(len(body_bytes)))
        self.end_headers()
        self.wfile.write(body_bytes)

    def do_GET(self):
        if self.path == "/healthz":
            body = json.dumps({"status": "ok", "mode": MODE})
            self.write_response(200, "application/json", body)
            return

        if MODE == "api" and self.path == "/api/message":
            body = json.dumps(
                {
                    "service": "incident-api",
                    "message": "Request crossed frontend -> Service -> API",
                }
            )
            self.write_response(200, "application/json", body)
            return

        if MODE == "frontend" and self.path == "/":
            api_url = f"{BACKEND_URL}/api/message"
            try:
                with urllib.request.urlopen(api_url, timeout=2) as response:
                    payload = json.loads(response.read().decode("utf-8"))
                    message = payload["message"]
            except (OSError, TimeoutError, json.JSONDecodeError, KeyError, TypeError) as exc:
                print(
                    f"backend_request_failed url={api_url} error={exc}",
                    flush=True,
                )
                self.write_response(
                    502,
                    "text/plain",
                    f"Frontend is running, but the API dependency failed: {exc}\n",
                )
                return

            body = f"""<!doctype html>
<html lang="en">
  <head><meta charset="utf-8"><title>Incident Lab</title></head>
  <body>
    <h1>Incident Lab Healthy</h1>
    <p>{message}</p>
  </body>
</html>
"""
            self.write_response(200, "text/html", body)
            return

        self.write_response(404, "text/plain", "Not found\n")

    def log_message(self, format, *args):
        print(
            f'access client={self.client_address[0]} message="{format % args}"',
            flush=True,
        )


if __name__ == "__main__":
    print(f"starting mode={MODE} port={PORT} backend_url={BACKEND_URL}", flush=True)
    server = http.server.ThreadingHTTPServer(("0.0.0.0", PORT), IncidentHandler)
    server.serve_forever()

k8s/app.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: incident-api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: incident-api
  template:
    metadata:
      labels:
        app: incident-api
    spec:
      containers:
        - name: app
          image: incident-app:1.0
          imagePullPolicy: Never
          env:
            - name: MODE
              value: api
            - name: PORT
              value: "8080"
          ports:
            - name: http
              containerPort: 8080
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 1
            periodSeconds: 2
---
apiVersion: v1
kind: Service
metadata:
  name: incident-api
spec:
  selector:
    app: incident-api
  ports:
    - name: http
      port: 8080
      targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: incident-frontend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: incident-frontend
  template:
    metadata:
      labels:
        app: incident-frontend
    spec:
      containers:
        - name: app
          image: incident-app:1.0
          imagePullPolicy: Never
          env:
            - name: MODE
              value: frontend
            - name: PORT
              value: "8080"
            - name: BACKEND_URL
              value: http://incident-api-wrong:8080
          ports:
            - name: http
              containerPort: 8080
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 1
            periodSeconds: 2
---
apiVersion: v1
kind: Service
metadata:
  name: incident-frontend
spec:
  selector:
    app: incident-frontend
  ports:
    - name: http
      port: 8080
      targetPort: 8080

probe.py

import sys
import urllib.error
import urllib.request

url = sys.argv[1]

try:
    with urllib.request.urlopen(url, timeout=3) as response:
        print(f"status={response.status}")
        print(response.read().decode("utf-8"))
except urllib.error.HTTPError as exc:
    print(f"status={exc.code}")
    print(exc.read().decode("utf-8"))
    raise SystemExit(1) from exc
except OSError as exc:
    print(f"request_failed={exc}")
    raise SystemExit(1) from exc

runbook.md

# Broken Release Incident Runbook

| Incident | User symptom | Kubernetes evidence | Application evidence | Root cause | Fix | Verification |
| --- | --- | --- | --- | --- | --- | --- |
| Dependency configuration | Frontend request fails while API works directly | Frontend and API Pods are Running and Ready | Frontend logs show `backend_request_failed` for the wrong Service name | Incorrect `BACKEND_URL` | Point the frontend to `http://incident-api:8080` | Frontend probe reports `status=200` |
| Readiness | Frontend cannot reach API while API containers keep running | API Pods are Running but not Ready, and the API EndpointSlice has a blank `ENDPOINTS` column | API access logs show repeated HTTP 404 responses for `/ready` | Readiness probe targets the wrong path | Restore the probe path to `/healthz` | API endpoint addresses return and frontend reports `status=200` |
| Service selector | Frontend cannot reach API while API Pods are Ready | Service selector does not match Pod labels and the EndpointSlice has a blank `ENDPOINTS` column | API process remains healthy | Selector changed to `app: incident-api-wrong` | Restore selector `app: incident-api` | Endpoint addresses return and frontend reports `status=200` |

That split result is the lab's first evidence boundary. Next, you will trace the failed dependency from healthy Pods into the application logs.

Repair the Dependency Configuration

The direct API probe already returned status=200. The frontend probe returned status=502.

Healthy Kubernetes Pods show that the workloads are running inside the cluster. The frontend's container logs can expose why its request still fails.

In this step, get ready to:

Trace the frontend failure through Pod state and application logs.
Roll out the corrected frontend dependency address.
Verify the repaired request path with a frontend probe.

Trace the failure with evidence

Pod status describes whether the containers satisfy Kubernetes health checks. Application logs reveal failures inside a process that still passes those checks.

Before you run these checks, predict whether healthy Pod state rules out a bad dependency address.

Compare the workload state with the frontend logs by running these commands:

kubectl get pods
kubectl describe pods -l app=incident-frontend
kubectl logs deployment/incident-frontend

What does the evidence prove?

The Pod list shows every workload is Running. Each Pod is also Ready.
The frontend description shows no Pod-level failure blocking the container.
The logs contain backend_request_failed for http://incident-api-wrong:8080/api/message.

The evidence places the fault inside the frontend configuration. Its dependency address names a Service that does not exist.

Missing the frontend failure log?

Send another request through the frontend port-forward from the previous step. Rerun the log command after the failed request.

If no Pod is selected, confirm the frontend label remains app=incident-frontend in k8s/app.yaml.

Help me interpret the frontend Pod state and logs.

Correct the dependency URL

A Kubernetes Service gives the API a stable name inside the cluster. Changing the address in the frontend Deployment starts a rollout with the corrected dependency configuration.

In the existing broken-release-lab workspace in Visual Studio Code, select k8s/app.yaml from the Explorer sidebar.
Find BACKEND_URL inside the incident-frontend Deployment.
Replace http://incident-api-wrong:8080 with http://incident-api:8080.
Save k8s/app.yaml.
Compare the corrected environment entry with this reference:

            - name: BACKEND_URL
              value: http://incident-api:8080

What does this configuration do?

The BACKEND_URL value now targets the existing incident-api Service on port 8080.

The setting sits inside the frontend Pod template. Updating that template starts a rolling update.

✔️ Awesome, I've got everything!

Your manifest now points the frontend at the correct API Service.

ⓧ I'd like to double check the full code

apiVersion: apps/v1
kind: Deployment
metadata:
  name: incident-api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: incident-api
  template:
    metadata:
      labels:
        app: incident-api
    spec:
      containers:
        - name: app
          image: incident-app:1.0
          imagePullPolicy: Never
          env:
            - name: MODE
              value: api
            - name: PORT
              value: "8080"
          ports:
            - name: http
              containerPort: 8080
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 1
            periodSeconds: 2
---
apiVersion: v1
kind: Service
metadata:
  name: incident-api
spec:
  selector:
    app: incident-api
  ports:
    - name: http
      port: 8080
      targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: incident-frontend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: incident-frontend
  template:
    metadata:
      labels:
        app: incident-frontend
    spec:
      containers:
        - name: app
          image: incident-app:1.0
          imagePullPolicy: Never
          env:
            - name: MODE
              value: frontend
            - name: PORT
              value: "8080"
            - name: BACKEND_URL
              value: http://incident-api:8080
          ports:
            - name: http
              containerPort: 8080
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 1
            periodSeconds: 2
---
apiVersion: v1
kind: Service
metadata:
  name: incident-frontend
spec:
  selector:
    app: incident-frontend
  ports:
    - name: http
      port: 8080
      targetPort: 8080

How does this reference help?

This is the complete manifest for the current lab state. The frontend dependency now matches the API Service name.

Apply the corrected manifest and monitor the frontend rollout by running:

kubectl apply -f k8s/app.yaml
kubectl rollout status deployment/incident-frontend --timeout=60s

What do these commands do?

The first command updates the resources defined in k8s/app.yaml.
The second command watches the frontend rollout for up to 60s.

The rollout completes after the replacement frontend Pod becomes ready. The corrected configuration is now running inside the cluster.

Frontend rollout not completing?

Check that BACKEND_URL contains http://incident-api:8080 with no extra characters.

Confirm the manifest still names the frontend container app.

Help me diagnose the stalled frontend rollout.

Verify the repaired request path

A port-forward exposes the frontend Service through local port 8080. The probe runs inside a Docker container against that local address.

Return to the terminal used for the frontend port-forward.
Press Ctrl+C if the earlier port-forward is still running.
Restart the frontend port-forward by running this command:

kubectl port-forward service/incident-frontend 8080:8080

What does this command do?

The command maps local port 8080 to port 8080 on the incident-frontend Service. It keeps control of this terminal until you stop it.

Leave the port-forward terminal running.
Switch back to the probe terminal from earlier.

Local port 8080 unavailable?

Another port-forward may still own local port 8080. Stop the older process before starting this one.

Help me restore the frontend port-forward.

Before you run the probe, predict whether correcting one dependency address is enough to restore the full request path.

Probe the repaired frontend through the local port-forward by running:

docker run --rm --network host incident-app:1.0 python probe.py http://127.0.0.1:8080/

What proves the repair worked?

The probe prints status=200 to confirm the HTTP request succeeded.
The response contains Incident Lab Healthy to confirm the frontend rendered its healthy page.
The message Request crossed frontend -> Service -> API confirms the request reached the API dependency.

You traced the failure without guessing. The complete frontend-to-API path now works again.

Probe still returning a failure?

Confirm the frontend port-forward terminal is still running. A stopped tunnel prevents the probe from reaching the Service.

Check the latest frontend logs for another backend_request_failed entry. Its URL identifies the dependency address used by the new Pod.

Help me diagnose the frontend probe after the configuration repair.

Return to runbook.md in Visual Studio Code.
Locate the Dependency configuration row.
Confirm the row records the user symptom from the failed probe.
Confirm the row records the healthy Kubernetes evidence.
Confirm the row records the frontend log evidence.
Confirm the row records the corrected BACKEND_URL value.
Confirm the row records the successful frontend probe.

Your dependency path is healthy again. Next, you'll investigate why a running API process can still disappear from Service traffic.

Diagnose the Readiness Black Hole

The dependency URL is fixed. Your frontend now reaches the API through its Kubernetes Service.

A normal rolling update preserves old ready Pods. Those replicas can hide a broken readiness probe, so this incident removes them before introducing the failure.

In this step, get ready to:

Create an API readiness failure without healthy replicas masking it.
Diagnose the empty EndpointSlice from observable evidence.
Restore healthy API traffic with the evidence recorded in runbook.md.

Create the readiness black hole

A readiness probe decides whether a running Pod is eligible for Service traffic. A failed probe keeps the container alive while removing it from the request path.

Why scale to zero first?

Rolling updates normally keep old ready replicas available while new replicas start. Scaling to zero removes that safety net.

The faulty probe can now expose the readiness boundary without healthy API Pods masking the outage.

Remove the healthy API replicas by running these commands:

kubectl scale deployment/incident-api --replicas=0
kubectl get pods -l app=incident-api

What do these commands prove?

The scale command sets the desired API replica count to zero.
The Pod query shows whether any API replicas still exist.

Repeat the Pod check from the previous command block until it returns no API Pods.

You've cleared the masking replicas. The next API Pods now have to prove readiness on their own.

API Pods still listed?

Wait for Kubernetes to finish terminating the replicas. Repeat the Pod check after a few seconds.

If the replicas return immediately, confirm the Deployment remains scaled to zero. Help me understand why my API Pods keep returning.

Switch back to k8s/app.yaml in Visual Studio Code.
Find the API Deployment readiness path set to /healthz.
Replace /healthz with /ready.
Save k8s/app.yaml.

Before you apply this manifest, do you think a running API container will also become ready?

Create the faulty rollout by running these commands:

kubectl apply -f k8s/app.yaml
kubectl get pods -l app=incident-api

What does this check expose?

Applying the manifest restores the desired API replica count to two. The Pod query separates process state from readiness state.

Repeat the Pod check from the previous command block until both API containers show a stable state.

You'll see both containers in Running state while each Pod remains 0/1 Ready. The processes are alive, but Kubernetes withholds them from Service traffic.

Pods not staying at 0/1 Ready?

Confirm you changed the readiness path inside the incident-api Deployment. The frontend Deployment has its own probe with the same healthy path.

Confirm you saved k8s/app.yaml before applying it. Help me diagnose why my API Pods are not showing 0/1 Ready.

Trace the missing Service endpoints

Readiness affects the request path before the application process stops. Pod details, application logs, and the EndpointSlice reveal that separation from three angles.

Before you gather the evidence, which layer do you expect to explain why running API containers receive no Service traffic?

Gather evidence from the API Pods and their EndpointSlice by running these commands:

kubectl describe pods -l app=incident-api
kubectl logs deployment/incident-api
kubectl get endpointslices -l kubernetes.io/service-name=incident-api

How does the evidence fit together?

The Pod descriptions show running containers that fail readiness checks for /ready.
The API logs show repeated HTTP 404 responses for the missing readiness path.
The API EndpointSlice shows a blank ENDPOINTS column because no API Pod is ready for Service traffic.

Endpoint addresses still present?

Confirm every API Pod shows 0/1 Ready. A ready API Pod can keep an endpoint address populated.

Confirm the EndpointSlice query targets the incident-api Service. Help me trace why my API EndpointSlice still has addresses.

Restore readiness and record the incident

The evidence points to one boundary. The API process works, but the incorrect probe path prevents Kubernetes from admitting its Pods into Service traffic.

Return to k8s/app.yaml in Visual Studio Code.
Find the API readiness path currently set to /ready.
Replace /ready with /healthz.
Save k8s/app.yaml.

Before you apply the correction, do you expect the existing API processes to become eligible for traffic once the healthy path returns?

Apply the corrected probe and monitor the API rollout by running these commands:

kubectl apply -f k8s/app.yaml
kubectl rollout status deployment/incident-api --timeout=60s

What does the rollout confirm?

The manifest restores the probe path implemented by the API. Rollout completion confirms the replacement API Pods reached their required ready state.

Before you inspect the EndpointSlice, do you expect its endpoint addresses to return with API readiness?

Check the API EndpointSlice by running this command:

kubectl get endpointslices -l kubernetes.io/service-name=incident-api

What should you see?

The ENDPOINTS column now contains API endpoint addresses. Ready Pods are eligible for Service traffic again.

Return to the terminal that held the frontend port-forward from the previous step.
Restart that same frontend port-forward if it is no longer running.

Before you rerun the frontend probe, do you expect the complete frontend-to-API request path to succeed again?

Rerun the frontend probe from the previous step.

You'll see status=200 with the frontend-to-API success message. Strong recovery. The API is ready, its EndpointSlice is populated, and the frontend request path works again.

Frontend still failing?

Confirm the API rollout completed before retrying the probe. Confirm the frontend port-forward is still attached to the incident-frontend Service.

If endpoint addresses returned but the request still fails, inspect the current frontend logs for the dependency error. Help me diagnose why the frontend still fails after API readiness recovered.

Compare your restored k8s/app.yaml with the complete file in the tabs below.

✔️ Awesome, I've got everything!

Your manifest is restored to the healthy API probe path.

ⓧ I'd like to double check the full code

apiVersion: apps/v1
kind: Deployment
metadata:
  name: incident-api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: incident-api
  template:
    metadata:
      labels:
        app: incident-api
    spec:
      containers:
        - name: app
          image: incident-app:1.0
          imagePullPolicy: Never
          env:
            - name: MODE
              value: api
            - name: PORT
              value: "8080"
          ports:
            - name: http
              containerPort: 8080
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 1
            periodSeconds: 2
---
apiVersion: v1
kind: Service
metadata:
  name: incident-api
spec:
  selector:
    app: incident-api
  ports:
    - name: http
      port: 8080
      targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: incident-frontend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: incident-frontend
  template:
    metadata:
      labels:
        app: incident-frontend
    spec:
      containers:
        - name: app
          image: incident-app:1.0
          imagePullPolicy: Never
          env:
            - name: MODE
              value: frontend
            - name: PORT
              value: "8080"
            - name: BACKEND_URL
              value: http://incident-api:8080
          ports:
            - name: http
              containerPort: 8080
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 1
            periodSeconds: 2
---
apiVersion: v1
kind: Service
metadata:
  name: incident-frontend
spec:
  selector:
    app: incident-frontend
  ports:
    - name: http
      port: 8080
      targetPort: 8080

The incident is recovered. Its value comes from preserving the evidence that separated a running process from a ready endpoint.

Return to runbook.md in Visual Studio Code.
Add this readiness row below the dependency-configuration row:

| Readiness | Frontend cannot reach API while API containers keep running | API Pods are Running but not Ready, and the API EndpointSlice has a blank `ENDPOINTS` column | API access logs show repeated HTTP 404 responses for `/ready` | Readiness probe targets the wrong path | Restore the probe path to `/healthz` | API endpoint addresses return and frontend reports `status=200` |

What does this row preserve?

The row links the user symptom to Pod state, application logs, and EndpointSlice evidence. It records the root cause before the fix.

The verification column captures both control-plane recovery and the restored user request.

Save runbook.md.
Confirm the readiness row appears directly below the dependency-configuration row.

Markdown table looking uneven?

Confirm the new row starts and ends with a pipe character. Keep every evidence field on the same line.

Count the fields against the table header if the columns no longer align. Help me repair the readiness row in my Markdown runbook.

Your runbook now distinguishes process health from traffic readiness. Next, you'll diagnose a Service selector that disconnects healthy Pods from the same request path.

Repair the Service Selector

Your API Pods are Ready again after you repaired the readiness probe. The frontend can reach the API through Kubernetes.

A Service can lose every backend when its selector stops matching Pod labels.

EndpointSlices reveal the missing selection link. You will use that evidence to make a surgical repair.

In this step, get ready to:

Break the API Service selector while the API Pods remain Ready.
Trace the mismatch through Service configuration plus EndpointSlice evidence.
Restore frontend traffic by correcting the Service selector.

Break the Service selector

The API Service selects Pods whose labels match its selector. Changing that selector breaks the routing relationship without changing the Pods themselves.

In VS Code, use the Explorer sidebar to select k8s/app.yaml.
Change the API Service selector from app: incident-api to app: incident-api-wrong.
Save k8s/app.yaml.
Apply the broken selector by running this command:

kubectl apply -f k8s/app.yaml

What does this command do?

The kubectl command reads k8s/app.yaml. Kubernetes updates the API Service selector from that manifest.

Manifest not applying?

Confirm that app: incident-api-wrong stays indented beneath the API Service selector.
Confirm that you saved k8s/app.yaml before applying it.

Help me diagnose why my Kubernetes manifest did not apply.

Confirm that the selector change left the API Pods healthy by running:

kubectl get pods -l app=incident-api

What does this check prove?

The label filter shows the API Pods targeted by your investigation. Their Ready state proves that the workload itself stayed healthy.

Both API Pods remain Ready. The selector edit changed the Service routing boundary.

Before you rerun the frontend probe, do you think Ready API Pods guarantee a successful request?

Return to the frontend port-forward terminal from earlier.
Use your shell history to restart the port-forward command if it has stopped.
Return to the probe terminal from earlier.
Press the Up Arrow key until the frontend probe command appears.
Press Enter to rerun the frontend probe.

You will see status=502 with the API dependency failure message.

This failure is intentional

The frontend still reaches its own healthy Pod. The API Service has no matching backends to receive the dependency request.

Ready Pods can coexist with a broken Service path. The next evidence check exposes that disconnect.

Inspect the selection boundary

A selector-backed Service depends on an exact match between its selector plus the labels on candidate Pods. The EndpointSlice records the endpoints produced by that match.

Inspect the API Pods plus both sides of the selection boundary by running these commands:

kubectl get pods -l app=incident-api
kubectl get service incident-api -o yaml
kubectl get endpointslices -l kubernetes.io/service-name=incident-api

How does this evidence fit together?

The Pod output confirms that the API workloads remain Ready with the app=incident-api label.
The Service output exposes the incorrect app: incident-api-wrong selector.
The EndpointSlice output shows a blank ENDPOINTS column because no Pod labels match that selector.

The evidence isolates the failure at the Service selection boundary. The API process remains healthy throughout the incident.

Restore Service traffic

The evidence points to one configuration value. Restoring the matching selector reconnects the Service to the existing Ready Pods.

Switch back to k8s/app.yaml in VS Code.
Change the API Service selector from app: incident-api-wrong to app: incident-api.
Save k8s/app.yaml.
Return to the terminal where you applied the broken manifest.
Press the Up Arrow key until the apply command from earlier appears.
Press Enter to apply the repaired manifest.

The terminal confirms that Kubernetes applied the manifest. The corrected selector can now repopulate the API EndpointSlice.

Before you run the final checks, what should change in the EndpointSlice once the selector matches the API Pod labels?

Verify the API EndpointSlice by running:

kubectl get endpointslices -l kubernetes.io/service-name=incident-api

What does this result prove?

Endpoint addresses prove that the Service selector matches Ready API Pods again. Kubernetes can route Service traffic to those backends.

You will see endpoint addresses in the ENDPOINTS column.

Still seeing blank endpoints?

Confirm that the API Service selector is app: incident-api.
Confirm that the API Pods remain Ready with the app=incident-api label.
Apply k8s/app.yaml again after saving the corrected selector.

Help me find why my API EndpointSlice still has no addresses.

Return to the probe terminal from earlier.
Press the Up Arrow key until the frontend probe command appears.
Press Enter to rerun the frontend probe.

You will see status=200 with the frontend-to-API success message.

That was precise incident work: your frontend now crosses the repaired Service boundary successfully.

Switch back to runbook.md in VS Code.
Compare the existing Service selector row with the evidence you observed.
Save runbook.md.

The row now captures the symptom plus the evidence. It also preserves the root cause plus the verified repair.

✔️ Awesome, I've got everything!

Your manifest plus runbook now match the recovered incident state.

ⓧ I'd like to double check the full code

The complete recovered files are shown below for comparison.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: incident-api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: incident-api
  template:
    metadata:
      labels:
        app: incident-api
    spec:
      containers:
        - name: app
          image: incident-app:1.0
          imagePullPolicy: Never
          env:
            - name: MODE
              value: api
            - name: PORT
              value: "8080"
          ports:
            - name: http
              containerPort: 8080
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 1
            periodSeconds: 2
---
apiVersion: v1
kind: Service
metadata:
  name: incident-api
spec:
  selector:
    app: incident-api
  ports:
    - name: http
      port: 8080
      targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: incident-frontend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: incident-frontend
  template:
    metadata:
      labels:
        app: incident-frontend
    spec:
      containers:
        - name: app
          image: incident-app:1.0
          imagePullPolicy: Never
          env:
            - name: MODE
              value: frontend
            - name: PORT
              value: "8080"
            - name: BACKEND_URL
              value: http://incident-api:8080
          ports:
            - name: http
              containerPort: 8080
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 1
            periodSeconds: 2
---
apiVersion: v1
kind: Service
metadata:
  name: incident-frontend
spec:
  selector:
    app: incident-frontend
  ports:
    - name: http
      port: 8080
      targetPort: 8080

What should match?

The API Service selector should be app: incident-api. The Deployment labels plus readiness path should remain unchanged.

# Broken Release Incident Runbook

| Incident | User symptom | Kubernetes evidence | Application evidence | Root cause | Fix | Verification |
| --- | --- | --- | --- | --- | --- | --- |
| Dependency configuration | Frontend request fails while API works directly | Frontend and API Pods are Running and Ready | Frontend logs show `backend_request_failed` for the wrong Service name | Incorrect `BACKEND_URL` | Point the frontend to `http://incident-api:8080` | Frontend probe reports `status=200` |
| Readiness | Frontend cannot reach API while API containers keep running | API Pods are Running but not Ready, and the API EndpointSlice has a blank `ENDPOINTS` column | API access logs show repeated HTTP 404 responses for `/ready` | Readiness probe targets the wrong path | Restore the probe path to `/healthz` | API endpoint addresses return and frontend reports `status=200` |
| Service selector | Frontend cannot reach API while API Pods are Ready | Service selector does not match Pod labels and the EndpointSlice has a blank `ENDPOINTS` column | API process remains healthy | Selector changed to `app: incident-api-wrong` | Restore selector `app: incident-api` | Endpoint addresses return and frontend reports `status=200` |

What should the runbook capture?

The Service selector row should connect the failed frontend request to Ready Pods plus blank endpoints. It should record the selector mismatch as the root cause.

Secret mission

Recover a Stalled Image Rollout

A release can stop progressing while users still receive healthy responses. Prove that old replicas keep traffic alive. Recover the API through rollback.

Clean Up Your Resources

Your local Kubernetes incident lab runs entirely on your Linux machine, so it has no ongoing cloud cost. Choose whether to keep it active, pause its containers, or remove its runtime resources.

Resources you used:

One local kind cluster named incident-lab. It contains two Deployments. It also contains two Services with managed EndpointSlices.
One local Docker image tagged incident-app:1.0.

Keep everything running

No action is needed. Choose this option if you want to run more incidents in the existing lab.

Leave the incident-lab cluster running for your next troubleshooting session.
Retain your source files with runbook.md as your reusable incident kit.

Pause - I'll come back to this later

Shut down the cluster container to free its memory while preserving the cluster for another session.

Save the cluster node container name by running this command:

kind get nodes --name incident-lab > .kind-nodes

What Does This Command Save?

This writes the cluster's node container name to .kind-nodes. The next command uses that saved name to stop the correct container.

Stop the saved cluster node by running this command:

docker stop $(cat .kind-nodes)

What Does Stopping the Node Do?

Docker stops the node container without deleting its stored cluster state. You can resume the same lab later.

You'll see the node container name after Docker stops it. Your cluster is now paused.

Cluster Container Did Not Stop?

Confirm Docker Engine is running.
Regenerate .kind-nodes with the first command if the file is empty.

Help me troubleshoot why my local kind cluster container did not stop.

When you return, start the saved node before reconnecting kubectl.

Start the saved cluster node by running this command:

docker start $(cat .kind-nodes)

What Does Starting the Node Do?

Docker restarts the existing node container. Your previous workloads remain inside the cluster.

Reconnect kubectl to the lab by running this command:

kubectl config use-context kind-incident-lab

What Does Selecting the Context Do?

This sets kind-incident-lab as the current kubectl context. Future kubectl commands target the resumed cluster.

Your kubectl context now points back to the resumed incident lab.

Cluster Did Not Resume?

Confirm the node container finished starting.
Run the context command again after the node finishes starting.

Help me troubleshoot why my paused kind cluster did not resume.

Delete - I don't want to use this again

Remove the cluster with its workloads. Keep your source files with runbook.md as a reusable incident kit.

Delete the incident-lab cluster by running this command:

kind delete cluster --name incident-lab

What Does Cluster Deletion Remove?

This deletes the cluster node with both Deployments. It also removes both Services with their managed EndpointSlices.

Cluster Deletion Did Not Complete?

Confirm Docker Engine is running.
Retry the same cluster deletion command.

Help me troubleshoot why my local kind cluster was not deleted.

Remove the local application image by running this command:

docker image rm incident-app:1.0

What Does Image Removal Do?

This removes the local incident-app:1.0 image from Docker. Rebuilding the lab later requires rebuilding the image from your saved source files.

Image Removal Did Not Complete?

Confirm the incident-lab cluster was deleted first.
Confirm no local container still uses incident-app:1.0.

Help me troubleshoot why Docker cannot remove my local incident image.

That's the cleanup complete: the cluster and local image no longer occupy Docker resources. Your source files with runbook.md remain ready for another lab.

Nice Work!

Lab complete! You built a disposable local Kubernetes incident environment that runs a working frontend-to-API request path. You can now isolate failures through evidence-led triage before restoring service.

You've learned how to:

Build a disposable incident lab that runs a two-service application inside a local cluster.
Trace layered failures from the user symptom through Pod state, container logs, readiness, Services, and EndpointSlices.
Produce an evidence-based incident runbook that turns observations into verified repairs.
Complete the optional Secret Mission to recover a stalled image rollout through rollback while healthy replicas preserve service availability.

Ready to quiz yourself?