Troubleshoot Kubernetes Deployments

Debug 5 real-world Kubernetes failures using the SRE troubleshooting method.

Introduction

30 Second Summary

A pod crashes at 2 AM. The on-call engineer stares at a wall of events, unsure where to start. The difference between a 47-minute panic and a 5-minute fix is a systematic debugging method.

In this project, you will deploy intentionally broken applications to a local Kubernetes cluster, then diagnose and fix them using the same workflow that production SREs use during real incidents.

What You'll Build

A Kubernetes Troubleshooting Lab with 5 realistic failure scenarios, a local cluster, and a personal debugging runbook.

By the end of this project, you'll have:

  • A running Minikube cluster with metrics-server enabled for resource monitoring.
  • Hands-on experience diagnosing and fixing 5 distinct production-style failures including CrashLoopBackOff, OOMKilled, ImagePullBackOff, Pending pods, and probe misconfiguration.
  • A personal troubleshooting runbook documenting the systematic 5-layer debugging method with exact commands.
  • Secret Mission: simulate a cascading failure where a node disruption triggers liveness probe timeouts in a dependent service, then debug it using cross-layer event correlation.

Are there any prerequisites?

You need a basic understanding of containers and Docker. No prior Kubernetes experience is required.

Set Up Your Local Kubernetes Environment

Before you can troubleshoot broken deployments, you need a Kubernetes cluster to run them on. A local cluster lets you break things freely without affecting any real services.

In this step, you'll install Docker Desktop, Minikube, and kubectl on your Windows machine. These three tools work together to give you a fully functional Kubernetes cluster running on your laptop.

In this step, get ready to:
  • Install Docker Desktop, Minikube, and kubectl.
  • Start a local Kubernetes cluster with the Docker driver.
  • Enable the metrics-server addon and create a lab namespace.
Install Docker Desktop and Minikube

Minikube needs a container runtime to create your cluster. Docker Desktop provides that runtime through its Docker engine.

  • Press the Windows key to open your search bar.
  • Type PowerShell and press Enter to open it.
  • Check if Docker is already installed by running this command:
docker --version

✔️ I see a version number

Docker is installed. Make sure Docker Desktop is currently running by looking for the whale icon in your system tray (bottom-right of your taskbar).

If the icon is there, you are ready to move on to installing Minikube.

ⓧ Command not found

  • Download Docker Desktop for Windows from the official install page.
  • Double-click Docker Desktop Installer.exe to run the installer.
  • When prompted, choose the WSL 2 backend option (this is the default and recommended setting).
  • Follow the installation wizard and click Finish when complete.
  • Restart your computer if prompted.
  • After restarting, Docker Desktop launches automatically. Wait for it to finish starting (the whale icon in your system tray stops animating).
  • Open a new PowerShell window and confirm the installation by running:
docker --version
  • You should see a version number printed to the terminal.

What is Docker Desktop?

Docker Desktop is an application that runs the Docker engine on your local machine. It provides the container runtime that Minikube uses as a driver to create your Kubernetes cluster.

  • Check if Minikube is already installed by running this command:
minikube version

✔️ I see a version number

Minikube is already installed. Move on to installing kubectl.

ⓧ Command not found

  • Install Minikube using winget by running this command:
winget install Kubernetes.minikube
  • Close and reopen your PowerShell window so the new PATH takes effect.
  • Run minikube version again to confirm it installed successfully.
  • Check if kubectl is already installed by running this command:
kubectl version --client

✔️ I see a version number

kubectl is already installed. You are ready to start your cluster.

ⓧ Command not found

  • Install kubectl using winget by running this command:
winget install -e --id Kubernetes.kubectl
  • Close and reopen your PowerShell window so the new PATH takes effect.
  • Run kubectl version --client again to confirm it installed successfully.

What is kubectl?

kubectl is the command-line tool for communicating with a Kubernetes cluster. Every command you run to inspect pods, read logs, or apply configuration files goes through kubectl.

Winget install not working?

If winget is not recognized, make sure you are running Windows 10 (version 1809 or later) or Windows 11. Winget comes pre-installed with the App Installer package from the Microsoft Store.

help me troubleshoot winget not being recognized in PowerShell on Windows

Start your Minikube cluster

With all three tools installed, you can now spin up a local single-node Kubernetes cluster. Minikube creates this cluster inside a Docker container on your machine.

  • Start your Minikube cluster with the Docker driver by running this command:
minikube start --driver=docker --memory=4096 --cpus=2

What do these flags do?

  • --driver=docker tells Minikube to use Docker Desktop as the container runtime.
  • --memory=4096 allocates 4 GB of RAM to your cluster, giving enough headroom for multiple broken pods.
  • --cpus=2 gives the cluster 2 CPU cores to work with.

This command takes 1-2 minutes. You will see Minikube downloading Kubernetes components and configuring your cluster.

  • Verify the cluster is running by checking the cluster info:
kubectl cluster-info
  • You should see output showing the Kubernetes control plane address (something like https://127.0.0.1:PORT).
  • Confirm the node is ready by running:
kubectl get nodes
  • You should see one node named minikube with a STATUS of Ready.

Minikube start failing?

Make sure Docker Desktop is running before you start Minikube. Check for the whale icon in your system tray. If Docker Desktop is running but the command still fails, try restarting Docker Desktop and running the command again.

help me troubleshoot minikube start failing with the Docker driver on Windows

Verify your environment

Now confirm that your full environment is healthy and create the namespace where all your troubleshooting scenarios will live.

  • Check that all system pods are running across the cluster:
kubectl get pods -A
  • You should see several pods in the kube-system namespace, all with STATUS Running.
  • Enable the metrics-server addon so you can monitor resource usage later:
minikube addons enable metrics-server

Why enable metrics-server?

The metrics-server collects CPU and memory usage data from your pods and nodes. You will use it later to check resource consumption when diagnosing OOMKilled pods.

  • Verify metrics-server is collecting data by running:
kubectl top nodes

Metrics take a moment to appear

If you see an error mentioning metrics not being available yet, wait 1-2 minutes and try again. The metrics-server needs time to start collecting data after it is first enabled.

  • You should see your minikube node listed with CPU and memory percentages.
  • Create the namespace for your troubleshooting lab by running:
kubectl create namespace troubleshoot-lab
  • Confirm the namespace was created by running:
kubectl get namespaces
  • You should see troubleshoot-lab in the list alongside the default system namespaces.

Why use a separate namespace?

Namespaces are like folders that keep your Kubernetes resources organized. By putting all your broken deployments in troubleshoot-lab, you can easily list, delete, or reset everything without affecting the system components.

Your local Kubernetes environment is ready. Next up, you will deploy your first intentionally broken application and watch it crash in real time.

Scenario 1 - CrashLoopBackOff

Your cluster is running and ready to break things. In production, one of the most common failures engineers encounter is CrashLoopBackOff. This status means a container keeps crashing immediately after starting, and Kubernetes is backing off between restart attempts.

In this step, you will deploy an application that is missing a critical environment variable. You will watch it crash, diagnose the root cause using the first two layers of the 5-layer debugging method, then fix it and confirm recovery.

In this step, get ready to:
  • Deploy a broken application that enters CrashLoopBackOff.
  • Diagnose the failure using kubectl describe and kubectl logs.
  • Fix the root cause and verify the pod recovers.
Deploy the broken application

The first scenario simulates a real-world misconfiguration. The application expects a ConfigMap to supply its database connection string, but that ConfigMap does not exist yet.

  • Create a project folder on your Desktop to store your YAML manifests by running these commands:
cd ~/Desktop
mkdir k8s-troubleshoot-lab
cd k8s-troubleshoot-lab
  • Confirm the folder was created by running:
ls

You should see the empty k8s-troubleshoot-lab directory listed in your Desktop output.

  • Create a new file called crash-loop.yaml in your k8s-troubleshoot-lab folder with the following content:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: crash-app
  labels:
    app: crash-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: crash-app
  template:
    metadata:
      labels:
        app: crash-app
    spec:
      containers:
        - name: crash-app
          image: busybox:1.36
          command: ["/bin/sh", "-c"]
          args:
            - |
              if [ -z "$DATABASE_URL" ]; then
                echo "ERROR: DATABASE_URL environment variable is not set"
                exit 1
              fi
              echo "Connected to $DATABASE_URL"
              sleep 3600
          envFrom:
            - configMapRef:
                name: app-config

What does this manifest do?

This deployment creates a single Pod running a lightweight busybox:1.36 container. The container's startup script checks if a DATABASE_URL environment variable exists.

If the variable is missing, the script prints an error and exits with code 1 (application error). The envFrom section tells Kubernetes to pull all environment variables from a ConfigMap named app-config. Since that ConfigMap does not exist yet, the container will fail to start.

  • Save crash-loop.yaml.
  • Confirm the file exists by running:
ls

You should see crash-loop.yaml listed in the output.

Don't see the file?

Make sure you saved the file inside the k8s-troubleshoot-lab folder on your Desktop. Check that the filename ends in .yaml, not .yaml.txt (Windows sometimes adds a hidden .txt extension).

help me find my crash-loop.yaml file

✔️ Awesome, I've got everything!

Great. Double check you have saved the file before continuing.

ⓧ I'd like to double check the full code

apiVersion: apps/v1
kind: Deployment
metadata:
  name: crash-app
  labels:
    app: crash-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: crash-app
  template:
    metadata:
      labels:
        app: crash-app
    spec:
      containers:
        - name: crash-app
          image: busybox:1.36
          command: ["/bin/sh", "-c"]
          args:
            - |
              if [ -z "$DATABASE_URL" ]; then
                echo "ERROR: DATABASE_URL environment variable is not set"
                exit 1
              fi
              echo "Connected to $DATABASE_URL"
              sleep 3600
          envFrom:
            - configMapRef:
                name: app-config
  • Apply the manifest to your troubleshoot-lab namespace by running:
kubectl apply -f crash-loop.yaml -n troubleshoot-lab
  • Watch the pod status in real time by running:
kubectl get pods -n troubleshoot-lab -w

What should I see?

After a few seconds, the pod status will show CreateContainerConfigError. This means Kubernetes cannot start the container because the referenced ConfigMap does not exist. After repeated failures, the status transitions to CrashLoopBackOff.

Press Ctrl+C to stop watching once you see CrashLoopBackOff or CreateContainerConfigError in the STATUS column.

Pod stuck in ContainerCreating?

If the pod shows ContainerCreating for more than 30 seconds, your cluster may still be pulling the busybox:1.36 image. Wait another minute. Minikube pulls images over your internet connection, so speed depends on your network.

help me debug why my pod is stuck in ContainerCreating

Diagnose using the 5-layer method

The pod is failing. In a real incident, this is where most engineers start guessing. Instead, you will follow a systematic approach. Start with kubectl describe pod to read the Events section.

  • First, get the exact pod name by running:
kubectl get pods -n troubleshoot-lab

Copy the full pod name from the NAME column. It will look something like crash-app-xxxxxxxxx-xxxxx.

  • Inspect the pod's events and status by running:
kubectl describe pod [[CRASH_POD="your crash-app pod name"]] -n troubleshoot-lab

What to look for in describe output

Scroll to the Events section at the bottom of the output. Read events from bottom to top (newest first). You will see a warning event mentioning that the ConfigMap app-config was not found.

Also look for the Last State section under Containers. It shows Exit Code: 1 which means the application itself returned an error (as opposed to Exit Code 137 which would mean the kernel killed it).

  • Check the previous container logs to see the application's error message by running:
kubectl logs [[CRASH_POD="your crash-app pod name"]] -n troubleshoot-lab --previous

What do the logs tell you?

The --previous flag retrieves logs from the container's last crashed run. You should see:

ERROR: DATABASE_URL environment variable is not set

Now you can correlate the evidence. The Events section shows CreateContainerConfigError because the ConfigMap does not exist. The application logs confirm the environment variable was never injected. Root cause identified: the ConfigMap app-config is missing from the troubleshoot-lab namespace.

Fix and verify the deployment

You have identified the root cause. The fix is simple: create the missing ConfigMap so Kubernetes can inject the environment variable into the container.

  • Create the missing ConfigMap by running:
kubectl create configmap app-config --from-literal=DATABASE_URL=postgres://localhost:5432/mydb -n troubleshoot-lab

What does this command do?

This creates a ConfigMap named app-config in the troubleshoot-lab namespace. It contains a single key-value pair: DATABASE_URL set to postgres://localhost:5432/mydb. Kubernetes will now be able to resolve the envFrom reference in your deployment.

  • Watch the pod automatically recover by running:
kubectl get pods -n troubleshoot-lab -w

Within 30 seconds to a few minutes (depending on the backoff timer), you should see the pod transition to Running status. Press Ctrl+C once it shows Running.

  • Confirm the application started successfully by checking its logs:
kubectl logs [[CRASH_POD="your crash-app pod name"]] -n troubleshoot-lab

You should see Connected to postgres://localhost:5432/mydb in the output. The application started successfully with the injected environment variable.

Pod still in CrashLoopBackOff?

Kubernetes uses exponential backoff between restart attempts (starting at 10 seconds, capping at around 5 minutes). If your pod was failing for a while, you may need to wait for the current backoff period to expire. You can speed this up by deleting the pod and letting the deployment create a fresh one:

kubectl delete pod [[CRASH_POD="your crash-app pod name"]] -n troubleshoot-lab

A new pod will be created immediately by the deployment controller and should start successfully since the ConfigMap now exists.

help me fix my crash-app pod that is still failing after I created the ConfigMap

Scenario 1 complete. You just diagnosed your first CrashLoopBackOff using nothing but kubectl describe and kubectl logs --previous. Next up, you will tackle a failure where the kernel itself kills your container for using too much memory.

Scenario 2 - OOMKilled

In Scenario 1, you diagnosed a CrashLoopBackOff caused by a missing ConfigMap. Now you will tackle a different type of crash.

Sometimes containers do not fail because of bad configuration. They fail because they run out of memory and the Linux kernel forcibly terminates them. In this step, you will deploy an application whose memory limit is set too low for its workload, then use the 5-layer method to identify the OOMKilled signal and fix it.

In this step, get ready to:
  • Deploy an application that exceeds its memory limit.
  • Diagnose the OOMKilled status using exit codes and describe output.
  • Fix the deployment by increasing the memory limit.
Deploy the memory-hungry application

Kubernetes lets you set memory limits on containers so a single runaway process cannot consume all the resources on a node. When a container exceeds its memory limit, the kernel's OOM (Out of Memory) killer terminates it immediately with no graceful shutdown.

  • Create a new file called oom-killed.yaml in the same folder where you created crash-loop.yaml. Paste in the following content:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: oom-app
  labels:
    app: oom-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: oom-app
  template:
    metadata:
      labels:
        app: oom-app
    spec:
      containers:
        - name: oom-app
          image: busybox:1.36
          command: ["/bin/sh", "-c"]
          args:
            - |
              echo "Allocating memory..."
              dd if=/dev/zero of=/dev/null bs=1M count=200
              # Alternative: use a loop to consume memory
              i=0
              while [ $i -lt 200 ]; do
                head -c 1048576 /dev/urandom | cat > /tmp/memfile_$i
                i=$((i + 1))
              done
              sleep 3600
          resources:
            requests:
              memory: "32Mi"
              cpu: "50m"
            limits:
              memory: "64Mi"
              cpu: "100m"

What does this manifest do?

This deployment creates a container that deliberately tries to allocate far more memory than its limit allows.

  • The resources.limits.memory: "64Mi" line caps the container at 64 megabytes of RAM.
  • The shell script inside args attempts to write 200 one-megabyte files to /tmp, which quickly exceeds the 64Mi limit.
  • When the container crosses 64Mi, the kernel kills the process immediately. Kubernetes then reports the pod status as OOMKilled.
  • Save oom-killed.yaml.
  • Apply the manifest to your cluster by running this command:
kubectl apply -f oom-killed.yaml -n troubleshoot-lab
  • Watch the pod status in real time by running:
kubectl get pods -n troubleshoot-lab -w

Within 10-30 seconds, you should see the pod transition from Running to OOMKilled and then into CrashLoopBackOff as Kubernetes tries to restart it repeatedly.

  • Press Ctrl+C to stop watching once you see the OOMKilled or CrashLoopBackOff status.

Pod stays in ContainerCreating?

If the pod is stuck in ContainerCreating for more than a minute, it may be pulling the busybox:1.36 image for the first time. Wait another minute for the pull to finish.

If you already used busybox in Scenario 1, check that your YAML indentation is correct. YAML is whitespace-sensitive.

help me debug why my oom-killed pod is stuck in ContainerCreating

Diagnose the OOM kill

You know the pod is crashing, but you need to confirm the root cause. The same diagnostic workflow from Scenario 1 applies here. Start with kubectl describe pod.

  • Get the name of your OOM pod by running:
kubectl get pods -n troubleshoot-lab
  • Find the pod whose name starts with oom-app- and note its full name.
  • Run the describe command to inspect the pod's state. Replace your-pod-name with the pod name you noted:
kubectl describe pod [[OOM_POD_NAME="your-pod-name"]] -n troubleshoot-lab
  • Scroll up in the output and look for the Last State section. You should see:
Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137

What does Exit Code 137 mean?

Exit Code 137 is calculated as 128 + 9. The 9 represents SIGKILL, which is the signal the Linux kernel's OOM killer sends to terminate a process that has exceeded its memory allocation.

This is different from Exit Code 1 (the application error you saw in Scenario 1). Exit Code 137 tells you the process did not crash on its own. The kernel killed it because it tried to use more memory than allowed.

  • Check actual memory usage before the pod gets killed by running:
kubectl top pod [[OOM_POD_NAME="your-pod-name"]] -n troubleshoot-lab

Timing matters for kubectl top

The kubectl top command only works while the container is running. Since this pod crashes quickly, you may see an error saying the pod has no metrics. That is fine. The kubectl describe output already confirmed the OOMKilled diagnosis.

Fix by adjusting resource limits

The fix for an OOMKilled pod is straightforward once you have confirmed the diagnosis. You need to increase the memory limit so the container has enough room to do its work.

  • Edit the live deployment to increase the memory limit to 256Mi by running:
kubectl edit deployment oom-app -n troubleshoot-lab

What does kubectl edit do?

This command opens the live deployment configuration in your default text editor. You will change the memory values directly, save the file, and Kubernetes will automatically roll out the update.

  • In the editor, find the resources section under containers.
  • Change memory: "64Mi" under limits to memory: "256Mi".
  • Save and close the editor. Kubernetes will confirm the edit was applied.
  • Verify the pod stabilizes by running:
kubectl get pods -n troubleshoot-lab

You should see the oom-app pod in a Running state with 0 recent restarts.

  • Confirm no more OOMKilled events appear by running describe on the new pod:
kubectl describe pod [[OOM_POD_NAME="your-new-pod-name"]] -n troubleshoot-lab

The Events section should show a clean startup with no OOMKilled entries.

Still seeing OOMKilled after editing?

Make sure you edited the limits field, not just the requests field. The limit is what the kernel enforces. The request is only used for scheduling.

If the old pod is still showing, wait 30 seconds for Kubernetes to terminate it and start the new one. Run kubectl get pods -n troubleshoot-lab again to check the updated pod.

help me fix my OOMKilled pod that keeps crashing after I edited the deployment

✔️ Awesome, I've got everything!

Great. Double check you have saved oom-killed.yaml and that your pod is stable in Running state.

ⓧ I'd like to double check the full code

Here is the full oom-killed.yaml file for comparison:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: oom-app
  labels:
    app: oom-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: oom-app
  template:
    metadata:
      labels:
        app: oom-app
    spec:
      containers:
        - name: oom-app
          image: busybox:1.36
          command: ["/bin/sh", "-c"]
          args:
            - |
              echo "Allocating memory..."
              dd if=/dev/zero of=/dev/null bs=1M count=200
              # Alternative: use a loop to consume memory
              i=0
              while [ $i -lt 200 ]; do
                head -c 1048576 /dev/urandom | cat > /tmp/memfile_$i
                i=$((i + 1))
              done
              sleep 3600
          resources:
            requests:
              memory: "32Mi"
              cpu: "50m"
            limits:
              memory: "64Mi"
              cpu: "100m"

Remember, your live deployment now has memory: "256Mi" in the limits section because you edited it with kubectl edit. The file on disk still shows the original broken state.

Two scenarios down. You now know how to spot both application-level crashes (Exit Code 1) and kernel-level kills (Exit Code 137). Next up, you will tackle a deployment that cannot even pull its container image.

Scenario 3 - ImagePullBackOff

You have now fixed two deployment failures: a missing ConfigMap and a memory limit that was too low. Both of those issues surfaced after the container image was pulled successfully.

This next scenario introduces a failure that happens before the container even starts. When Kubernetes cannot find the container image you specified, the pod gets stuck in a state called ImagePullBackOff. You will deploy a manifest referencing a non-existent image tag, diagnose the pull failure, then fix it with a rolling update.

In this step, get ready to:
  • Deploy a manifest with a non-existent container image tag.
  • Diagnose the ImagePullBackOff status using describe and events.
  • Fix the image reference and confirm the rollout completes.
Deploy with a bad image tag

A container image tag tells Kubernetes exactly which version of a container to download from a registry. If the tag does not exist, the pull will fail indefinitely.

  • Create a new file called image-pull.yaml in your project folder by running:
New-Item image-pull.yaml
  • Open image-pull.yaml and paste the following deployment manifest:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: image-app
  labels:
    app: image-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: image-app
  template:
    metadata:
      labels:
        app: image-app
    spec:
      containers:
        - name: nginx
          image: nginx:nonexistent-tag-v99
          ports:
            - containerPort: 80

What does this manifest do?

This deployment tries to pull nginx:nonexistent-tag-v99 from Docker Hub. That tag does not exist, so the kubelet will fail to download the image and the pod will never start.

In production, this happens when someone typos a tag in CI/CD, references a deleted image, or points to a private registry without credentials.

  • Save image-pull.yaml.
  • Apply the manifest to your troubleshooting namespace by running:
kubectl apply -f image-pull.yaml -n troubleshoot-lab
  • Watch the pod status by running:
kubectl get pods -n troubleshoot-lab -w

After a few seconds, the pod should show a status of ErrImagePull followed by ImagePullBackOff. Press Ctrl+C to stop watching once you see this.

What is ImagePullBackOff?

Kubernetes first reports ErrImagePull when a pull attempt fails. After repeated failures, it switches to ImagePullBackOff and retries with increasing delays. The pod never reaches Running because the container image cannot be downloaded.

Not seeing ImagePullBackOff?

Make sure you applied the manifest to the correct namespace (-n troubleshoot-lab). If the pod shows Running, double check that image-pull.yaml references nginx:nonexistent-tag-v99 and not a valid tag.

Help me debug why my pod is not showing ImagePullBackOff.

Diagnose the image pull failure

The pod is stuck, but the status alone does not tell you which image failed or why. You need the Events section from kubectl describe to see the exact error the registry returned.

  • Get the pod name by running:
kubectl get pods -n troubleshoot-lab -l app=image-app
  • Describe the pod to see the full event history by running:
kubectl describe pod -l app=image-app -n troubleshoot-lab

What should I look for in the output?

Scroll to the Events section at the bottom. Read from the bottom up (newest first). You should see an event with reason Failed that mentions the image name and the phrase manifest unknown.

This confirms that the registry does not have an image tagged nonexistent-tag-v99. The fix is to update the deployment to use a tag that actually exists.

Fix the image reference

Instead of editing the YAML file and reapplying, you can fix the image directly on the live deployment using kubectl set image. This triggers a rolling update that replaces the broken pod with one using the correct image.

  • Update the deployment to use a valid nginx image by running:
kubectl set image deployment/image-app nginx=nginx:1.27-alpine -n troubleshoot-lab

What does this command do?

The syntax is kubectl set image deployment/<name> <container-name>=<new-image>. Here, nginx is the container name defined in the manifest, and nginx:1.27-alpine is a valid lightweight image from Docker Hub.

Kubernetes immediately creates a new pod with the correct image and terminates the stuck one.

  • Watch the rollout complete by running:
kubectl rollout status deployment/image-app -n troubleshoot-lab

You should see output ending with a message confirming the deployment successfully rolled out.

  • Verify the pod is running with the correct image by running:
kubectl describe pod -l app=image-app -n troubleshoot-lab | findstr Image

You should see Image: nginx:1.27-alpine in the output, confirming the fix.

Rollout not completing?

If kubectl rollout status hangs, the new image might also be invalid. Run kubectl get pods -n troubleshoot-lab to check the latest pod status. Make sure you typed nginx:1.27-alpine exactly as shown.

Help me fix my image-app deployment rollout.

✔️ Awesome, I've got everything!

Great. Double check you have saved image-pull.yaml and that your image-app pod shows Running status.

ⓧ I'd like to double check the full code

Here is the full image-pull.yaml file you created (the original broken manifest). The live deployment has been fixed via kubectl set image to use nginx:1.27-alpine.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: image-app
  labels:
    app: image-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: image-app
  template:
    metadata:
      labels:
        app: image-app
    spec:
      containers:
        - name: nginx
          image: nginx:nonexistent-tag-v99
          ports:
            - containerPort: 80

You have now diagnosed and fixed an ImagePullBackOff failure. The key signal was the manifest unknown message in the Events section. Next up, you will tackle a pod that gets stuck in Pending because it requests more resources than the cluster has available.

Scenario 4 - Pending Pod

In the previous scenarios, your pods at least attempted to start. They crashed, ran out of memory, or failed to pull an image. A Pending pod is different. It never starts at all.

When a pod requests more CPU or memory than any node in the cluster can provide, the Kubernetes scheduler cannot place it anywhere. The pod sits in Pending state indefinitely. In this step, you will deploy an impossibly large pod, diagnose the scheduling failure, and fix the resource requests.

In this step, get ready to:
  • Deploy a pod requesting far more resources than your Minikube node has.
  • Diagnose the FailedScheduling event using kubectl describe.
  • Fix the resource requests to fit within your cluster capacity.
Deploy an over-requesting pod

Your Minikube cluster was started with 4096 MB of memory and 2 CPUs. You are about to deploy a pod requesting 32Gi of memory and 16 CPUs. The scheduler will have no choice but to leave it Pending.

  • Create a file called pending-pod.yaml in your project folder and paste in the following manifest:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pending-app
  labels:
    app: pending-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pending-app
  template:
    metadata:
      labels:
        app: pending-app
    spec:
      containers:
        - name: pending-app
          image: busybox:1.36
          command: ["sleep", "3600"]
          resources:
            requests:
              memory: "32Gi"
              cpu: "16"
            limits:
              memory: "32Gi"
              cpu: "16"

What does this manifest do?

This deployment creates a simple container that just sleeps. The interesting part is the resources block. It requests 32Gi of memory and 16 CPUs before the pod can be scheduled.

Your Minikube node only has 4 GB of memory and 2 CPUs. The scheduler will never find a node large enough to place this pod.

  • Save pending-pod.yaml.
  • Apply the manifest to your troubleshoot-lab namespace by running this command:
kubectl apply -f pending-pod.yaml -n troubleshoot-lab
  • Watch the pod status by running:
kubectl get pods -n troubleshoot-lab -w

You should see the pod stuck in Pending status. It will never transition to Running on its own. Press Ctrl+C to stop watching.

Not seeing Pending status?

If the pod shows ContainerCreating or Running, double-check that your pending-pod.yaml has memory: "32Gi" and cpu: "16" in the requests block. These values must be far larger than your node capacity.

help me figure out why my pod is not stuck in Pending status

Diagnose why the pod is pending

A Pending pod has a different debugging entry point than a crashing pod. There are no container logs because the container never started. The Events section of kubectl describe holds the answer.

  • Get the pod name by running:
kubectl get pods -n troubleshoot-lab
  • Describe the pending pod to see its events by running:
kubectl describe pod -l app=pending-app -n troubleshoot-lab

What should I see?

Scroll down to the Events section at the bottom. You should see a FailedScheduling event with a message like: 0/1 nodes are available: 1 Insufficient memory, 1 Insufficient cpu.

This tells you exactly what happened. The scheduler tried to place the pod, found zero nodes with enough resources, and gave up.

  • Check your node's actual capacity by running:
kubectl describe node minikube

What am I looking for in the node description?

Look for the Allocatable section. This shows the maximum resources that can be assigned to pods. Compare these values against the 32Gi memory and 16 CPUs your pod requested.

You will see something like cpu: 2 and memory: ~4Gi in the allocatable section. Your pod asked for 8x the available CPU and 8x the available memory.

Fix the resource requests

The fix is straightforward. Reduce the resource requests to values your single-node cluster can actually provide.

  • Edit the deployment directly in the cluster by running:
kubectl edit deployment pending-app -n troubleshoot-lab

What does kubectl edit do?

This opens the live deployment manifest in your default text editor. You will change the resource values directly in the editor, save the file, and close it. Kubernetes applies your changes immediately.

  • In the editor, find the resources section under containers.
  • Change memory: "32Gi" to memory: "256Mi" in both requests and limits.
  • Change cpu: "16" to cpu: "250m" in both requests and limits.
  • Save and close the editor.

What do these new values mean?

The value 256Mi means 256 mebibytes of memory. The value 250m means 250 millicores of CPU, which is one quarter of a single CPU core. These values fit comfortably within your Minikube node's capacity.

  • Watch the pod get scheduled and reach Running state by running:
kubectl get pods -n troubleshoot-lab -w

You should see the pending-app pod transition from Pending to ContainerCreating to Running. Press Ctrl+C to stop watching once it reaches Running.

  • Confirm the pod is healthy by running:
kubectl get pods -n troubleshoot-lab

All pods in the namespace should now show Running status with zero restarts for pending-app.

Pod still showing Pending?

If the pod is still stuck, run kubectl describe pod -l app=pending-app -n troubleshoot-lab again. If you still see Insufficient memory or Insufficient cpu in the Events, your edit did not save correctly.

Run kubectl get deployment pending-app -n troubleshoot-lab -o yaml to see the current resource values in the cluster. Verify they show 256Mi and 250m.

help me fix my pending pod after editing resources

✔️ Awesome, I've got everything!

Great. Double check you have saved pending-pod.yaml (the original broken manifest) and confirmed your pending-app pod is now Running in the cluster.

ⓧ I'd like to double check the full code

Here is the original pending-pod.yaml manifest (this is the broken version you saved to disk):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pending-app
  labels:
    app: pending-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pending-app
  template:
    metadata:
      labels:
        app: pending-app
    spec:
      containers:
        - name: pending-app
          image: busybox:1.36
          command: ["sleep", "3600"]
          resources:
            requests:
              memory: "32Gi"
              cpu: "16"
            limits:
              memory: "32Gi"
              cpu: "16"

The live deployment in your cluster should now have resource requests of 256Mi memory and 250m CPU (changed via kubectl edit).

You have now diagnosed and fixed all four common pod failure states: CrashLoopBackOff, OOMKilled, ImagePullBackOff, and Pending. Next up, you will tackle a trickier scenario where the pod appears healthy but Kubernetes keeps killing it anyway.

Scenario 5 - Liveness Probe Misconfiguration

In the previous scenario, you fixed a pod stuck in Pending by aligning resource requests with your cluster's capacity.

Now you are tackling one of the trickiest failure modes in Kubernetes: a liveness probe that is misconfigured.

The container starts fine and the application is perfectly healthy, but Kubernetes keeps killing and restarting it because the probe is checking the wrong endpoint. The pod never crashes on its own. This makes it look like the application is broken when the real problem is in the YAML configuration.

In this step, get ready to:
  • Deploy a container with a liveness probe checking the wrong port and path.
  • Diagnose why Kubernetes repeatedly kills a healthy container.
  • Fix the probe configuration to match the application's actual health endpoint.
Deploy with a misconfigured probe

A liveness probe tells Kubernetes how to check if your container is still alive. If the probe fails repeatedly, the kubelet kills the container and restarts it.

In this scenario, you will deploy an nginx container that serves traffic on port 80. The liveness probe, however, checks /healthz on port 8080, which nginx never listens on.

  • Create a file called probe-fail.yaml in your project folder by running:
New-Item probe-fail.yaml
  • Open probe-fail.yaml and paste the following deployment manifest:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: probe-app
  labels:
    app: probe-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: probe-app
  template:
    metadata:
      labels:
        app: probe-app
    spec:
      containers:
        - name: nginx
          image: nginx:1.27-alpine
          ports:
            - containerPort: 80
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 0
            periodSeconds: 5
            timeoutSeconds: 1
            failureThreshold: 3

What does this manifest do?

The container runs nginx:1.27-alpine which serves HTTP traffic on port 80.

The livenessProbe is configured to send an HTTP GET request to /healthz on port 8080. Since nginx only listens on port 80 and has no /healthz endpoint, every probe attempt will fail with a connection refused error.

With failureThreshold: 3 and periodSeconds: 5, Kubernetes will kill the container after just 15 seconds of failed probes.

  • Save probe-fail.yaml.
  • Apply the manifest to your lab namespace by running:
kubectl apply -f probe-fail.yaml -n troubleshoot-lab
  • Watch the pod status in real time by running:
kubectl get pods -n troubleshoot-lab -w

After about 15-20 seconds, you should see the pod's RESTARTS column incrementing. The STATUS may briefly show Running before flipping back as the kubelet kills the container each time.

  • Press Ctrl+C to stop watching once you see at least 2 restarts.

Pod not restarting?

Make sure you saved probe-fail.yaml before running kubectl apply. Also confirm you included the -n troubleshoot-lab flag.

If you see ImagePullBackOff instead, double-check that the image is nginx:1.27-alpine with no typos.

help me debug why my probe-fail.yaml pod is not restarting as expected

Diagnose the probe failure

The pod keeps restarting, but the application logs show nginx starting successfully every time. This is the classic signal of a probe misconfiguration: the app is healthy, but Kubernetes thinks it is not.

  • Get the pod name by running:
kubectl get pods -n troubleshoot-lab -l app=probe-app
  • Copy the pod name from the NAME column (it looks like probe-app- followed by random characters).
  • Run the describe command to inspect the pod's events:
kubectl describe pod [[PROBE_POD="your probe-app pod name from the previous command"]] -n troubleshoot-lab

What should I see?

Scroll to the Events section at the bottom of the output. You will see Warning events with the reason Unhealthy indicating the liveness probe failed. The message will mention a connection refused error on port 8080.

You will also see a Killing event saying the container failed its liveness probe and will be restarted. This confirms the probe is what is causing the restarts, not the application itself.

  • Confirm the application is healthy by checking its logs:
kubectl logs [[PROBE_POD="your probe-app pod name"]] -n troubleshoot-lab

The logs show nginx starting normally with no errors. This proves the container is healthy and the problem is purely in the probe configuration.

Why is this scenario dangerous in production?

A misconfigured liveness probe kills healthy containers. In production with multiple replicas, this can cascade into all pods restarting simultaneously, causing a full service outage even though the application code is perfectly fine.

The fix is almost always a 2-3 line YAML change, but finding the root cause requires recognizing the pattern: high restart count plus clean application logs equals a probe problem.

Fix the probe configuration

The fix requires updating the probe to check an endpoint that nginx actually serves. Nginx responds with a 200 status on / at port 80, so you will point the probe there.

  • Create a new file called probe-fail-fixed.yaml by running:
New-Item probe-fail-fixed.yaml
  • Open probe-fail-fixed.yaml and paste the corrected deployment manifest:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: probe-app
  labels:
    app: probe-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: probe-app
  template:
    metadata:
      labels:
        app: probe-app
    spec:
      containers:
        - name: nginx
          image: nginx:1.27-alpine
          ports:
            - containerPort: 80
          livenessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3

What changed in the fixed probe?

  • path changed from /healthz to / so the probe hits an endpoint nginx actually serves.
  • port changed from 8080 to 80 to match the port nginx listens on.
  • initialDelaySeconds: 10 gives nginx time to start before the first probe runs.
  • periodSeconds: 10 reduces probe frequency so brief hiccups do not trigger unnecessary restarts.
  • timeoutSeconds: 5 gives the probe more time to get a response, adding a safety margin.
  • Save probe-fail-fixed.yaml.
  • Apply the fixed manifest by running:
kubectl apply -f probe-fail-fixed.yaml -n troubleshoot-lab
  • Watch the pod stabilize by running:
kubectl get pods -n troubleshoot-lab -l app=probe-app -w

You should see a new pod start and stay in Running status with the RESTARTS column staying at 0. Press Ctrl+C once you confirm it is stable (wait at least 30 seconds).

  • Confirm no more probe failures by running:
kubectl describe pod [[PROBE_POD_FIXED="your new probe-app pod name"]] -n troubleshoot-lab

The Events section should show only normal events like Scheduled, Pulled, Created, and Started with no Unhealthy or Killing warnings.

Still seeing restarts?

Make sure the path in your fixed YAML is / (a single forward slash) and the port is 80. A typo in either field keeps the probe misconfigured.

Also verify initialDelaySeconds is set to 10. Without this, the probe may fire before nginx has finished starting.

help me fix my liveness probe that is still failing after I applied the fix

✔️ Awesome, I've got everything!

Great. Double check you have saved both probe-fail.yaml and probe-fail-fixed.yaml.

ⓧ I'd like to double check the full code

Here is the broken manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: probe-app
  labels:
    app: probe-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: probe-app
  template:
    metadata:
      labels:
        app: probe-app
    spec:
      containers:
        - name: nginx
          image: nginx:1.27-alpine
          ports:
            - containerPort: 80
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 0
            periodSeconds: 5
            timeoutSeconds: 1
            failureThreshold: 3

Here is the fixed manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: probe-app
  labels:
    app: probe-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: probe-app
  template:
    metadata:
      labels:
        app: probe-app
    spec:
      containers:
        - name: nginx
          image: nginx:1.27-alpine
          ports:
            - containerPort: 80
          livenessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3

You have now diagnosed and fixed all 5 failure scenarios. Next up, you will compile everything you have learned into a personal troubleshooting runbook that you can reference during real incidents.

Build Your Troubleshooting Runbook

You have now diagnosed and fixed five distinct production-style failures. The debugging commands you ran are powerful, but only if you can recall them under pressure during a real incident.

In this step, you will compile everything into a personal troubleshooting runbook. This is the document you open at 2 AM when a production pod is crashing and your brain is foggy.

In this step, get ready to:
  • Document the 5-layer debugging method with exact commands for each layer.
  • Create a quick-reference exit code table and status-to-action map.
Document the 5-layer method

The 5-layer method is the systematic approach that SREs use to narrow down failures: start broad (cluster health), then work downward until you reach the specific application. Skipping layers is how people spend 47 minutes debugging a pod when the actual problem is a drained node.

  • Create a new file called runbook.md in the same folder as your YAML manifests.
  • Paste the following content into the file:
# Kubernetes Troubleshooting Runbook

## The 5-Layer Method

Always troubleshoot top-down. Do not skip layers.

### Layer 1: Cluster Health
```
kubectl cluster-info
kubectl get --raw /healthz
kubectl get pods -n kube-system
```

### Layer 2: Node Health
```
kubectl get nodes
kubectl describe node <node-name>
kubectl top nodes
```

### Layer 3: Workload Status
```
kubectl get pods -A | findstr /V Running | findstr /V Completed
kubectl get events -A --sort-by='.lastTimestamp'
kubectl get pods -A --sort-by='.status.containerStatuses[0].restartCount'
```

### Layer 4: Networking
```
kubectl get endpoints -n <namespace>
kubectl run dnstest --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default
kubectl get pods -n kube-system -l k8s-app=kube-dns
```

### Layer 5: Application
```
kubectl logs <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace> --previous
kubectl exec -it <pod-name> -n <namespace> -- /bin/sh
```

## First 3 Commands (Every Incident)

1. `kubectl describe pod <pod-name> -n <namespace>`
2. `kubectl logs <pod-name> --previous -n <namespace>`
3. `kubectl get events -n <namespace> --sort-by='.lastTimestamp'`

What does this document?

Each layer targets a different scope of your cluster. Layer 1 confirms the cluster control plane is responsive. Layer 2 checks that nodes are healthy and have capacity. Layer 3 looks at workload-level pod statuses and events.

Layer 4 verifies networking and DNS resolution. Layer 5 digs into the application itself with logs and interactive shells. The "First 3 Commands" section is your muscle-memory shortcut that resolves about 80% of pod failures without needing to go deeper.

  • Save runbook.md.
  • Confirm the file exists by running:
ls runbook.md

You should see runbook.md listed in the output.

Don't see runbook.md?

Make sure you saved the file in the same directory where your YAML manifests are. Run ls without arguments to see all files in your current directory.

If the file is elsewhere, use cd to navigate to the correct folder and re-create it there.

Help me find where my runbook.md file was saved

Create a quick-reference exit code table

Exit codes are the signal that tells you how a container died. Memorizing four numbers lets you instantly narrow down whether the problem is your application code, the kernel, or a graceful shutdown.

  • Open runbook.md and add the following content at the bottom of the file:
## Exit Code Reference

| Exit Code | Signal | Meaning |
|-----------|--------|--------|
| 0 | - | Success (short-lived process in a Deployment = misconfiguration) |
| 1 | - | Application error (check logs) |
| 137 | SIGKILL (9) | OOMKilled or manual kill |
| 143 | SIGTERM (15) | Graceful shutdown |

## Common Status -> Action Map

| Status | First Action |
|--------|-------------|
| CrashLoopBackOff | `kubectl logs --previous` |
| Pending | `kubectl describe pod` (check Events for FailedScheduling) |
| ImagePullBackOff | `kubectl describe pod` (check image name and tag) |
| OOMKilled | `kubectl describe pod` (check Last State, then increase memory limit) |
| CreateContainerConfigError | Check ConfigMap/Secret exists in same namespace |

What does this reference cover?

The Exit Code Reference maps directly to the scenarios you fixed. Exit Code 1 was your CrashLoopBackOff scenario (missing ConfigMap). Exit Code 137 was your OOMKilled scenario (128 + 9 = SIGKILL from the kernel OOM killer).

The Status-to-Action Map gives you a one-line decision tree. When you see a pod status, you immediately know which kubectl command to run first. No guessing, no Googling under pressure.

  • Save runbook.md.
  • Verify the complete file by running:
Get-Content runbook.md

You should see the full runbook content printed to your terminal, including all 5 layers, the first 3 commands list, the exit code table, and the status-to-action map.

Seeing garbled output or missing sections?

Make sure you pasted the second section after the existing content, not replacing it. The file should start with # Kubernetes Troubleshooting Runbook and end with the Status-to-Action Map table.

If sections are missing, re-paste the content from the code block above. Ensure there are no extra blank lines between the tables.

Help me fix my runbook.md file

✔️ Awesome, I've got everything!

Great. Double check you have saved runbook.md and the file includes all five layers, the first 3 commands, the exit code table, and the status-to-action map.

ⓧ I'd like to double check the full code

Here is the complete runbook.md file for comparison:

# Kubernetes Troubleshooting Runbook

## The 5-Layer Method

Always troubleshoot top-down. Do not skip layers.

### Layer 1: Cluster Health
```
kubectl cluster-info
kubectl get --raw /healthz
kubectl get pods -n kube-system
```

### Layer 2: Node Health
```
kubectl get nodes
kubectl describe node <node-name>
kubectl top nodes
```

### Layer 3: Workload Status
```
kubectl get pods -A | findstr /V Running | findstr /V Completed
kubectl get events -A --sort-by='.lastTimestamp'
kubectl get pods -A --sort-by='.status.containerStatuses[0].restartCount'
```

### Layer 4: Networking
```
kubectl get endpoints -n <namespace>
kubectl run dnstest --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default
kubectl get pods -n kube-system -l k8s-app=kube-dns
```

### Layer 5: Application
```
kubectl logs <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace> --previous
kubectl exec -it <pod-name> -n <namespace> -- /bin/sh
```

## First 3 Commands (Every Incident)

1. `kubectl describe pod <pod-name> -n <namespace>`
2. `kubectl logs <pod-name> --previous -n <namespace>`
3. `kubectl get events -n <namespace> --sort-by='.lastTimestamp'`

## Exit Code Reference

| Exit Code | Signal | Meaning |
|-----------|--------|--------|
| 0 | - | Success (short-lived process in a Deployment = misconfiguration) |
| 1 | - | Application error (check logs) |
| 137 | SIGKILL (9) | OOMKilled or manual kill |
| 143 | SIGTERM (15) | Graceful shutdown |

## Common Status -> Action Map

| Status | First Action |
|--------|-------------|
| CrashLoopBackOff | `kubectl logs --previous` |
| Pending | `kubectl describe pod` (check Events for FailedScheduling) |
| ImagePullBackOff | `kubectl describe pod` (check image name and tag) |
| OOMKilled | `kubectl describe pod` (check Last State, then increase memory limit) |
| CreateContainerConfigError | Check ConfigMap/Secret exists in same namespace |

Your runbook is complete. You now have a battle-tested reference document that maps every common failure to a specific debugging command. Next up, you will put this knowledge to the ultimate test with a cascading failure scenario inspired by a real production incident.

Secret mission

Cascading Failure - The Spot Instance Incident

In production, failures rarely happen in isolation. A single node going down can trigger a chain reaction that takes out seemingly unrelated services. In this mission, you will simulate a real-world cascading failure inspired by a payments service outage that took 47 minutes to diagnose.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely on your local machine, so there are no ongoing cloud costs. However, the Minikube cluster and Docker Desktop continue to consume system memory and CPU while running.

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Minikube cluster (running with Docker driver, 4096MB memory, 2 CPUs).
  • Kubernetes namespace troubleshoot-lab with all deployments and services.
  • YAML manifest files on your Desktop (crash-loop.yaml, oom-killed.yaml, image-pull.yaml, pending-pod.yaml, probe-fail.yaml, probe-fail-fixed.yaml, redis-deployment.yaml, payments-service.yaml, payments-service-fixed.yaml).
  • runbook.md troubleshooting runbook file.
  • Docker Desktop, Minikube, and kubectl installed on your machine.

Keep everything running

No action needed. Choose this if you are still actively building or want to revisit the lab later without setup time.

  • Your Minikube cluster stays running and continues to consume approximately 4GB of memory.
  • Docker Desktop stays active in your system tray.
  • All YAML files and your runbook.md remain on your Desktop for future reference.

Pause - I'll come back to this later

Shut down the cluster to free up memory and CPU, but keep all your files and cluster configuration intact so you can restart quickly.

  • Stop the Minikube cluster by running:
minikube stop

This preserves your cluster state. When you want to resume, run:

minikube start --driver=docker
  • You can also quit Docker Desktop from the system tray icon to free additional memory.
  • All your YAML files and runbook.md remain untouched on your Desktop.

Delete - I don't want to use this again

Remove all project resources and start fresh. This frees all memory and disk space used by the lab.

  • Delete the entire Minikube cluster (including all deployments, namespaces, and the ConfigMap) by running:
minikube delete
  • Delete the YAML files and runbook from your Desktop by running:
del crash-loop.yaml, oom-killed.yaml, image-pull.yaml, pending-pod.yaml, probe-fail.yaml, probe-fail-fixed.yaml, redis-deployment.yaml, payments-service.yaml, payments-service-fixed.yaml, runbook.md
  • Uninstall Minikube by running:
winget uninstall --id Kubernetes.minikube
  • Uninstall kubectl by running:
winget uninstall --id Kubernetes.kubectl
  • Uninstall Docker Desktop from Windows Settings.
  • Press the Windows key, search for Add or remove programs, and press Enter.
  • Find Docker Desktop in the list, click the three-dot menu, and select Uninstall.

Nice Work!

Nice Work!

You just built a complete Kubernetes troubleshooting lab and proved you can systematically diagnose production-style deployment failures. That is the exact skill that separates junior from senior DevOps engineers.

You've learned how to:

  • Diagnose and fix 5 distinct Kubernetes deployment failures (CrashLoopBackOff, OOMKilled, ImagePullBackOff, Pending, and probe misconfiguration) using exit codes and event analysis.
  • Apply the 5-layer debugging method (Cluster > Node > Workload > Networking > Application) to isolate root causes in any Kubernetes incident.
  • Build a personal troubleshooting runbook documenting exact commands, exit code references, and a status-to-action map for rapid incident response.
  • Secret Mission: Simulated and debugged a cascading infrastructure failure using cross-layer event correlation, proving you can trace root causes across system boundaries.

Ready to quiz yourself?