Kubernetes Monitoring with Prometheus

Deploy a full Prometheus and Grafana stack to monitor a Kubernetes cluster.

Introduction

30 Second Summary

Imagine your app crashes at 3 AM and you only find out when users start complaining. Production teams avoid this by watching their systems in real time and getting alerts before things break.

In this project, you will deploy a monitoring stack on a local Kubernetes cluster that watches your workloads, visualizes resource trends on dashboards, and fires alerts before containers run out of memory.

What You'll Build

A complete Kubernetes observability stack using Prometheus, Grafana, and Alertmanager running on a local minikube cluster.

By the end of this project, you'll have:

  • A running kube-prometheus-stack deployment with Prometheus, Grafana, Alertmanager, and kube-state-metrics collecting real-time cluster metrics.
  • Custom PromQL queries that reveal CPU usage, memory pressure, and pod restart counts across your workloads.
  • A custom Grafana dashboard with panels showing resource trends and custom PrometheusRule alerting rules that catch memory pressure before containers crash.
  • Secret Mission: Add a recording rule for efficient pre-computed metrics and configure webhook notifications for critical alerts.

Are there any prerequisites?

You should have basic Kubernetes knowledge (pods, deployments, namespaces, kubectl commands) and be familiar with YAML syntax.

Before We Start

Before We Start...

Set Up Your Local Kubernetes Environment

To build a monitoring stack, you first need a running Kubernetes cluster. Production clusters run on cloud infrastructure, but for learning, a local cluster gives you the same experience without the cost.

In this step, you will install minikube (a local Kubernetes cluster tool) and Helm (a package manager for Kubernetes), then spin up your cluster with enough resources to run the full monitoring stack.

In this step, get ready to:
  • Install minikube and verify it works.
  • Install Helm and verify it works.
  • Start your minikube cluster with extra memory for the monitoring stack.
Install minikube

Minikube creates a single-node Kubernetes cluster on your local machine. It also bundles kubectl (the Kubernetes command-line tool) so you can interact with your cluster immediately after installation.

  • Install minikube by running the command for your operating system:

macOS

brew install minikube

What does this command do?

This uses Homebrew (the macOS package manager) to download and install the latest stable version of minikube.

Windows

winget install Kubernetes.minikube

What does this command do?

This uses the Windows Package Manager to download and install the latest stable version of minikube.

  • Verify minikube is installed by running:
minikube version

✔️ I see a version number

Minikube is installed and ready to go. Move on to the next substep.

ⓧ Command not found

Your terminal cannot find minikube. Try closing and reopening your terminal so it picks up the new binary in your PATH. If that does not help, re-run the install command above.

Help me troubleshoot minikube installation.

Install Helm

Helm is the package manager for Kubernetes. You will use it to install the entire Prometheus and Grafana monitoring stack with a single command instead of writing dozens of YAML files by hand.

  • Install Helm by running the command for your operating system:

macOS

brew install helm

What does this command do?

This uses Homebrew to download and install the latest stable version of Helm.

Windows

winget install Helm.Helm

What does this command do?

This uses the Windows Package Manager to download and install the latest stable version of Helm.

  • Verify Helm is installed by running:
helm version

✔️ I see a version number

Helm is installed and ready to go. Move on to the next substep.

ⓧ Command not found

Your terminal cannot find Helm. Try closing and reopening your terminal so it picks up the new binary in your PATH. If that does not help, re-run the install command above.

Help me troubleshoot Helm installation.

Start your minikube cluster

The monitoring stack you will install (Prometheus, Grafana, Alertmanager, and several exporters) needs more memory and CPU than minikube allocates by default. You will start the cluster with extra resources using Docker Desktop as the virtualization driver.

Why Docker as the driver?

Minikube supports multiple drivers (Hyper-V, VirtualBox, Docker). The Docker driver works identically on macOS and Windows, making it the simplest cross-platform option.

It runs Kubernetes inside a Docker container rather than a full virtual machine, which means faster startup times.

  • Make sure Docker Desktop is installed and running on your machine. If you do not have it yet, download it from docker.com/get-started.
  • Confirm Docker is running by opening Docker Desktop. You should see a green indicator showing the engine is active.

Docker Desktop not running?

If Docker Desktop is not open, press Cmd+Space (macOS) or the Windows key (Windows) to open search. Type Docker Desktop and press Enter to launch it. Wait for the engine to start before continuing.

On Windows, make sure Docker Desktop is set to use Linux containers (the default). You can check this in the Docker Desktop system tray icon. Help me troubleshoot Docker Desktop.

  • Start your minikube cluster with extra resources by running this command:
minikube start --memory=4096 --cpus=2 --driver=docker

What does this command do?

  • --memory=4096 allocates 4 GB of RAM to the cluster. The default 2 GB is not enough for the monitoring stack.
  • --cpus=2 allocates 2 CPU cores to the cluster for processing metrics and running dashboards.
  • --driver=docker tells minikube to use Docker Desktop as the virtualization backend.

This command takes 1-3 minutes to complete. You will see progress messages as minikube downloads images and starts the Kubernetes control plane.

  • Once the command finishes, verify your cluster is running by checking the node status:
kubectl get nodes

You should see one node with a Ready status. This means your local Kubernetes cluster is up and accepting workloads.

✔️ I see one node in Ready state

Your cluster is running and ready for the next step.

ⓧ I see an error or the node is NotReady

  • Make sure Docker Desktop is running. If it was just installed, restart your computer and try again.
  • If you see a timeout error, delete the cluster and recreate it by running:
minikube delete
minikube start --memory=4096 --cpus=2 --driver=docker

Help me troubleshoot my minikube cluster.

Your local Kubernetes environment is ready. Next up, you will deploy the full Prometheus and Grafana monitoring stack onto this cluster using Helm.

Deploy the kube-prometheus-stack

Your minikube cluster is running and ready for workloads. Right now, you have a healthy Kubernetes node, but zero visibility into what is happening on it.

In this step, you will deploy the kube-prometheus-stack using Helm. This single chart bundles six monitoring components into one install, giving you full observability over your cluster in minutes.

In this step, get ready to:
  • Add the Prometheus community Helm chart repository.
  • Install the kube-prometheus-stack into a dedicated monitoring namespace.
  • Verify the Prometheus UI is collecting metrics from your cluster.
Add the Helm repository and install the monitoring stack

Helm charts are hosted in repositories, similar to how packages live in package registries. The Prometheus community maintains an official repository with the kube-prometheus-stack chart.

  • Add the Prometheus community Helm repository, update it, and install the monitoring stack by running these commands:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus-stack prometheus-community/kube-prometheus-stack --namespace monitoring --create-namespace

What do these commands do?

  • helm repo add registers the Prometheus community chart repository so Helm knows where to find the chart.
  • helm repo update fetches the latest chart index from the repository.
  • helm install prometheus-stack deploys the chart with the release name prometheus-stack. The --namespace monitoring --create-namespace flags create a new namespace called monitoring and install everything into it.
  • Wait for the install to complete. You should see output ending with STATUS: deployed and a list of notes about the release.

Seeing a connection error?

Make sure your minikube cluster is still running. Run minikube status to confirm all components show Running. If the cluster stopped, start it again with minikube start.

If you see a timeout error, your cluster may not have enough resources. Confirm you started minikube with --memory=4096 --cpus=2 in Step 1.

Help me troubleshoot this Helm install error

Verify all monitoring pods are running

The kube-prometheus-stack deploys six components, each as its own pod. It takes a minute or two for all pods to pull their container images and start up.

  • Check the status of all pods in the monitoring namespace by running:
kubectl get pods -n monitoring
  • Wait until every pod shows Running in the STATUS column. If some show ContainerCreating or Init, wait 30 seconds and run the command again.

What is each component?

  • Prometheus server scrapes metrics from all your cluster workloads and stores them as time-series data.
  • Grafana provides dashboards for visualizing those metrics with graphs and panels.
  • Alertmanager receives alerts from Prometheus and routes notifications (email, Slack, webhooks).
  • kube-state-metrics exposes metadata about Kubernetes objects (deployments, pods, nodes) as Prometheus metrics.
  • node-exporter collects hardware and OS-level metrics from each node (CPU, memory, disk).
  • Prometheus Operator manages Prometheus and Alertmanager instances using Kubernetes custom resources.
  • Confirm you see pods for all six components. You should see approximately 6-8 pods total, all with Running status and full readiness (e.g. 1/1 or 2/2 in the READY column).

Pods stuck in Pending or CrashLoopBackOff?

Pending usually means your cluster does not have enough memory. Delete the cluster with minikube delete and recreate it with minikube start --memory=4096 --cpus=2 --driver=docker, then re-run the Helm install commands above.

CrashLoopBackOff can indicate a transient startup issue. Wait 2-3 minutes for Kubernetes to retry. If a pod stays in this state for more than 5 minutes, run kubectl logs <pod-name> -n monitoring to see the error.

Help me troubleshoot my monitoring pods

Access the Prometheus UI

Prometheus exposes a web UI for running queries and checking the health of scraped targets. Since your cluster runs locally inside minikube, you need to forward the Prometheus service port to your machine.

  • Forward the Prometheus service to your local machine by running:
kubectl port-forward -n monitoring svc/prometheus-stack-kube-prom-prometheus 9090:9090

What does this command do?

This command connects port 9090 on your local machine to port 9090 on the Prometheus service running inside the cluster. As long as this command is running in your terminal, you can access the Prometheus UI through your browser.

  • Open your browser and navigate to http://localhost:9090.
  • Click Status in the top navigation bar, then click Targets.
  • Verify you see a list of scrape targets, most showing a green UP state. This confirms Prometheus is actively collecting metrics from your cluster.

Keep this terminal open

The port-forward command occupies the terminal session while it runs. Open a new terminal tab or window for any subsequent commands. If you close this terminal or press Ctrl+C, the port-forward stops and the Prometheus UI becomes inaccessible.

Browser showing connection refused?

Make sure the port-forward command is still running in your terminal. If it exited, re-run it.

If you see an error about port 9090 already being in use, another application is using that port. Stop the other application or use a different local port: kubectl port-forward -n monitoring svc/prometheus-stack-kube-prom-prometheus 9091:9090 and then open http://localhost:9091 instead.

Help me access the Prometheus UI

Your full monitoring stack is live and scraping metrics from every component in the cluster. Next up, you will deploy a stress-test workload that generates visible CPU and memory pressure for Prometheus to track.

Deploy a Stress-Test Workload

Your monitoring stack is deployed and scraping cluster-level metrics. However, the default system pods don't generate the kind of resource pressure that makes monitoring dashboards interesting.

In this step, you'll deploy a container that deliberately consumes memory close to its configured limit. This creates visible resource pressure in Prometheus, the same kind of pressure that triggers OOMKilled crashes in production environments.

In this step, get ready to:
  • Create a stress-test deployment YAML with explicit resource limits.
  • Deploy the workload to your cluster and verify it's running.
  • Confirm metrics appear in the Prometheus UI using PromQL queries.
Create the stress-test deployment manifest

Prometheus only becomes useful when there are workloads generating real resource consumption. You'll create a deployment that uses the polinux/stress image to allocate 100MB of memory inside a container limited to 128Mi.

  • Create a new file called stress-deployment.yaml and paste in the following YAML:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: memory-stress
  namespace: default
  labels:
    app: memory-stress
spec:
  replicas: 1
  selector:
    matchLabels:
      app: memory-stress
  template:
    metadata:
      labels:
        app: memory-stress
    spec:
      containers:
        - name: stress
          image: polinux/stress
          command: ["stress"]
          args: ["--vm", "1", "--vm-bytes", "100M", "--vm-hang", "1"]
          resources:
            requests:
              memory: "64Mi"
              cpu: "50m"
            limits:
              memory: "128Mi"
              cpu: "200m"

What does this YAML do?

This manifest creates a Deployment called memory-stress that runs one replica of the polinux/stress container image.

  • --vm 1 --vm-bytes 100M spawns one process that continuously allocates 100MB of memory.
  • --vm-hang 1 keeps that memory held for 1 second before freeing and re-allocating, creating a sustained pressure pattern.
  • The resources.requests section tells Kubernetes the minimum resources to reserve (64Mi memory, 50m CPU).
  • The resources.limits section sets the hard ceiling (128Mi memory, 200m CPU). If the container exceeds 128Mi, the kernel OOM killer terminates it.

Allocating 100MB against a 128Mi limit means the container runs at roughly 78% memory utilization. Enough pressure to be visible in dashboards without immediately crashing.

  • Save stress-deployment.yaml.

✔️ Awesome, I've got everything!

Great. Make sure you've saved the file before moving on.

ⓧ I'd like to double check the full code

Here's the complete stress-deployment.yaml for comparison:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: memory-stress
  namespace: default
  labels:
    app: memory-stress
spec:
  replicas: 1
  selector:
    matchLabels:
      app: memory-stress
  template:
    metadata:
      labels:
        app: memory-stress
    spec:
      containers:
        - name: stress
          image: polinux/stress
          command: ["stress"]
          args: ["--vm", "1", "--vm-bytes", "100M", "--vm-hang", "1"]
          resources:
            requests:
              memory: "64Mi"
              cpu: "50m"
            limits:
              memory: "128Mi"
              cpu: "200m"
Apply the deployment and verify the pod is running
  • Deploy the stress workload to your cluster by running this command:
kubectl apply -f stress-deployment.yaml
  • Verify the pod starts successfully by running:
kubectl get pods -n default

You should see a pod named memory-stress- followed by a random suffix, with a status of Running.

Pod not reaching Running status?

  • If you see ContainerCreating, wait 30-60 seconds and run the command again. The image needs to download first.
  • If you see ImagePullBackOff, check your internet connection. Your minikube cluster needs network access to pull the polinux/stress image from Docker Hub.
  • If you see Pending, your minikube cluster may not have enough resources. Confirm you started minikube with --memory=4096.

Get more details on a failing pod with this command:

kubectl describe pod -l app=memory-stress

Help me debug why my memory-stress pod isn't starting

Run initial PromQL queries to confirm metrics are flowing

Now that your stress workload is running, Prometheus should be scraping its resource metrics automatically. PromQL is Prometheus's query language for extracting and transforming metric data.

  • Switch back to your browser at http://localhost:9090 (the Prometheus UI from the previous step).
  • Click the Expression input field at the top of the page.
  • Paste in this memory query and click the blue Execute button:
container_memory_working_set_bytes{namespace="default"}

You should see one or more results showing byte values for your memory-stress container. Click the Graph tab to see the metric plotted over time.

Why container_memory_working_set_bytes?

Kubernetes has two memory metrics: container_memory_usage_bytes and container_memory_working_set_bytes. The kernel OOM killer uses the working set metric to decide when to terminate a container.

The working set excludes reclaimable memory (like filesystem cache), so it reflects the memory your container actually needs. This is the metric that triggers OOMKilled events.

  • Now try a CPU query. Clear the expression field, paste in this query, and click Execute:
rate(container_cpu_usage_seconds_total{namespace="default"}[5m])

You should see CPU usage values for your stress container. The numbers represent CPU cores consumed per second, averaged over a 5-minute window.

What does rate() do?

CPU usage in Prometheus is tracked as a counter that only goes up. The rate() function calculates the per-second average rate of increase over a time range (here, 5 minutes). This turns a raw counter into a useful "CPU cores used" value.

The [5m] window smooths out short spikes and gives you a stable trend line. Shorter windows show more noise. Longer windows hide sudden changes.

Not seeing any results?

  • If the query returns empty, your pod may not have been running long enough for Prometheus to scrape it. Wait 1-2 minutes and click Execute again.
  • Confirm your port-forward to Prometheus is still active. If the terminal running it was closed, restart it with: kubectl port-forward -n monitoring svc/prometheus-stack-kube-prom-prometheus 9090:9090.
  • Double-check the query includes namespace="default" to filter for your stress workload specifically.

Help me troubleshoot why my PromQL query returns no data

Your stress workload is generating real metrics that Prometheus is actively scraping. Next up, you'll dive deeper into PromQL to write queries that reveal CPU trends, memory proximity to limits, and restart patterns across your cluster.

Explore PromQL and Key Kubernetes Metrics

Your monitoring stack is collecting metrics from every container in the cluster. But raw metric names like container_memory_working_set_bytes are not useful on their own.

PromQL is Prometheus's query language. It lets you aggregate, filter, and compute rates across time-series data. In this step, you will learn the three most important queries for Kubernetes resource monitoring: CPU usage, memory pressure, and pod restarts.

In this step, get ready to:
  • Query CPU usage per pod using the rate() function.
  • Query memory usage and its proximity to limits to predict OOMKilled events.
  • Query pod restart counts using the increase() function.
Query CPU usage per pod

CPU usage in Prometheus is tracked by a counter metric called container_cpu_usage_seconds_total. Counters only go up, so you need to compute the per-second rate of change to see actual CPU consumption.

  • Return to the Prometheus UI at http://localhost:9090 from earlier.
  • Clear the Expression field, then paste this PromQL query and click Execute:
sum(rate(container_cpu_usage_seconds_total{container!="", pod!=""}[5m])) by (namespace, pod)

What does this query do?

  • rate(...[5m]) calculates the per-second average rate of increase over the last 5 minutes. This turns the ever-increasing counter into a usable "CPU cores consumed" value.
  • {container!="", pod!=""} filters out system-level metrics that do not belong to any specific pod.
  • sum(...) by (namespace, pod) groups the results so you see one value per pod, even if a pod has multiple containers.
  • The 5-minute window smooths out brief spikes. A shorter window (1m) shows more detail but is noisier.
  • Click the Graph tab to see CPU usage over time.
  • You should see your memory-stress pod consuming a small amount of CPU (the stress tool uses mostly memory, so CPU usage will be low but visible).

Seeing empty results?

Make sure the filter labels match exactly: container!="" uses two double-quote characters with nothing between them, not a single character.

If you still see no data, your stress pod may not have been running long enough. Wait 2-3 minutes for Prometheus to scrape enough samples, then try again.

Help me debug empty PromQL results in Prometheus

Query memory usage and proximity to limits

Memory is where most Kubernetes incidents begin. When a container exceeds its memory limit, the kernel terminates it with an OOMKilled status. The key is catching containers that are approaching their limit before the kill happens.

  • Clear the Expression field, paste this query, and click Execute:
sum(container_memory_working_set_bytes{container!=""}) by (namespace, pod)

Why container_memory_working_set_bytes instead of container_memory_usage_bytes?

container_memory_usage_bytes includes cached data that the kernel can reclaim under pressure. It makes memory look higher than it actually is.

container_memory_working_set_bytes is the metric the kernel OOM killer actually uses when deciding whether to terminate a container. This is the metric that predicts OOMKilled events.

  • You should see your memory-stress pod reporting a value around 100MB (approximately 104857600 bytes). This makes sense because the stress tool allocates 100M.

Knowing the raw bytes is useful, but what really matters is how close each pod is to its limit. The next query computes that ratio.

  • Clear the Expression field, paste this ratio query, and click Execute:
(sum(container_memory_working_set_bytes{container!=""}) by (namespace, pod) / sum(kube_pod_container_resource_limits{resource="memory"}) by (namespace, pod)) > 0.8

What does the ratio query do?

This divides current memory usage by the configured memory limit for each pod. The > 0.8 filter shows only pods that are above 80% of their limit.

Your memory-stress pod requests 64Mi but has a limit of 128Mi. It allocates 100M, which puts it at roughly 78% of the limit. Depending on timing, it may appear in results when overhead pushes it above 80%.

  • If you see your memory-stress pod in the results, it means the pod is dangerously close to being OOMKilled. If no results appear, the pod is currently below the 80% threshold.

Where does kube_pod_container_resource_limits come from?

This metric is provided by kube-state-metrics, one of the components installed with the kube-prometheus-stack. It reads the Kubernetes API and exposes cluster state (limits, requests, pod phases) as Prometheus metrics.

Seeing 'no data' for the ratio query?

The ratio query only returns results for pods that have memory limits configured AND are above 80%. If your stress pod is not shown, it is currently below the threshold.

Try removing the > 0.8 filter to see the ratio for all pods regardless of threshold. You should see values between 0 and 1.

Help me understand why my memory ratio query returns no data

Query pod restart counts

Pods that restart frequently signal instability. A single restart might be a fluke, but multiple restarts in an hour indicate a crash loop or resource problem. The increase() function shows how many restarts happened recently, rather than the lifetime total.

  • Clear the Expression field, paste this query, and click Execute:
sum(increase(kube_pod_container_status_restarts_total[1h])) by (namespace, pod)

What does increase() do?

kube_pod_container_status_restarts_total is a counter that tracks the total number of restarts since the pod was created. Looking at the raw counter is not helpful because a pod created last week might show 5 restarts across its entire lifetime.

increase(...[1h]) calculates how much the counter increased in the last hour specifically. This tells you "how many restarts happened recently" which is the signal that matters for incident detection.

  • You should see your memory-stress pod showing 0 (or close to 0) restarts. A healthy stress pod should not be restarting because it stays within its 128Mi limit.
  • Click the Graph tab. A flat line at 0 means stability. Any upward slope would indicate restart events over time.

When restart counts matter most

In a production cluster, you would set an alert on this metric (for example, more than 3 restarts in an hour). You will do exactly that in the next step when you write custom alerting rules.

You now have the three essential queries for Kubernetes resource monitoring. Next up, you will turn these queries into a persistent Grafana dashboard with live-updating panels so you do not have to re-run queries manually.

Build a Custom Grafana Dashboard

You now have PromQL queries that surface CPU usage, memory pressure, and restart counts. Running ad-hoc queries in the Prometheus UI works for debugging, but a production team needs persistent visualizations that show trends at a glance.

In this step, you will access Grafana (the visualization layer included in kube-prometheus-stack) and build a custom dashboard with three panels. Each panel turns one of the queries you explored in the previous step into a live, auto-refreshing chart.

In this step, get ready to:
  • Access and log into the Grafana UI via port-forward.
  • Create a custom dashboard with CPU, memory, and restart panels.
  • Explore the pre-built dashboards that ship with kube-prometheus-stack.
Access the Grafana UI

Grafana is deployed as part of your kube-prometheus-stack release, but it is only accessible inside the cluster by default. You will use kubectl port-forward to expose it on your local machine.

  • Open a new terminal tab or window. Your existing terminal is occupied by the Prometheus port-forward from the previous step.
  • Start the Grafana port-forward by running this command:
kubectl port-forward -n monitoring svc/prometheus-stack-grafana 3000:80

What does this command do?

The Grafana service inside the cluster listens on port 80. This command maps your local port 3000 to the service's port 80, so you can reach Grafana at http://localhost:3000 in your browser.

  • Open your browser and navigate to http://localhost:3000.
  • Log in with username admin and password prom-operator.

Why is the password prom-operator?

The kube-prometheus-stack Helm chart sets the default Grafana admin password to prom-operator in its values.yaml. This is different from Grafana's standalone default of admin.

  • Click Dashboards in the left-side main menu.
  • Click New and select New Dashboard.

You should see an empty dashboard canvas ready for panels.

Can't reach localhost:3000?

  • Make sure the port-forward command is still running in your terminal. If it exited, re-run it.
  • Verify the Grafana pod is running with kubectl get pods -n monitoring. Look for a pod whose name contains grafana in Running state.
  • If port 3000 is already in use on your machine, stop the conflicting process or choose a different local port (e.g., 3001:80).

Ask for help: I can't connect to Grafana on localhost:3000 after running port-forward. What should I check?

Create your dashboard panels

You will add three panels to your dashboard. Each panel uses one of the PromQL queries you explored earlier, turning raw metrics into persistent visualizations.

Start by creating the Pod CPU Usage panel.

  • On the empty dashboard, click + Add visualization.
  • In the data source dialog, select Prometheus.
  • Paste the following PromQL query into the query editor at the bottom of the panel:
sum(rate(container_cpu_usage_seconds_total{namespace="default", container!=""}[5m])) by (pod)
  • Click Refresh to execute the query.
  • In the visualization list on the right, confirm Time series is selected (this is the default).
  • Under Panel options on the right sidebar, set the Title to Pod CPU Usage.
  • Click Save at the top of the panel editor.

What does this query measure?

The rate() function calculates the per-second increase of the CPU counter over a 5-minute window. Wrapping it in sum() by (pod) groups the result so you see one line per pod. The value represents CPU cores consumed.

You should see a time series line for your memory-stress pod showing its CPU consumption over time.

Panel shows No data?

  • Make sure you selected Prometheus as the data source, not the Grafana built-in source.
  • Verify that your memory-stress deployment is still running by checking kubectl get pods -n default in a terminal.
  • Double-check that the query contains namespace="default" (with straight quotes, not curly quotes).

Ask for help: My Grafana panel shows No data even though my stress pod is running. How do I debug this?

Now create the Pod Memory vs Limit panel.

  • Click Back to dashboard at the top left of the panel editor.
  • Click Add in the dashboard header and select Visualization.
  • Select Prometheus as the data source.
  • Paste this query into the query editor:
container_memory_working_set_bytes{namespace="default", container!=""} / on(namespace, pod, container) kube_pod_container_resource_limits{resource="memory"}
  • Click Refresh to execute the query.
  • Confirm Time series is selected as the visualization type.
  • Under Panel options, set the Title to Pod Memory vs Limit.
  • Scroll down to the Thresholds section in the right sidebar.
  • Click + Add threshold.
  • Enter 0.8 as the threshold value and set the color to red.
  • Under Show thresholds, select As lines.
  • Click Save at the top of the panel editor.

What does this query measure?

This query divides a container's actual memory working set by its configured memory limit. A result of 1.0 means the container is at 100% of its limit and will be OOMKilled if it goes higher. The threshold line at 0.8 gives you a visual warning zone.

The on(namespace, pod, container) clause tells Prometheus how to join the two metrics. Without it, the division would not know which limit belongs to which container.

You should see a time series line hovering around 0.78 (since the stress container uses ~100MB against a 128Mi limit), with a red threshold line drawn at 0.8.

Threshold line not visible?

  • Make sure Show thresholds is set to As lines (not Off).
  • Verify the threshold value is 0.8 (not 80). Since the query output is a ratio between 0 and 1, the threshold must also be a decimal.

Ask for help: I added a threshold to my Grafana time series panel but I cannot see the line on the chart.

Finally, create the Pod Restarts panel.

  • Click Back to dashboard.
  • Click Add and select Visualization.
  • Select Prometheus as the data source.
  • Paste this query into the query editor:
sum(increase(kube_pod_container_status_restarts_total{namespace="default"}[1h])) by (pod)
  • Click Refresh.
  • In the visualization list, click All visualizations and select Stat.
  • Under Panel options, set the Title to Pod Restarts (Last Hour).
  • Click Save.

Why use a Stat panel here?

Restart counts are best shown as a single number rather than a time series graph. The Stat visualization displays one large value per pod, making it immediately obvious if something is restarting frequently. A value of 0 means healthy. Anything above 3 in an hour signals trouble.

You should see a stat value of 0 for your memory-stress pod (it has not crashed because its memory usage stays just below the limit).

Stat panel shows no value?

  • Make sure you selected Stat from the All visualizations list, not from the suggested visualizations (which defaults to Time series).
  • If the value shows N/A, the pod may not have restarted at all. That is expected. A missing time series for the counter still returns empty, which shows as no value in some Grafana versions. This confirms your pod is stable.

Ask for help: My Stat panel in Grafana shows N/A instead of a restart count number.

Save your dashboard and explore pre-built ones
  • Click Back to dashboard to return to the full dashboard view with all three panels.
  • Click Save dashboard at the top right.
  • Enter Custom Kubernetes Monitoring as the dashboard title.
  • Click Save.
  • Click Exit edit to view your completed dashboard.

You should see all three panels displaying live data side by side. The CPU panel shows a time series line, the memory panel shows a ratio line with a red threshold at 0.8, and the restarts panel shows a stat value.

Now explore the dashboards that came pre-installed with kube-prometheus-stack.

  • Click Dashboards in the left-side menu.
  • Click Browse to see all available dashboards.
  • Open the dashboard called Kubernetes / Compute Resources / Pod.

What are these pre-built dashboards?

The kube-prometheus-stack ships with dozens of production-ready dashboards maintained by the Kubernetes monitoring community. They cover node resources, namespace quotas, workload details, and more. These are excellent references for learning advanced Grafana techniques like template variables and repeating panels.

Your custom dashboard focuses on your specific workload. The pre-built ones give you cluster-wide visibility without any extra configuration.

Your monitoring stack now has persistent dashboards that surface resource trends at a glance. Next, you will write custom alerting rules that fire automatically when these metrics cross dangerous thresholds.

Write Custom Alerting Rules

Your Grafana dashboard shows what is happening right now, but it cannot wake you up at 3 AM when a pod is about to crash. Alerting rules close that gap by continuously evaluating PromQL expressions and firing notifications when thresholds are breached.

In this step, you will write a PrometheusRule custom resource that defines three alerting rules. These rules catch memory pressure, frequent restarts, and crash loops before they impact users.

In this step, get ready to:
  • Create a PrometheusRule manifest with three alerting rules.
  • Apply the manifest and verify Prometheus picks up your rules.
  • Trigger an alert and watch it fire in Alertmanager.
Create a PrometheusRule manifest

The Prometheus Operator watches for CRD resources of kind PrometheusRule. When it finds one with the correct label, it automatically loads the rules into Prometheus without a restart.

  • Create a new file called custom-alerts.yaml in the same folder as your stress-deployment.yaml by running:
touch custom-alerts.yaml
  • Confirm the file exists by running:
ls

You should see both stress-deployment.yaml and custom-alerts.yaml in the output.

  • Open custom-alerts.yaml in your editor and paste the following content:
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: custom-pod-alerts
  namespace: monitoring
  labels:
    release: prometheus-stack
spec:
  groups:
    - name: pod-resource-alerts
      interval: 30s
      rules:
        - alert: HighMemoryUsage
          expr: |
            (
              sum(container_memory_working_set_bytes{container!=""}) by (namespace, pod)
              /
              sum(kube_pod_container_resource_limits{resource="memory"}) by (namespace, pod)
            ) > 0.8
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} memory usage above 80%"
            description: "Pod is using {{ $value | humanizePercentage }} of its memory limit for more than 5 minutes. Risk of OOMKilled."

        - alert: PodFrequentRestart
          expr: |
            sum(increase(kube_pod_container_status_restarts_total[1h])) by (namespace, pod) > 3
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} is restarting frequently"
            description: "Pod has restarted {{ $value }} times in the last hour."

        - alert: PodCrashLooping
          expr: |
            kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"} == 1
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} is crash looping"
            description: "Container {{ $labels.container }} in pod {{ $labels.pod }} has been in CrashLoopBackOff for 5 minutes."

What does this manifest do?

The release: prometheus-stack label is critical. The Prometheus Operator only picks up rules that match this label selector (the name of your Helm release).

Each rule has an expr (a PromQL query that returns results when the condition is true) and a for duration (how long the condition must hold before the alert fires).

  • HighMemoryUsage fires when a pod's working set memory exceeds 80% of its limit for 5 minutes. This is the metric the kernel OOM killer uses.
  • PodFrequentRestart fires when a pod restarts more than 3 times in one hour. Frequent restarts often signal a misconfiguration or resource starvation.
  • PodCrashLooping fires when a container enters CrashLoopBackOff for 5 minutes. This is a critical alert because the pod is stuck and cannot self-recover.
  • Save custom-alerts.yaml.

File not saving?

Make sure you have not accidentally indented with tabs instead of spaces. YAML requires spaces for indentation. Most editors have a setting to convert tabs to spaces automatically.

help me fix YAML indentation errors in my custom-alerts.yaml

✔️ Awesome, I've got everything!

Great. Double check you have saved custom-alerts.yaml before moving on.

ⓧ I'd like to double check the full code

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: custom-pod-alerts
  namespace: monitoring
  labels:
    release: prometheus-stack
spec:
  groups:
    - name: pod-resource-alerts
      interval: 30s
      rules:
        - alert: HighMemoryUsage
          expr: |
            (
              sum(container_memory_working_set_bytes{container!=""}) by (namespace, pod)
              /
              sum(kube_pod_container_resource_limits{resource="memory"}) by (namespace, pod)
            ) > 0.8
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} memory usage above 80%"
            description: "Pod is using {{ $value | humanizePercentage }} of its memory limit for more than 5 minutes. Risk of OOMKilled."

        - alert: PodFrequentRestart
          expr: |
            sum(increase(kube_pod_container_status_restarts_total[1h])) by (namespace, pod) > 3
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} is restarting frequently"
            description: "Pod has restarted {{ $value }} times in the last hour."

        - alert: PodCrashLooping
          expr: |
            kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"} == 1
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} is crash looping"
            description: "Container {{ $labels.container }} in pod {{ $labels.pod }} has been in CrashLoopBackOff for 5 minutes."
Apply the PrometheusRule and verify in Prometheus

Now that the manifest exists on disk, you need to apply it to your cluster so the Prometheus Operator can load the rules.

  • Apply the manifest by running:
kubectl apply -f custom-alerts.yaml
  • Verify the resource was created by running:
kubectl get prometheusrule -n monitoring

You should see custom-pod-alerts listed in the output.

  • Switch back to the Prometheus UI at http://localhost:9090 in your browser.
  • Navigate to Status in the top menu, then click Rules.

You should see a group called pod-resource-alerts containing your three rules: HighMemoryUsage, PodFrequentRestart, and PodCrashLooping.

Why do rules appear under Status > Rules?

The Prometheus Operator watches for PrometheusRule resources in the cluster. When it finds one with the matching release label, it regenerates the Prometheus configuration and hot-reloads the rules. No pod restart needed.

Rules not showing up?

Check that the release: prometheus-stack label in your YAML matches your Helm release name exactly. Also verify the namespace is monitoring (the same namespace where the stack is installed).

It can take 30-60 seconds for the Operator to detect the new resource. Refresh the Rules page after a minute if you do not see them immediately.

help me debug why my PrometheusRule is not appearing in the Prometheus rules page

Trigger an alert and observe it in Alertmanager

Your rules are loaded, but none of them are firing yet because the stress workload sits just below the 80% memory threshold. Let's push it over the edge and watch the alerting pipeline in action.

  • Open a new terminal window for the Alertmanager port-forward.
  • Start the port-forward by running:
kubectl port-forward -n monitoring svc/prometheus-stack-kube-prom-alertmanager 9093:9093
  • Open http://localhost:9093 in your browser.

You should see the Alertmanager UI showing any active or silenced alerts. It may be empty right now since no alerts have fired yet.

  • Open another terminal window to increase the stress workload's memory consumption.
  • Patch the deployment to allocate more memory by running:
kubectl patch deployment memory-stress --type='json' -p='[{"op":"replace","path":"/spec/template/spec/containers/0/args","value":["--vm","1","--vm-bytes","110M","--vm-hang","1"]}]'

What does this patch do?

This changes the stress container's memory allocation from 100M to 110M without editing the YAML file. Against the 128Mi limit, 110M puts the pod above the 80% threshold that triggers HighMemoryUsage.

  • Verify the new pod is running by running:
kubectl get pods -l app=memory-stress

You should see the pod restarting with the new memory allocation. Wait for it to reach Running status.

  • Return to the Prometheus UI at http://localhost:9090.
  • Click Alerts in the top navigation menu.

After a few minutes, you should see HighMemoryUsage appear in Pending state. The rule's for: 5m clause means it must stay above 80% for a full 5 minutes before transitioning to Firing.

  • Wait approximately 5 minutes, then refresh the Alerts page.
  • Confirm the alert transitions from Pending to Firing.
  • Switch to the Alertmanager UI at http://localhost:9093 and confirm the fired alert appears there as well.

What is the difference between Pending and Firing?

Pending means the PromQL expression returned results, but the for duration has not elapsed yet. This prevents brief spikes from triggering false alarms.

Firing means the condition held for the full for duration. At this point, Prometheus sends the alert to Alertmanager, which handles routing and notifications.

Alert stuck in Pending or not appearing?

Make sure you waited at least 5 minutes after patching the deployment. The for: 5m clause requires the condition to hold continuously.

If the pod is being OOMKilled and restarting, the memory might drop briefly during restarts. Check kubectl get pods -l app=memory-stress to verify the pod stays in Running state.

help me troubleshoot why my HighMemoryUsage alert is not firing in Prometheus

You have a complete alerting pipeline running on your local cluster. Prometheus evaluates your rules every 30 seconds, holds alerts in Pending until the for window passes, then forwards fired alerts to Alertmanager. Next up, you will extend this setup in the Secret Mission with recording rules and webhook notifications.

Secret mission

Add a Recording Rule and Webhook Alert Notification

Your monitoring stack detects problems. But what happens when an alert fires at 3 AM and nobody is watching the Alertmanager UI? In this mission, you will add a recording rule that pre-computes expensive metrics and configure Alertmanager to push critical alerts to a webhook endpoint. This closes the loop from detection to notification.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely on a local minikube cluster, so there are no ongoing cloud costs. However, the cluster uses approximately 4GB of RAM while running.

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • minikube cluster (all Kubernetes resources live here).
  • kube-prometheus-stack Helm release in the monitoring namespace.
  • Stress-test deployment (memory-stress) in the default namespace.
  • Local YAML files (stress-deployment.yaml, custom-alerts-with-recording.yaml, webhook-receiver.yaml).

Keep everything running

No action needed. Choose this if you are still actively experimenting with PromQL queries, building new dashboard panels, or testing additional alerting rules.

  • Your minikube cluster continues running in the background, using approximately 4GB of RAM and 2 CPU cores.
  • Port-forwards stay active as long as those terminal windows remain open.
  • If you close your terminal, re-run the port-forward commands from the project to access the UIs again.

Pause - I'll come back to this later

Stop the minikube cluster to free up RAM and CPU. Your cluster state is preserved and you can resume later.

  • Close any terminal windows running port-forward commands (press Ctrl+C in each).
  • Stop the minikube cluster by running:
minikube stop
  • When you are ready to resume, run minikube start to bring everything back.

Delete - I don't want to use this again

Remove all project resources and start fresh. This deletes the entire minikube cluster and all Kubernetes resources inside it.

  • Close any terminal windows running port-forward commands (press Ctrl+C in each).
  • Uninstall the Helm release by running:
helm uninstall prometheus-stack -n monitoring
  • Delete the minikube cluster by running:
minikube delete
  • Remove the local YAML files by running:
rm stress-deployment.yaml custom-alerts-with-recording.yaml webhook-receiver.yaml
  • On Windows, use this command instead:
del stress-deployment.yaml, custom-alerts-with-recording.yaml, webhook-receiver.yaml
  • After these commands, all Kubernetes resources, monitoring data, and project files are removed from your machine.

Nice Work!

Nice Work!

You just built a production-grade Kubernetes monitoring and alerting stack from scratch. Instead of waiting for pods to crash and scrambling through logs, you now have real-time dashboards and alerts that catch problems before they reach users.

Here's what you accomplished:

  • Deployed the full kube-prometheus-stack (Prometheus, Grafana, Alertmanager, kube-state-metrics, node-exporter) on a local Kubernetes cluster using Helm.
  • Built a custom Grafana dashboard with panels showing real-time CPU usage, memory trends relative to limits, and pod restart counts.
  • Wrote PrometheusRule custom resources with alerting rules that catch memory pressure before OOMKilled and verified the full alerting pipeline from rule evaluation through to Alertmanager.
  • Secret Mission: Added recording rules for efficient pre-computed namespace metrics and configured webhook notifications for critical alerts.

Ready to quiz yourself?