Monitor Kubernetes with Prometheus

Deploy a full Prometheus and Grafana stack to monitor a local Kubernetes cluster.

Introduction

30 Second Summary

When an app slows down or crashes, the first question is always the same: what changed? In the real world, engineering teams answer that question by watching live metrics from their servers.

In this project, you will deploy a complete monitoring stack on a local Kubernetes cluster. You'll use real-time dashboards to visualize cluster health and diagnose a simulated resource problem.

What You'll Build

A local Kubernetes observability setup where Prometheus collects cluster metrics and Grafana visualizes them in pre-built dashboards.

By the end of this project, you'll have:

A running local Kubernetes cluster with a full Prometheus and Grafana monitoring stack deployed via Helm.
Pre-built Grafana dashboards showing real-time CPU, memory, and pod metrics across your cluster.
A diagnosed resource problem using PromQL queries and dashboard visualizations.
Secret Mission: A custom alerting rule that fires when a pod exceeds its CPU threshold.

Are there any prerequisites?

This project requires a Mac with Homebrew already installed. You'll also need at least 8GB of total RAM since the monitoring stack runs inside a local virtual machine.

Before We Start

Before We Start...

Install Your Tools

Before you can monitor a Kubernetes cluster, you need the tools to create and manage one.

In this step, you'll install four tools using Homebrew. Docker Desktop provides the container runtime. Minikube spins up a local cluster. kubectl sends commands to the cluster. Helm deploys pre-packaged applications.

In this step, get ready to:

Install Docker Desktop and start the Docker engine.
Install Minikube, kubectl, and Helm via Homebrew.
Verify all tools are connected and working.

Install Docker Desktop

Docker Desktop provides the container runtime that Minikube uses to run your Kubernetes cluster locally.

Check if Docker is already installed by running this command:

docker --version

✔️ I see a version number

Docker is already installed on your Mac. Skip ahead to launching it below.

ⓧ Command not found

Install Docker Desktop by running this command:

brew install --cask docker

What does this command do?

The --cask flag tells Homebrew to install a full macOS application (with a GUI) instead of a command-line-only tool.

Docker Desktop is about 1 GB, so this download may take a few minutes.

Seeing a permissions error?

If Homebrew asks for your Mac password, type it and press Enter. The password won't appear on screen as you type.

help me fix a Homebrew permissions error when installing Docker Desktop

Whether Docker was already installed or you just installed it, you need to make sure Docker Desktop is running.

Press Cmd+Space (macOS) to open Spotlight.
Type Docker and press Enter to open Docker Desktop.
Wait for the whale icon in your menu bar to stop animating. Once it is still, the Docker engine is ready.

First time opening Docker Desktop?

Docker Desktop may ask you to accept a service agreement and grant system permissions. Accept both to continue.

You can skip any onboarding survey or sign-in prompts. You do not need a Docker account for this project.

Confirm Docker is available by running this command:

docker --version

You should see a version number printed, confirming the Docker CLI is available.

Still seeing command not found?

If you just installed Docker Desktop, close your terminal and open a new one. The docker command is added to your PATH when Docker Desktop starts for the first time.

help me get docker --version to work after installing Docker Desktop on macOS

Install Minikube, kubectl, and Helm

These three CLI tools work together. Minikube creates the cluster, kubectl sends commands to it, and Helm deploys pre-packaged applications onto it.

Check if any of these tools are already installed by running these commands one at a time:

minikube version
kubectl version --client
helm version

✔️ I see version numbers for all three

All three tools are already installed. Skip ahead to the next substep below.

ⓧ One or more commands were not found

Install all three tools by running this command:

brew install minikube kubectl helm

What if I already have one of these?

Homebrew skips any tool that is already installed at the latest version. Running this command is safe even if you have one or two of the three already.

This install may take a minute or two while Homebrew downloads each tool.

Verify each tool installed successfully by running:

minikube version
kubectl version --client
helm version

Each command should print a version number without errors.

One of the installs failed?

Make sure Homebrew itself is up to date by running brew update first, then retry the install command.

help me fix a failed brew install for minikube, kubectl, or helm on macOS

Verify everything is connected

This final check confirms that the Docker daemon is actively running. This proves your toolchain is ready for the next step.

Confirm the Docker daemon is running by running this command:

docker ps

You should see an empty table with column headers (CONTAINER ID, IMAGE, COMMAND, etc.) and no error messages.

Seeing 'Cannot connect to the Docker daemon'?

This means Docker Desktop is installed but the engine has not finished starting. Check that Docker Desktop is open and the whale icon in your menu bar has stopped animating.

If the icon is still, try quitting Docker Desktop completely and reopening it.

help me connect to the Docker daemon on macOS

Your toolbox is ready. Next up, you'll use Minikube to spin up a real Kubernetes cluster on your Mac.

Create Your Kubernetes Cluster

Your tools are installed and Docker is running. Now it's time to create the Kubernetes cluster where your entire monitoring stack will live.

Minikube creates a single-node Kubernetes cluster running inside Docker on your Mac. It gives you a real Kubernetes environment without needing cloud servers.

In this step, get ready to:

Start a local Kubernetes cluster with enough resources for the monitoring stack.
Explore your cluster's nodes and namespaces.
Inspect the system components Kubernetes runs automatically.

Start Minikube with adequate resources

The monitoring stack you'll deploy later needs a decent amount of memory. You'll give your cluster 4 GB of RAM and 2 CPUs so everything has room to run.

Create your Kubernetes cluster by running this command:

minikube start --memory=4096 --cpus=2

What do these flags mean?

The --memory=4096 flag allocates 4 GB of RAM to the cluster. The --cpus=2 flag gives it two CPU cores.

Without enough resources, the monitoring stack's pods would crash or fail to start. Your Mac should have at least 8 GB of total RAM to comfortably run this cluster.

This command takes 1-3 minutes to complete. Minikube is downloading a Kubernetes image, creating a container in Docker, and starting all the core cluster components.

Wait for the output to finish. You should see a final line that says something like "Done! kubectl is now configured" to use the Minikube cluster.

Seeing an error about Docker?

If you see a message about Docker not being available, make sure Docker Desktop is open and the engine is running. Look for the Docker whale icon in your menu bar. It should not show "Docker Desktop starting" anymore.

If the command fails with an error about insufficient memory, close memory-intensive applications and try again.

Help me troubleshoot Minikube startup errors

Explore your cluster

Your cluster is running. Now you can use kubectl to inspect what's inside it.

Check that your cluster has a node in Ready state by running:

kubectl get nodes

You should see one node named minikube with a STATUS of Ready. This is your single-node cluster.

What is a node?

A node is a machine (physical or virtual) that runs your containers. In production, clusters have many nodes for redundancy and scale. Your Minikube cluster has just one, which is perfect for learning.

View the namespaces Kubernetes created automatically by running:

kubectl get namespaces

You should see four namespaces: default, kube-system, kube-public, and kube-node-lease.

What are namespaces?

Namespaces are a way to organize resources inside a cluster. Think of them like folders. The default namespace is where your applications go by default. The kube-system namespace holds the internal components that keep Kubernetes running.

Later, you'll create a monitoring namespace to keep your Prometheus and Grafana resources separate from everything else.

Understand what you have

Kubernetes runs several system components behind the scenes to manage your cluster. These pods handle everything from networking to scheduling your applications.

View the system pods already running in your cluster by running:

kubectl get pods -n kube-system

You should see several pods with a STATUS of Running. These include the core Kubernetes components.

What are these system pods?

coredns handles DNS resolution inside the cluster so pods can find each other by name.
etcd stores all cluster state and configuration data.
kube-apiserver is the front door to your cluster. Every kubectl command talks to this.
kube-scheduler decides which node a new pod should run on.
kube-controller-manager runs background tasks like restarting crashed pods.
kube-proxy handles network routing so traffic reaches the right pods.

What is a pod?

A pod is the smallest deployable unit in Kubernetes. It wraps one or more containers together with shared networking and storage. When you deploy an application, Kubernetes creates pods to run it.

Your Kubernetes cluster is live with all its system components running. Next up, you'll deploy a full monitoring stack on top of it using a single Helm command.

Deploy the Monitoring Stack

Your cluster is running, but right now you have no way to see what is happening inside it. Monitoring gives you real-time visibility into CPU usage, memory consumption, and pod health.

In this step, you will deploy a full monitoring stack using a single Helm command. This installs Prometheus (for collecting metrics), Grafana (for visualizing them), and several supporting components into your cluster.

In this step, get ready to:

Add the Prometheus community Helm chart repository.
Install the kube-prometheus-stack in a dedicated namespace.
Verify all monitoring pods are running.

Add the Prometheus community Helm repository

Helm uses repositories to find chart packages. Before you can install the monitoring stack, you need to tell Helm where to find the chart.

Add the Prometheus community chart repository and update your local chart index by running these commands:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

What do these commands do?

The first command registers a new chart repository called prometheus-community with Helm. This is where the kube-prometheus-stack chart lives.

The second command refreshes Helm's local index of available charts. This ensures you get the latest version when you install.

You should see a message confirming the repository has been added, followed by an update confirmation.

Seeing a connection error?

Make sure you have an active internet connection. Helm needs to download the chart index from the remote URL.

If you see a message that the repository already exists, that is fine. The helm repo update command still refreshes the index.

help me troubleshoot this Helm repo add error

Create a monitoring namespace and install the stack

A namespace is a way to group related resources inside your cluster. You will create a dedicated monitoring namespace to keep all the monitoring components separate from your application workloads.

Create the namespace and install the full monitoring stack by running these commands:

kubectl create namespace monitoring
helm install prometheus-stack prometheus-community/kube-prometheus-stack --namespace monitoring --set grafana.adminPassword=admin123 --wait --timeout 10m

What does the install command do?

kubectl create namespace monitoring creates an isolated space in your cluster for the monitoring tools.
helm install prometheus-stack gives this installation the release name prometheus-stack. You will use this name to reference the deployment later.
prometheus-community/kube-prometheus-stack tells Helm which chart to install from the repository you just added.
--set grafana.adminPassword=admin123 sets the Grafana login password explicitly so you do not have to retrieve it from a Kubernetes secret.
--wait --timeout 10m tells Helm to wait up to 10 minutes for all pods to become ready before returning.

What is a Helm chart?

A Helm chart is a package that bundles all the Kubernetes manifests needed to deploy an application. Instead of writing dozens of YAML files by hand, a single chart installs everything: deployments, services, config maps, and more.

The kube-prometheus-stack chart deploys Prometheus, Grafana, Alertmanager, Node Exporter, kube-state-metrics, and the Prometheus Operator in one command.

This command takes several minutes

The --wait flag means your terminal will not return a prompt until all pods are ready. This typically takes 3-5 minutes depending on your internet speed (images need to be downloaded).

Do not close or interrupt the terminal while it is running. Wait for the output that includes STATUS: deployed to confirm success.

Seeing a timeout or pod errors?

If the command times out, Docker Desktop may not have enough resources allocated. Open Docker Desktop, go to Settings then Resources, and ensure at least 4 GB of memory is allocated.

If you see an error about the namespace already existing, that means you previously created it. You can skip the kubectl create namespace command and run only the helm install line.

help me troubleshoot this Helm install error

Verify all pods are running

The Helm command deployed many components into your cluster. Now you will confirm that every pod started successfully.

Check the status of all pods in the monitoring namespace by running:

kubectl get pods -n monitoring

What should I see?

You should see approximately 6-8 pods, all with STATUS showing Running. The pods include:

prometheus-stack-grafana (the dashboard UI).
prometheus-stack-kube-prometheus-prometheus (the metrics collector).
alertmanager (handles alert routing).
node-exporter (exposes hardware-level metrics from the node).
kube-state-metrics (exposes Kubernetes object metrics like pod count and status).
prometheus-operator (manages the Prometheus configuration automatically).

Confirm that every pod in the list shows Running in the STATUS column and all containers are ready (e.g. 1/1 or 2/2 in the READY column).

Some pods showing Pending or CrashLoopBackOff?

If pods are stuck in Pending, your cluster may not have enough resources. Verify your Minikube was started with --memory=4096 in the previous step.

If a pod shows CrashLoopBackOff, wait 1-2 minutes and run kubectl get pods -n monitoring again. Some pods restart once during initial setup and then stabilize.

help me debug my monitoring pods that are not starting

Your monitoring stack is fully deployed. Next up, you will access Grafana in your browser and explore the pre-built dashboards that visualize your cluster metrics in real time.

Access Grafana and Explore Dashboards

Your monitoring stack is deployed and every pod is running. But right now all that data is trapped inside the cluster with no way to see it.

Grafana is the visualization layer of your stack. It connects to Prometheus and turns raw metrics into readable dashboards. In this step, you'll open a tunnel to Grafana and explore the pre-built dashboards that ship with the kube-prometheus-stack.

In this step, get ready to:

Port-forward Grafana to your browser.
Explore pre-built Kubernetes resource dashboards.
Read cluster metrics and identify resource usage patterns.

Port-forward Grafana to your browser

Grafana is running inside your Kubernetes cluster, which means your browser can't reach it directly. Port-forwarding creates a tunnel from a port on your Mac to the Grafana service inside the cluster.

Start the port-forward to Grafana by running this command:

kubectl port-forward svc/prometheus-stack-grafana 3000:80 -n monitoring

This command blocks your terminal

The port-forward process keeps running in the foreground. You'll see output like Forwarding from 127.0.0.1:3000 -> 80 and the terminal won't accept new commands.

Leave this terminal open and running for the rest of the step. If you need to run other commands later, open a new terminal window.

Open your browser and navigate to:

http://localhost:3000

Enter admin in the Email or username field.
Enter admin123 in the Password field.
Click Log in.

You should see the Grafana home page with a welcome banner and navigation options in the left sidebar.

Can't reach localhost:3000?

Check that your port-forward terminal is still running and hasn't printed an error. If it exited, run the command again.
Make sure no other application is using port 3000. If something else is bound to that port, stop it first or close the conflicting app.
Verify the Grafana pod is still running with kubectl get pods -n monitoring. The Grafana pod should show Running status.
Help me troubleshoot why I can't access Grafana through port-forward on localhost:3000

Explore the pre-built dashboards

The kube-prometheus-stack ships with dozens of pre-built dashboards covering cluster resources, node health, and pod-level metrics. You don't need to build anything from scratch.

Click Dashboards in the left sidebar.
You'll see a list of folders and dashboards. Look for a folder or search for Kubernetes / Compute Resources / Cluster.
Click on Kubernetes / Compute Resources / Cluster to open it.

You should see a dashboard with panels showing CPU usage, memory usage, and network bandwidth across your entire cluster. The data is live and updating.

What does this dashboard show?

The Cluster dashboard gives you a bird's-eye view of resource consumption broken down by namespace. Each panel shows a different metric (CPU cores, memory bytes, network traffic) aggregated across all pods in that namespace.

Click the back arrow or return to Dashboards in the left sidebar.
Search for and open Kubernetes / Compute Resources / Node (Pods).

This dashboard breaks down resource usage per individual pod running on your node. You can see exactly how much CPU and memory each pod is consuming.

Why two dashboards?

The Cluster dashboard answers "which namespace is using the most resources?" The Node (Pods) dashboard answers "which specific pod is the heaviest?" Together they let you zoom from a high-level overview down to individual pod behavior.

Read the metrics

Dashboards are only useful if you can interpret what they show. Let's practice reading two important concepts that appear throughout these panels.

Go back to the Kubernetes / Compute Resources / Cluster dashboard.
Look at the CPU usage panels. Identify which namespace is consuming the most CPU.

The monitoring namespace should be the largest consumer. This makes sense because that's where all the Prometheus, Grafana, and exporter pods are running.

Look for columns or legends labeled requests and usage (or limits and usage) in the CPU and memory panels.

What are requests vs actual usage?

When a pod is deployed, it can declare how much CPU and memory it needs (its requests) and the maximum it's allowed to use (its limits). The actual usage is what the pod is really consuming right now.

If actual usage is far below requests, the pod is over-provisioned and wasting cluster capacity. If usage is hitting limits, the pod might be getting throttled or killed. This gap between what's requested and what's actually used is one of the most important things to monitor.

You've got a full observability view into your cluster. Next up, you'll deploy a sample application and watch its metrics appear in these dashboards in real time.

Deploy a Sample App and Observe Its Metrics

You've explored the pre-built dashboards and seen how Grafana visualizes cluster-level metrics. Now it's time to see what happens when you deploy your own application into the cluster.

In this step, you'll deploy a simple web server, watch it appear in your monitoring dashboards in real time, and then query Prometheus directly to see the raw metrics behind the graphs.

In this step, get ready to:

Deploy an nginx application with multiple replicas.
Watch the new pods appear in Grafana dashboards.
Query Prometheus directly with PromQL to see raw memory metrics.

Deploy an nginx application

So far, only the monitoring stack itself is running in your cluster. To see how the dashboards track real workloads, you need an application generating metrics.

Create an nginx deployment with 3 replicas by running this command:

kubectl create deployment nginx-demo --image=nginx --replicas=3

What does this command do?

This creates a Deployment resource named nginx-demo in the default namespace.

--image=nginx tells Kubernetes to pull the official nginx container image.
--replicas=3 creates three identical pods running that image, spreading the workload.

Confirm your pods are running by running:

kubectl get pods

You should see three pods with names starting with nginx-demo and a status of Running. It may take 30-60 seconds for all three to reach that state.

Pods stuck in ContainerCreating?

This usually means the nginx image is still being downloaded. Wait a minute and run kubectl get pods again.

If pods show ErrImagePull or ImagePullBackOff, check that Docker Desktop is running and your Mac has internet access.

help me debug why my nginx pods won't start

Watch the deployment appear in Grafana

Your monitoring stack automatically scrapes metrics from every new pod. Within a minute or two, your nginx pods should appear in the dashboards you explored earlier.

Switch to your browser where Grafana is open at http://localhost:3000.
Click Dashboards in the left sidebar.
Open the dashboard called Kubernetes / Compute Resources / Cluster.
Find the namespace filter dropdown at the top of the dashboard.
Select default from the namespace filter.

You should see your three nginx pods and their CPU and memory consumption appearing in the charts. The values will be small since nginx is lightweight when idle.

Why can I see these pods without any extra configuration?

The kube-prometheus-stack deploys components like Node Exporter and kube-state-metrics that automatically discover and scrape every pod in the cluster. You don't need to configure anything per-application for basic resource metrics.

Query Prometheus directly

Dashboards are great for a quick overview, but sometimes you need to run your own custom queries against the raw metric data. PromQL is Prometheus's query language for slicing and filtering metrics.

Open a new terminal window. Your existing terminal is still running the Grafana port-forward, so leave it open.

Why a new terminal?

Port-forward commands block the terminal they run in. Each port-forward needs its own terminal window or tab to keep running.

Forward the Prometheus UI to your browser by running this command in your new terminal:

kubectl port-forward svc/prometheus-stack-kube-prometheus-prometheus 9090:9090 -n monitoring

What does this command do?

This maps port 9090 on your Mac to port 9090 on the Prometheus service running inside your cluster. The Prometheus UI becomes accessible at http://localhost:9090.

Open http://localhost:9090 in your browser.
Find the query input box at the top of the page.
Type the following PromQL query into the box:

container_memory_working_set_bytes{namespace="default"}

Click Execute.

You should see a table of raw memory metric values for each container in the default namespace. Each row corresponds to one of your nginx pods.

What is this query doing?

container_memory_working_set_bytes is a metric that Prometheus collects from every container. It reports the current memory in use (in bytes) that cannot be reclaimed by the operating system.

The {namespace="default"} part is a label filter. It narrows the results to only containers running in the default namespace, which is where your nginx pods live.

Not seeing any results?

Make sure you typed the query exactly, including the curly braces and quotes around default.

If the Prometheus page didn't load, check that the port-forward command is still running in your second terminal. If it exited, run it again.

help me troubleshoot my Prometheus query

You've deployed a real application and watched it flow through the full observability pipeline. Next up, you'll simulate a resource problem and use these same tools to diagnose what went wrong.

Simulate a Problem and Diagnose It

In the previous step, you deployed an application and watched its metrics appear in your monitoring dashboards. Now comes the real test of your observability setup.

You'll simulate a CPU resource problem, then use Grafana dashboards and PromQL queries to diagnose exactly what went wrong. This is the same workflow engineers use to investigate production performance issues.

In this step, get ready to:

Deploy a resource-constrained pod that triggers CPU throttling.
Observe CPU throttling symptoms in a Grafana dashboard.
Diagnose the root cause using PromQL queries in Prometheus.

Create a resource-constrained pod

A pod that requests more CPU than its limit allows will get throttled by Kubernetes. The container keeps running, but its performance degrades because the kernel restricts how much CPU time it can use.

Open a new terminal tab or window. Your other two terminals are busy with port-forwards.
Move to your Desktop by running this command:

cd ~/Desktop

Create a file called stress-deployment.yaml and add the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: stress-demo
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: stress-demo
  template:
    metadata:
      labels:
        app: stress-demo
    spec:
      containers:
      - name: stress
        image: polinux/stress
        command: ["stress"]
        args: ["--cpu", "2", "--timeout", "600"]
        resources:
          requests:
            cpu: "50m"
            memory: "64Mi"
          limits:
            cpu: "100m"
            memory: "128Mi"

What does this manifest do?

This deployment creates a container that tries to use 2 CPU cores worth of stress load. However, the resources.limits.cpu field caps it at 100m (100 millicores, or 10% of one CPU core).

The container wants far more CPU than it is allowed to have. Kubernetes enforces the limit, causing the container to be throttled. This is the exact scenario that causes slow application response times in production.

Save stress-deployment.yaml.
Apply this deployment to your cluster by running:

kubectl apply -f stress-deployment.yaml

Verify the pod is running by running:

kubectl get pods

You should see a pod named stress-demo- followed by a random string, with STATUS showing Running.

Pod not showing as Running?

If the pod shows ContainerCreating, wait 30 seconds and run kubectl get pods again. The image needs to download first.
If the pod shows ErrImagePull, check that your Minikube cluster has internet access. Try minikube ssh -- ping -c 1 google.com to verify connectivity.
Make sure the YAML file is saved with no extra spaces or indentation errors. YAML is sensitive to whitespace.

Help me troubleshoot why my stress-demo pod is not running in Kubernetes. I applied a YAML deployment file and the pod is not in Running status.

✔️ Awesome, I've got everything!

Great. Make sure stress-deployment.yaml is saved and the pod shows Running status.

ⓧ I'd like to double check the full code

apiVersion: apps/v1
kind: Deployment
metadata:
  name: stress-demo
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: stress-demo
  template:
    metadata:
      labels:
        app: stress-demo
    spec:
      containers:
      - name: stress
        image: polinux/stress
        command: ["stress"]
        args: ["--cpu", "2", "--timeout", "600"]
        resources:
          requests:
            cpu: "50m"
            memory: "64Mi"
          limits:
            cpu: "100m"
            memory: "128Mi"

Observe the symptoms in Grafana

The stress container has been running for a few seconds now. Its CPU throttling should already be visible in your Grafana dashboards. Let's go look for the evidence.

Switch to your browser with Grafana open at http://localhost:3000.
Click the hamburger menu icon (three horizontal lines) in the top-left corner.
Click Dashboards in the left navigation menu.
Search for and open the dashboard called Kubernetes / Compute Resources / Pod.
In the namespace dropdown at the top of the dashboard, select default.
In the pod dropdown, select the pod whose name starts with stress-demo.

Look for the CPU Throttling panel. It shows the percentage of time your container was throttled. You should see a high throttling percentage because the container is trying to use far more CPU than its 100m limit allows.

What does CPU throttling look like?

When a container hits its CPU limit, the Linux kernel pauses it for short intervals. The throttling panel shows this as a percentage. A value near 100% means the container is constantly being paused and restarted by the kernel.

In production, high CPU throttling causes slow response times and timeouts. The application appears healthy (no crashes or restarts) but performs poorly. This makes it tricky to diagnose without proper monitoring.

Give it a minute

If you do not see throttling data yet, wait 1-2 minutes for Prometheus to scrape the new metrics. Refresh the dashboard page after waiting.

Reason about the root cause using PromQL

Dashboards give you a visual overview, but sometimes you need to query the raw metrics directly. PromQL is the query language for Prometheus. It lets you write precise questions about your metrics data.

Switch to your browser tab with the Prometheus UI at http://localhost:9090.
Click on the query input box at the top of the page.
Paste the following PromQL query to see the throttling rate:

rate(container_cpu_cfs_throttled_seconds_total{namespace="default"}[5m])

Click Execute.

What does this query show?

This query calculates the per-second rate of CPU throttling over the last 5 minutes for all containers in the default namespace. A non-zero value means the container is being actively throttled by its CPU limit.

The rate() function converts a raw counter (total throttled seconds) into a per-second rate, making it easier to interpret. The [5m] window smooths out short spikes.

You should see results for your stress-demo pod with a non-zero throttling rate. Now let's compare it to the actual CPU usage.

Clear the query box and paste this second query to see actual CPU usage:

rate(container_cpu_usage_seconds_total{namespace="default"}[5m])

Click Execute.

How do these two queries diagnose the problem?

The first query shows how much time the container is being throttled. The second query shows how much CPU the container is actually consuming. When usage is at or near the limit (0.1 cores for a 100m limit) AND throttling is high, you have confirmed the diagnosis: the pod needs a higher CPU limit.

This is how engineers reason about resource issues in production. The metrics tell you whether to increase the limit, optimize the application code, or add more replicas to spread the load.

You should see the stress-demo container using CPU at or near 0.1 (which is 100m, its limit). Combined with the high throttling rate from the first query, you have now diagnosed the root cause: the container is CPU-starved.

Not seeing any query results?

Make sure your Prometheus port-forward is still running in its terminal. If it stopped, restart it with kubectl port-forward svc/prometheus-stack-kube-prometheus-prometheus 9090:9090 -n monitoring.
Check that the stress-demo pod is still running. The stress container has a 600-second timeout, so it will stop after 10 minutes. Reapply the manifest if needed.
Verify the namespace filter is default in your query. A typo in the label filter will return empty results.

Help me troubleshoot why my PromQL query is returning no results in the Prometheus UI. I am querying CPU metrics for pods in the default namespace.

Secret mission

Create a Custom PrometheusRule Alert

You've explored dashboards and queried metrics manually. But in production, engineers don't sit and watch dashboards all day. They set up automated alerts that fire when something goes wrong. Your mission: create a custom alert that fires when any pod uses more than 80% of its CPU limit.

Clean Up Your Resources

This project runs entirely on your local machine, so there are no ongoing cloud costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

A Minikube virtual machine running a Kubernetes cluster (uses ~4GB RAM while running).
Docker Desktop running in the background.
CLI tools installed via Homebrew (minikube, kubectl, helm).
Local YAML files (stress-deployment.yaml).

Keep everything running

No action needed. Choose this if you plan to keep experimenting with your cluster. The Minikube cluster persists across terminal sessions.

Your cluster stays running and accessible via kubectl commands.
Docker Desktop continues running in your menu bar.
Port-forward commands will need to be restarted if you closed those terminal windows.

Pause - I'll come back to this later

Shut down the cluster to free up RAM and CPU, but keep all your data so you can resume later.

Stop the Minikube cluster by running:

minikube stop

This preserves your cluster state. Restart later with minikube start.
Quit Docker Desktop from the menu bar icon to free additional memory.

Delete - I don't want to use this again

Remove all project resources completely and free up disk space.

Delete the Minikube cluster and free disk space (~2-3GB) by running:

minikube delete

Remove the local YAML file by running:

rm stress-deployment.yaml

Uninstall the CLI tools by running:

brew uninstall minikube kubectl helm

Uninstall Docker Desktop by running:

brew uninstall --cask docker

Nice Work!

Congratulations! You've built a professional-grade observability setup from scratch on your own machine.

You've learned how to:

Built a local Kubernetes cluster and deployed a full monitoring stack (Prometheus, Grafana, Alertmanager) with a single Helm command.
Explored pre-built Grafana dashboards showing real-time CPU, memory, and pod metrics across your cluster.
Simulated a resource problem and diagnosed it using PromQL queries and dashboard visualizations to reason about root causes.
Created a custom alerting rule using PrometheusRule CRDs that fires when pod CPU usage exceeds a threshold.

Ready to quiz yourself?