Monitor an AKS Cluster with Prometheus

Set up Prometheus and Grafana to monitor a live Kubernetes cluster on Azure.

Introduction

30 Second Summary

When an application running in the cloud slows down or crashes, teams need instant visibility into what went wrong. Monitoring tools like Prometheus and Grafana give you that real-time insight into your infrastructure's health.

In this project, you will set up a complete monitoring solution for a Kubernetes cluster in Azure. You'll collect metrics automatically, visualize them on dashboards, and configure alerts that fire when something goes wrong.

What You'll Build

A production-grade monitoring stack using Azure Monitor managed service for Prometheus and Azure Managed Grafana to observe a live AKS cluster.

By the end of this project, you'll have:

  • A running AKS cluster with Azure Monitor managed Prometheus automatically collecting Kubernetes metrics.
  • An Azure Managed Grafana workspace with prebuilt Kubernetes dashboards visualizing real-time cluster health.
  • A custom Grafana dashboard with PromQL queries and a configured alert rule monitoring pod memory usage.
  • Secret Mission: Deploy a load generator to stress-test your cluster and watch metrics spike and alerts fire in real time.

Are there any prerequisites?

You'll need an Azure subscription (pay-as-you-go works), Azure CLI installed, and kubectl installed. The project walks you through creating all cloud resources from scratch.

Create the Azure Infrastructure

Before you can monitor anything, you need something to monitor. A monitoring stack has three layers: the thing being observed (your cluster), the metrics backend that stores data (Azure Monitor workspace), and the visualization layer that makes sense of it all (Grafana).

In this step, you will create all three layers as Azure resources in a single resource group. By the end, your cluster will already be shipping Prometheus metrics into Azure Monitor with zero additional configuration.

In this step, get ready to:
  • Create a resource group and Azure Monitor workspace.
  • Create an Azure Managed Grafana workspace.
  • Create an AKS cluster with Prometheus metrics enabled.
Sign in to Azure and create the resource group

Every Azure resource lives inside a resource group. You will create one resource group to hold all the monitoring infrastructure for easy management and cleanup.

  • Sign in to your Azure account by running this command:
az login

What does this command do?

This opens your browser so you can authenticate with your Azure subscription. Once you sign in, the CLI stores a token locally so future commands run against your account.

  • You may see a browser window open asking you to sign in. Select your Azure account and close the browser tab once authentication succeeds.
  • Back in your terminal, you should see a JSON list of your subscriptions confirming the sign-in worked.

Browser didn't open?

If you are on a remote machine or the browser does not open, run az login --use-device-code instead. It gives you a URL and a code to enter manually.

help me sign in to Azure CLI

  • Create a resource group called monitoring-rg by running this command:
az group create --name monitoring-rg --location eastus

What does this command do?

This creates a container called monitoring-rg in the eastus region. All resources you create in this project will live inside this group.

  • You should see JSON output with "provisioningState": "Succeeded" confirming the resource group exists.
  • Create an Azure Monitor workspace by running this command:
az monitor account create --name prometheus-workspace --resource-group monitoring-rg --location eastus

What is an Azure Monitor workspace?

An Azure Monitor workspace is a dedicated store for Prometheus metrics. When you enable metrics collection on your cluster later, this is where all the time-series data lands.

It is separate from a Log Analytics workspace. Think of it as a purpose-built database optimized for Prometheus-style metrics.

  • You should see JSON output confirming the workspace was created with "provisioningState": "Succeeded".
  • Note the resource ID from the "id" field in the output. You will need this in a later command. It looks like /subscriptions/.../providers/Microsoft.Monitor/accounts/prometheus-workspace.
  • If you missed the ID, retrieve it by running this command:
az monitor account show --name prometheus-workspace --resource-group monitoring-rg --query id --output tsv
  • Copy the resource ID and save it somewhere handy (a text file or a sticky note).

Seeing a 'not found' error?

Make sure the --name and --resource-group values match exactly what you used in the create command. Names are case-insensitive but must be spelled correctly.

help me retrieve my Azure Monitor workspace resource ID

Create an Azure Managed Grafana workspace

Azure Managed Grafana gives you a fully hosted Grafana instance. It handles authentication, scaling, and upgrades so you can focus on building dashboards.

  • Create a Grafana workspace by running this command:
az grafana create --name [[GRAFANA_NAME="a globally unique name e.g. monitoring-grafana-jane"]] --resource-group monitoring-rg

Why does the name need to be unique?

Your Grafana workspace gets a public URL ending in .grafana.azure.com. Because it is a DNS name, it must be globally unique across all Azure customers. Include your name or initials to avoid conflicts.

Extension auto-install

The az grafana create command automatically installs the amg CLI extension if it is not already present. If you see a prompt asking to install an extension, type y and press Enter.

  • This command takes 2-3 minutes to complete. Wait for the JSON output to appear.
  • In the output, look for the "endpoint" field. It contains your Grafana URL (e.g. https://your-name.grafana.azure.com). Save this URL for later.
  • Also note the "id" field from the output. This is the Grafana resource ID you will pass to the AKS create command.
  • If you need to retrieve the Grafana resource ID later, run this command:
az grafana show --name [[GRAFANA_NAME="a globally unique name e.g. monitoring-grafana-jane"]] --resource-group monitoring-rg --query id --output tsv

Command failed with 'extension not found'?

If the auto-install did not trigger, manually install the extension by running:

az extension add --name amg

Then retry the az grafana create command.

help me install the amg extension for Azure Managed Grafana

Create an AKS cluster with Prometheus metrics enabled

Now for the main event. You will create an AKS cluster and link it to both the Azure Monitor workspace and the Grafana instance in a single command. This tells Azure to deploy the metrics collection agent automatically.

  • Create the AKS cluster by running this command (replace the two resource ID placeholders with the IDs you saved earlier):
az aks create \
  --resource-group monitoring-rg \
  --name monitoring-cluster \
  --node-count 1 \
  --node-vm-size Standard_DS2_v2 \
  --enable-azure-monitor-metrics \
  --azure-monitor-workspace-resource-id [[MONITOR_WORKSPACE_ID="your Azure Monitor workspace resource ID"]] \
  --grafana-resource-id [[GRAFANA_ID="your Grafana workspace resource ID"]] \
  --generate-ssh-keys

What do these flags do?

  • --enable-azure-monitor-metrics deploys a DaemonSet called ama-metrics onto every node. It automatically scrapes kubelet, cAdvisor, kube-state-metrics, and node exporter.
  • --azure-monitor-workspace-resource-id tells the agent where to ship the collected metrics.
  • --grafana-resource-id links Grafana to the Azure Monitor workspace and deploys prebuilt Kubernetes dashboards automatically.
  • --node-vm-size Standard_DS2_v2 picks a 2 vCPU, 7 GB RAM machine. This is enough to run the metrics agent and a sample workload while keeping costs low.
  • --generate-ssh-keys creates SSH keys if you do not already have them. This avoids a prompt about missing keys.

Why Azure Managed Prometheus instead of self-hosted?

Self-hosted Prometheus requires you to manage storage, high availability, and scaling yourself. The Azure managed service handles all of that and still uses standard PromQL for queries.

You learn the same query language and dashboard techniques, but skip the operational overhead that production teams automate away anyway.

  • This command takes 3-5 minutes to complete. Wait for the full JSON output to appear.
  • Look for "provisioningState": "Succeeded" in the output to confirm your cluster is ready.

Cluster creation failed?

Check that you pasted the full resource IDs (starting with /subscriptions/) without any extra spaces or line breaks.

If you see a quota error, your subscription may not have enough cores in the eastus region. Try a different region by adding --location westus2 (and recreate the resource group and workspaces in that region too).

help me troubleshoot my AKS cluster creation failure

Your monitoring foundation is in place. You have an AKS cluster that is already shipping Prometheus metrics to your Azure Monitor workspace, with Grafana ready to visualize them. Next up, you will connect to the cluster and see those metrics flowing in real time.

Verify Metrics Collection and Explore Prebuilt Dashboards

Your Azure infrastructure is in place. The AKS cluster, Azure Monitor workspace, and Grafana workspace are all connected.

But how do you know metrics are actually flowing? In this step, you will connect to your cluster, confirm the metrics agent is running, and explore the dashboards that Grafana provides out of the box.

In this step, get ready to:
  • Connect to the AKS cluster and verify the Prometheus metrics agent is running.
  • Open Grafana and explore the prebuilt Kubernetes dashboards.
  • Run a PromQL query in Grafana Explore to confirm live metrics are flowing.
Connect to the AKS cluster and verify the metrics agent

When you enabled Prometheus metrics on your cluster, Azure deployed a DaemonSet called ama-metrics onto every node. This agent automatically scrapes Prometheus metrics from kubelet, cAdvisor, kube-state-metrics, and node exporter.

Before you can interact with the cluster, you need to download its credentials so kubectl knows where to connect.

  • Download the cluster credentials by running this command:
az aks get-credentials --resource-group monitoring-rg --name [[CLUSTER_NAME="your AKS cluster name"]]

What does this command do?

This command fetches the kubeconfig for your AKS cluster and merges it into your local ~/.kube/config file. After running it, kubectl commands will target your cluster automatically.

  • Verify the ama-metrics DaemonSet is running by executing this command:
kubectl get ds -n kube-system

What should I see?

You should see a DaemonSet named ama-metrics in the output with at least 1 pod in the DESIRED and READY columns. This confirms the Prometheus metrics agent is running on your node and scraping cluster metrics.

  • Confirm the READY column shows 1 (matching the number of nodes in your cluster).

Don't see ama-metrics in the output?

  • Metrics can take 2-3 minutes to fully deploy after cluster creation. Wait a moment and run the command again.
  • Make sure you included the --enable-azure-monitor-metrics flag when creating your AKS cluster in Step 1.
  • Verify your kubectl context is pointing at the correct cluster by running kubectl config current-context.

Help me troubleshoot why ama-metrics is not showing up in my kube-system namespace.

Open Grafana and explore the prebuilt Kubernetes dashboards

Because you linked the Grafana workspace to your Azure Monitor workspace during cluster creation, Azure automatically deployed a set of prebuilt Prometheus dashboards. These give you immediate visibility into cluster health without any manual setup.

  • Open your Azure Managed Grafana endpoint URL in a new browser tab. The URL ends with .grafana.azure.com.
  • Sign in with the same Microsoft account you use for the Azure portal.
  • In the left-hand menu, click Dashboards.
  • Expand the Managed Prometheus folder to see the list of prebuilt dashboards.

What prebuilt dashboards are available?

You should see dashboards covering multiple perspectives of your Kubernetes cluster:

  • Cluster-level compute resources (overall CPU and memory across all nodes).
  • Namespace-level views (resource usage broken down by namespace).
  • Pod-level views (individual pod CPU, memory, and network).
  • Node-level views (per-node hardware utilization).
  • Open any dashboard that interests you (for example, the cluster compute resources view).
  • Explore the panels showing CPU usage, memory consumption, and network traffic for your cluster.

Give metrics a few minutes

If the dashboards show "No data" initially, wait 2-3 minutes for metrics to begin flowing from the cluster to the Azure Monitor workspace. Refresh the page after waiting.

Run a PromQL query in Grafana Explore

Prebuilt dashboards are great for a quick overview, but real monitoring power comes from writing your own queries. PromQL (Prometheus Query Language) lets you slice and filter metrics any way you want. The Explore view in Grafana gives you a scratchpad to test queries before adding them to dashboards.

  • In the Grafana left-hand menu, click Explore.
  • At the top of the Explore page, select your Prometheus data source from the dropdown (it is labelled with your Azure Monitor workspace name).
  • In the query editor, type the following PromQL query:
container_cpu_usage_seconds_total
  • Click Run query (or press Shift+Enter).

What does this query show?

The container_cpu_usage_seconds_total metric reports the cumulative CPU time consumed by each container in your cluster. If you see data points appearing in the graph or table view, your Prometheus pipeline is working end-to-end.

This metric flows from cAdvisor on each node, through the ama-metrics agent, into the Azure Monitor workspace, and finally into Grafana for visualization.

  • Confirm you see a graph or table with data points. This proves live metrics are flowing from your cluster to Grafana.

Seeing no data in Explore?

  • Make sure you selected the Prometheus data source (not Azure Monitor) from the dropdown at the top of the Explore page.
  • Set the time range to "Last 15 minutes" using the time picker in the top right corner of the page.
  • If your cluster was just created, wait 2-3 more minutes for metrics ingestion to begin and try again.

Help me troubleshoot why my PromQL query returns no data in Grafana Explore.

Your monitoring pipeline is confirmed working. Metrics are flowing from the cluster into Azure Monitor and visualizing in Grafana. Next up, you will deploy a sample application to generate real workload metrics you can track.

Deploy a Sample Application

Your monitoring infrastructure is collecting cluster-level metrics, but an empty cluster only tells half the story. To see how Prometheus tracks real workloads, you need an application generating CPU, memory, and network activity.

In this step, you will deploy an NGINX web server with multiple replicas, expose it with a LoadBalancer service, and send continuous traffic to it. Then you will confirm that workload-specific metrics appear in Grafana Explore.

In this step, get ready to:
  • Deploy an NGINX workload with multiple replicas to your AKS cluster.
  • Expose the deployment with a LoadBalancer service and generate traffic.
  • Query deployment-specific metrics in Grafana Explore using PromQL.
Deploy an NGINX workload and expose it with a LoadBalancer

A Kubernetes Deployment manages a set of identical pods. By creating multiple replicas, you give Prometheus several containers to scrape metrics from.

  • Create the NGINX deployment with 3 replicas by running this command:
kubectl create deployment nginx-demo --image=nginx --replicas=3

What does this command do?

This tells Kubernetes to create a Deployment named nginx-demo using the official NGINX container image.

The --replicas=3 flag launches three identical pods, giving you multiple containers that generate metrics simultaneously.

  • Verify the pods are running by running:
kubectl get pods -l app=nginx-demo

You should see three pods with a Running status. It may take 30-60 seconds for all pods to start.

Pods stuck in Pending or ContainerCreating?

Wait 1-2 minutes for the node to pull the NGINX image. Run kubectl describe pod followed by the pod name to check events for errors.

If a pod stays in Pending, your single-node cluster may be under memory pressure. Help me troubleshoot pods stuck in Pending on my AKS cluster

Now you need to make this deployment accessible over the network. A Kubernetes Service of type LoadBalancer provisions an external IP through Azure's cloud provider.

  • Expose the deployment with a LoadBalancer service by running:
kubectl expose deployment nginx-demo --type=LoadBalancer --port=80

What does this command do?

This creates a Service resource that routes external traffic on port 80 to the NGINX pods. Because you specified --type=LoadBalancer, Azure automatically provisions a public IP address and maps it to this service.

  • Check for the external IP by running:
kubectl get service nginx-demo

Look for an IP address in the EXTERNAL-IP column. If you see <pending>, wait 1-2 minutes and run the command again.

External IP stays as pending?

Azure typically provisions the IP within 2 minutes. If it remains pending longer, check that your AKS cluster has the proper network permissions by running kubectl describe service nginx-demo and reviewing the Events section.

Help me debug why my AKS LoadBalancer service external IP is stuck as pending

  • Once you see an IP address in the EXTERNAL-IP column, note it down for the next substep.
Generate traffic to the application

Prometheus needs active network and memory activity to produce interesting metrics. You will launch a lightweight pod inside the cluster that sends continuous HTTP requests to your NGINX service.

  • Start a traffic generator pod by running this command:
kubectl run traffic-gen --image=busybox --restart=Never -- /bin/sh -c "while true; do wget -q -O- http://[[NGINX_IP=\"your NGINX service external IP\"]]; sleep 1; done"

What does this command do?

This creates a single pod named traffic-gen running the BusyBox image. The --restart=Never flag tells Kubernetes not to restart it if it exits.

The command after -- runs an infinite loop that fetches the NGINX homepage every second using wget. This generates steady network and CPU metrics that Prometheus can scrape.

  • Verify the traffic generator is running:
kubectl get pods traffic-gen

You should see traffic-gen with a Running status.

Traffic-gen pod shows CrashLoopBackOff?

This usually means the external IP is unreachable or was mistyped. Run kubectl logs traffic-gen to see the wget error. Delete the pod with kubectl delete pod traffic-gen and re-run the command with the correct IP.

Help me fix my traffic-gen busybox pod that keeps crashing

Query deployment metrics in Grafana Explore

With traffic flowing, Prometheus is now scraping container-level metrics for your NGINX pods. You can query these metrics directly using PromQL in Grafana's Explore view.

  • Switch back to the Grafana browser tab from earlier.
  • Click the menu icon (three horizontal lines) in the top-left corner.
  • Select Explore from the left sidebar.
  • Make sure the data source dropdown at the top shows your Prometheus data source (it should be pre-selected).
  • Enter the following PromQL query in the query field:
container_memory_working_set_bytes{namespace="default"}

What does this query do?

This PromQL query asks Prometheus for the current memory usage (working set) of every container in the default namespace. Since your nginx-demo pods and traffic-gen pod all run in the default namespace, you will see a time series for each container.

  • Click Run query (or press Shift+Enter).

You should see multiple time series lines in the graph, one for each container in the default namespace. The legend shows pod names like nginx-demo-xxxxx confirming your deployment's metrics are flowing.

Wait for metrics to appear

If the query returns no results, wait 2-3 minutes. Prometheus scrapes metrics on an interval, and new pods need time to be discovered. Re-run the query after waiting.

Still seeing no data for your namespace?

Confirm your pods are actually running in the default namespace by switching back to your terminal and running kubectl get pods -n default. Also verify the Prometheus data source is selected (not a different source like Azure Monitor).

Help me troubleshoot why container_memory_working_set_bytes shows no data in Grafana for my default namespace

Your sample application is deployed, traffic is flowing, and you have confirmed that workload-specific metrics appear in Grafana. Next up, you will build a custom dashboard with dedicated panels for CPU, memory, and network metrics.

Build a Custom Grafana Dashboard

In the previous step, you deployed a sample workload and confirmed that Prometheus is collecting metrics from your application. You can now see real data flowing in the Explore view.

Prebuilt dashboards are useful for a quick overview, but production teams need dashboards tailored to their specific applications. In this step, you will build a custom Grafana dashboard with three panels focused on your deployed workload: CPU usage rate, memory consumption, and network throughput.

In this step, get ready to:
  • Create a new dashboard with a CPU usage rate panel using PromQL.
  • Add a memory usage panel with proper unit formatting.
  • Add a network I/O panel and save the dashboard.
Create a CPU usage rate panel

A PromQL rate query calculates how fast a counter is increasing per second over a time window. This is the standard way to visualize CPU consumption for containers.

  • In your Azure Managed Grafana browser tab, click Dashboards in the left sidebar menu.
  • Click New and select New Dashboard.
  • Click + Add visualization.
  • In the data source dialog, select your Prometheus data source.

The panel editor opens with a query area at the bottom and visualization options on the right.

  • In the query editor, click the Code tab on the Builder/Code toggle to switch to raw PromQL mode.
  • In the query text field, paste the following PromQL query:
rate(container_cpu_usage_seconds_total{namespace="default"}[5m])
  • Click Run query (or press Shift + Enter).

You should see a time series graph appear with multiple lines representing CPU usage rate per container in the default namespace.

  • In the panel editor on the right side, under Panel options, enter CPU Usage Rate as the panel title.
  • Click Apply in the top-right corner to save the panel and return to the dashboard view.

What does this query do?

The rate() function calculates the per-second rate of increase for the container_cpu_usage_seconds_total counter over a 5-minute window.

The {namespace="default"} label filter ensures you only see containers running in your application's namespace. Each line on the graph represents one container's CPU consumption rate in CPU-seconds per second.

Not seeing any data?

Make sure you selected the correct Prometheus data source (not Azure Monitor). Check that your traffic generator pod is still running with kubectl get pods to confirm the nginx-demo pods are active.

If the graph is empty, try expanding the time range selector in the top-right corner to Last 15 minutes or Last 30 minutes. Metrics may take 2-3 minutes to appear after deployment.

Help me troubleshoot why my Grafana panel shows no data for my Prometheus CPU query.

Add a memory usage panel

Memory metrics show the current working set size of each container. Unlike CPU, this is a gauge (current value) rather than a counter, so you query it directly without the rate() function.

  • Back on the dashboard view, click + Add visualization to add a second panel.
  • Select the Prometheus data source again.
  • Switch to Code mode in the query editor.
  • Paste the following PromQL query:
container_memory_working_set_bytes{namespace="default"}
  • Click Run query.

You should see lines representing the memory consumption of each container in bytes.

  • Under Panel options on the right, enter Memory Usage as the title.
  • In the panel editor sidebar, scroll down to Standard options.
  • In the Unit dropdown, search for and select bytes (under the Data category).
  • Click Apply to save the panel.

Why set the unit to bytes?

Without a unit, Grafana displays raw numbers like 52428800 on the Y-axis. Setting the unit to bytes tells Grafana to auto-format values into human-readable units like MB or GB. This makes the panel much easier to read at a glance.

Panel showing unexpected values?

If you see very small or zero values, confirm you typed the metric name exactly as container_memory_working_set_bytes with underscores, not hyphens.

Help me fix my Grafana memory panel that shows unexpected values.

Add a network I/O panel and save the dashboard

Network receive bytes is another counter metric. Wrapping it in rate() gives you the bytes-per-second flowing into each container.

  • Click + Add visualization to create the third panel.
  • Select the Prometheus data source.
  • Switch to Code mode.
  • Paste this PromQL query:
rate(container_network_receive_bytes_total{namespace="default"}[5m])
  • Click Run query.
  • Under Panel options, enter Network I/O as the title.
  • Click Apply to save the panel.

You should now see three panels on your dashboard showing CPU rate, memory usage, and network throughput.

  • Click the Save dashboard icon (disk icon) at the top of the dashboard.
  • Enter AKS App Monitoring in the dashboard name field.
  • Click Save.

What does network receive rate tell you?

The container_network_receive_bytes_total counter tracks all incoming bytes to each container's network interface. Applying rate() over 5 minutes smooths out spikes and gives you a steady view of inbound traffic.

This is helpful for spotting traffic surges, identifying which pods receive the most requests, and correlating network load with CPU or memory consumption.

Dashboard not saving?

If you see a permissions error, make sure your Azure account has at least the Grafana Editor role on the Azure Managed Grafana resource. Your account was assigned this role automatically during workspace creation, but verify if you used a different account to sign in.

Help me resolve a permissions error when saving my Grafana dashboard.

Your custom dashboard is live and tracking your application in real time. Next up, you will configure a Prometheus alert rule that automatically fires when pod memory exceeds a threshold.

Configure a Prometheus Alert Rule

Your custom dashboard shows live metrics, but dashboards only help if you're watching them. In a production environment, you need automated alerts that notify you when something goes wrong.

In this step, you'll create a Prometheus rule group in Azure Monitor that continuously evaluates your cluster's metrics and fires an alert when pod memory usage exceeds a threshold.

In this step, get ready to:
  • Create a Prometheus rule group with a recording rule and an alert rule in the Azure portal.
  • Configure the alert to fire when pod memory exceeds 80% of requested memory.
  • Verify the alert rule is active in the Azure portal.
Navigate to the Azure Monitor workspace and create a rule group

A Prometheus rule group is a collection of rules that are evaluated together against metrics in your Azure Monitor workspace. You can include both recording rules (which precompute expensive queries) and alert rules (which fire notifications) in a single group.

  • Open the Azure portal at portal.azure.com in your browser.
  • In the portal search bar at the top, type Monitor and select Monitor from the results.
  • In the left menu, click Alerts.
  • Click Prometheus rule groups to see all rule groups in your subscriptions.
  • Click + Create to start creating a new rule group.

What is a Prometheus rule group?

A rule group bundles related rules so they're evaluated at the same interval. Every rule in the group queries metrics from the same Azure Monitor workspace.

Recording rules precompute expensive PromQL expressions and store the result as new metrics. Alert rules fire Azure Monitor alerts when a PromQL condition is true for a specified duration.

On the Scope tab:

  • Select your Azure Monitor workspace from the Azure Monitor workspace dropdown. This is the workspace you created in monitoring-rg.
  • Leave the Cluster setting on All clusters in the workspace.
  • Click Next to move to the Details tab.

On the Details tab:

  • Select your subscription from the Subscription dropdown.
  • Select monitoring-rg from the Resource group dropdown.
  • Enter aks-memory-rules in the Name field.
  • Enter Recording and alert rules for AKS memory monitoring in the Description field.
  • Leave Evaluate every at the default of 1 minute.
  • Make sure Enabled is toggled on.

Now you'll add a recording rule to the group. Recording rules precompute a PromQL expression so that querying the result later is fast.

  • Click Add recording rule.
  • Enter pod:memory_utilization:ratio in the Name field.
  • Enter the following PromQL expression in the Expression field:
container_memory_working_set_bytes{namespace="default"} / on(pod,namespace) kube_pod_container_resource_requests{resource="memory",namespace="default"}
  • Make sure Enabled is toggled on.
  • Click Run Query to preview the results in the chart. You should see time series data appear if your pods have memory requests configured.

What does this recording rule do?

This expression divides the actual memory being used by a container (container_memory_working_set_bytes) by the memory that was requested for that container (kube_pod_container_resource_requests).

The result is a ratio. A value of 0.8 means the pod is using 80% of its requested memory. By precomputing this ratio, alert rules can evaluate faster without recalculating it every time.

Add the alert rule for memory threshold

Now you'll add an alert rule to the same group that fires when any pod in the default namespace exceeds 80% of its requested memory.

  • Click Add alert rule.
  • Enter PodMemoryOver80Percent in the Name field.
  • Set Severity to 3 (Informational).
  • Enter the following PromQL expression in the Expression field:
container_memory_working_set_bytes / on(pod,namespace) kube_pod_container_resource_requests{resource="memory"} > 0.8
  • Set Wait for to PT5M. This means the condition must be true for 5 consecutive minutes before the alert fires.
  • In the Annotations section, add a key description with value Pod {{ $labels.pod }} in namespace {{ $labels.namespace }} is using more than 80% of its requested memory..
  • Make sure Enabled is toggled on.
  • Leave Automatically resolve alerts checked so alerts resolve when memory drops below 80%.

What does this alert expression mean?

The expression container_memory_working_set_bytes / on(pod,namespace) kube_pod_container_resource_requests{resource="memory"} > 0.8 compares actual memory usage to the requested amount.

The on(pod,namespace) clause tells PromQL to match the two metrics by their pod and namespace labels. The > 0.8 threshold means the alert only fires when usage exceeds 80%.

The Wait for period of 5 minutes prevents brief memory spikes from triggering false alarms.

Now finish creating the rule group:

  • Click Next to move to the Tags tab. You can skip adding tags.
  • Click Next to move to the Review + create tab.
  • Review your settings. You should see 1 recording rule and 1 alert rule listed.
  • Click Create.

Wait for the deployment to complete. This typically takes under a minute.

Getting a validation error?

Make sure the Expression field doesn't have any trailing spaces or line breaks. Copy the PromQL query as a single line.

If you see an error about the workspace, confirm you selected the correct Azure Monitor workspace on the Scope tab.

Help me fix this Prometheus rule group validation error.

Verify the alert rule is active

After creating the rule group, you need to confirm that Azure Monitor is actively evaluating the alert.

  • In the Azure portal, click Monitor in the left navigation menu.
  • Click Alerts.
  • Click Prometheus rule groups.
  • Find your aks-memory-rules rule group in the list and click on it.
  • Expand the rule group to see the rules. Your PodMemoryOver80Percent alert rule should show as enabled.

What should I see?

Your rule group should appear in the list with status showing as enabled. When you expand it, both the pod:memory_utilization:ratio recording rule and the PodMemoryOver80Percent alert rule should be listed as enabled.

If your traffic generator is pushing pods close to their memory limits, you may already see the alert in a "Pending" or "Firing" state under Monitor > Alerts. If not, the alert remains inactive until the threshold is breached.

  • Click Alerts in the left menu of Monitor to see fired or pending alerts.
  • If the Monitor service filter does not appear, click Add Filter and add it.
  • Set the Monitor service filter to Prometheus to see only Prometheus alerts.

Alerts need time to evaluate

The alert rule evaluates every 1 minute and requires the condition to be true for 5 consecutive minutes (the Wait for period) before firing. If your pods aren't close to their memory limits yet, you'll see the alert trigger during the Secret Mission when you run a stress test.

Don't see your rule group?

Make sure you're looking in the Prometheus rule groups view, not the standard alert rules view. Prometheus rules are managed separately from other Azure Monitor alert types.

Check that the subscription filter at the top of the page includes the subscription where monitoring-rg lives.

Help me find my Prometheus rule group in the Azure portal.

Your monitoring stack is now fully armed. You have live dashboards showing real-time metrics and an automated alert that will notify you the moment a pod starts consuming too much memory. In the Secret Mission, you'll push your cluster to the limit and watch this alert fire in real time.

Secret mission

Stress Test Your Monitoring

Your monitoring stack is live and your alert rule is waiting. But will it actually fire when things go wrong? Deploy a memory-hungry stress test pod and find out whether your dashboards and alerts respond to real pressure.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. Your AKS cluster and Azure Managed Grafana workspace accrue charges while running.

Cost warning

Your AKS node costs approximately $0.10/hr and your Grafana workspace costs approximately $0.043/hr per Standard Unit. If you leave everything running, you can expect charges of approximately $3-5/day combined.

If you are done with this project, deleting the resource group is the fastest way to stop all charges immediately.

Resources you used:

  • Resource group monitoring-rg containing all Azure resources.
  • AKS cluster (1 node Standard_DS2_v2) with Prometheus metrics collection enabled.
  • Azure Monitor workspace storing Prometheus metrics.
  • Azure Managed Grafana workspace (Standard tier) with prebuilt and custom dashboards.
  • Prometheus rule group with alert rule.
  • Local kubectl credentials configured for the AKS cluster.

Keep everything running

No action needed. Choose this if you are still actively building or plan to continue experimenting with your monitoring stack right away.

  • Your AKS cluster, Grafana workspace, and Azure Monitor workspace will continue running and accruing charges.
  • Your custom Grafana dashboard and Prometheus alert rule remain active.
  • Your local kubectl credentials remain configured in ~/.kube/config.

Pause - I'll come back to this later

Stop your AKS cluster to eliminate node compute charges while keeping your configuration intact. You can restart it later.

  • Stop the AKS cluster by running:
az aks stop --name [[CLUSTER_NAME="your AKS cluster name"]] --resource-group monitoring-rg

What does stopping do?

Stopping the cluster halts both the control plane and agent nodes. You will not be charged for compute while the cluster is stopped. All your deployments, services, and configurations are preserved.

Your Azure Managed Grafana workspace will continue to incur a small charge while stopped. The Azure Monitor workspace retains your stored metrics at no additional ongoing cost beyond ingestion.

  • When you are ready to resume, start the cluster again by running:
az aks start --name [[CLUSTER_NAME="your AKS cluster name"]] --resource-group monitoring-rg

The cluster restarts with its previous state and node count.

Delete - I don't want to use this again

Remove all project resources and stop all charges immediately. This is irreversible.

  • Delete the entire resource group (this removes your AKS cluster, Grafana workspace, Azure Monitor workspace, Prometheus rule group, and all associated resources) by running:
az group delete --name monitoring-rg --yes --no-wait

What does this command do?

The --yes flag skips the confirmation prompt. The --no-wait flag returns control of your terminal immediately while Azure deletes resources in the background.

Full deletion takes a few minutes. You can confirm by running az group list after a few minutes to verify monitoring-rg no longer appears.

  • Remove the local kubectl credentials for the deleted cluster by running:
kubectl config delete-context [[CONTEXT_NAME="your cluster context name"]]

Find your context name

Run kubectl config get-contexts to list all configured contexts. The context for your AKS cluster typically matches the cluster name.

That's a Wrap!

That's a Wrap!

Nice work! You just built a production-grade monitoring stack on Azure using managed Prometheus and Grafana.

You've learned how to:

  • Deploy an AKS cluster with Azure Monitor managed Prometheus automatically collecting Kubernetes metrics from kubelet, cAdvisor, kube-state-metrics, and node exporter.
  • Connect Azure Managed Grafana and build a custom dashboard with PromQL queries tracking CPU usage, memory consumption, and network I/O.
  • Configure a Prometheus alert rule that fires when pod memory exceeds a defined threshold.
  • Secret Mission: Stress-tested your cluster with a memory-intensive pod and watched metrics spike and alerts fire in real time.

Ready to quiz yourself?