Respond to an AKS CPU Incident

Create an AKS CPU incident, investigate telemetry, and recover with autoscaling.

Introduction

30 Second Summary

A busy service can slow down long before it stops working. Without a clear trail, teams can lose time figuring out where the strain started.

In this project, you will trigger a controlled CPU incident on your existing Azure Kubernetes Service cluster. You will use Windows Subsystem for Linux plus Azure Monitor to detect the pressure before a Horizontal Pod Autoscaler adds capacity automatically.

What You'll Build

You will watch a terminal load test push one replica into CPU pressure before a dashboard trace follows the workload through automatic recovery.

By the end of this project, you'll have:

A repeatable controlled CPU incident you can start from WSL, with live pod usage rising in the terminal.
A visible dashboard evidence trail that shows the workload pressure building during the incident.
An automatic recovery demonstration where the workload expands beyond one replica under load before returning toward its quiet baseline.
Secret Mission: A deliberate two-replica ceiling that you diagnose through the ScalingLimited condition before restoring the safer ten-replica policy.

Are there any prerequisites?

This project assumes an active Azure subscription with an existing managed-identity AKS cluster that has spare capacity in a non-production sandbox. Your Azure account needs at least Contributor access for onboarding plus permission to view monitoring data.

Before We Start

This project deliberately creates CPU pressure on an existing Azure Kubernetes Service cluster. Before the hands-on work begins, choose a non-production cluster or approved sandbox where that controlled incident is safe.

Set Up WSL and Connect to AKS

You have chosen a safe Azure Kubernetes Service (AKS) cluster for the controlled incident. The evidence you collect later depends on every command reaching that exact cluster.

This step gives you one consistent Linux shell through Windows Subsystem for Linux (WSL). You will verify Azure CLI before configuring kubectl access.

In this step, get ready to:

Verify that Ubuntu runs through WSL 2.
Check Azure CLI 2.90.0 or later inside Ubuntu.
Connect kubectl to the intended AKS cluster.

Open Ubuntu in WSL 2

WSL runs Linux distributions directly on Windows. Checking its version first confirms that Ubuntu uses the Linux environment required for every later command.

Administrator mode gives Windows permission to install or convert WSL when needed. All AKS work remains inside Ubuntu after this check.

Press the Windows key to open Windows search.
Type Windows Terminal.
Right-click the Windows Terminal result.
Select Run as administrator.
Check your installed WSL distributions by running this command:

wsl.exe --list --verbose

What does this command show?

This lists every installed Linux distribution. The VERSION column shows whether Ubuntu uses WSL 1 or WSL 2.

✔️ Ubuntu shows version 2

Your Ubuntu distribution already uses WSL 2. Continue below to open its Bash shell.

ⓧ Ubuntu shows version 1

Your Ubuntu distribution needs conversion to WSL 2. Use the distribution name shown in the command output.

Store the listed Ubuntu name as your Ubuntu distribution name.
Convert that distribution to WSL 2 by running this command:

wsl.exe --set-version [[WSL_DISTRO="your Ubuntu distribution name"]] 2

What does this conversion do?

This changes the selected Ubuntu distribution to the WSL 2 architecture. Let the conversion finish before reopening Ubuntu.

Confirm the conversion by running the version check again:

wsl.exe --list --verbose

What confirms the conversion?

The Ubuntu row should now show 2 in the VERSION column. That confirms the distribution uses WSL 2.

ⓧ Ubuntu is not listed

WSL needs its one-time Windows bootstrap. This command enables WSL while installing Ubuntu as the default Linux distribution.

Install WSL from the elevated Windows Terminal by running this command:

wsl --install

What does this installation do?

This enables the required Windows features. It installs Ubuntu with WSL 2 as the default architecture.

Restart Windows after the installation finishes.
Return to this project after the restart.
Confirm Ubuntu uses WSL 2 by running this command in Windows Terminal:

wsl.exe --list --verbose

What confirms WSL is ready?

The output should include an Ubuntu row with 2 in the VERSION column. You now have the Linux environment for the project.

Press the Windows key to open Windows search.
Type Ubuntu.
Press Enter to open Ubuntu.
Create a Linux username if Ubuntu asks during its first launch.
Create a Linux password if Ubuntu asks during its first launch.
Confirm that Ubuntu is running a Linux kernel by running this command:

uname -a

What does the kernel check prove?

This prints details about the kernel running inside Ubuntu. Linux kernel information confirms that your remaining terminal work is happening in WSL.

You should see Linux kernel details in the output. Keep this Ubuntu window open for the rest of the project.

Ubuntu not opening correctly?

Confirm that the Ubuntu row still shows WSL 2 in Windows Terminal. Restart Windows if you completed a fresh installation or conversion.

help me diagnose why Ubuntu is not opening through WSL 2

Verify Azure CLI and kubectl

Tools installed on Windows do not automatically exist inside Ubuntu. These checks confirm that your WSL shell has its own working Azure and Kubernetes clients.

Check the Azure CLI version inside Ubuntu by running this command:

az version

What does this version check show?

This prints the installed Azure CLI version plus its dependent libraries. This project requires Azure CLI 2.90.0 or later.

✔️ I see version 2.90.0 or higher

Your Azure CLI is ready inside WSL. You can move on to the kubectl check.

ⓧ I see an older version

The installed Azure CLI needs an upgrade. Ubuntu may ask for your Linux password during this process.

Upgrade Azure CLI with Microsoft's Ubuntu installer by running this command:

curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

What does the installer do?

This official installer configures Microsoft's Ubuntu package source. It installs the current Azure CLI package through the system package manager.

Confirm the upgraded version by running the check again:

az version

What confirms the upgrade?

The Azure CLI version should now be 2.90.0 or later. This confirms the WSL installation meets the project requirement.

ⓧ Command not found

Azure CLI is missing from Ubuntu. Ubuntu may ask for your Linux password while the package is installed.

Install Azure CLI with Microsoft's Ubuntu installer by running this command:

curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

What does the installer add?

This official installer adds Microsoft's package source to Ubuntu. It installs Azure CLI for commands you run inside WSL.

Confirm the installation by running the version check:

az version

What confirms the installation?

A version response of 2.90.0 or later confirms that Azure CLI is available inside Ubuntu.

Azure CLI installation failing?

Confirm that Ubuntu has internet access. Check that your Linux password grants permission for commands that use sudo.

help me troubleshoot the Azure CLI installation inside Ubuntu on WSL

Check whether kubectl is installed inside Ubuntu by running this command:

kubectl version --client

What does the client check prove?

The --client flag checks the local kubectl installation without contacting a cluster. A client version response confirms that the command is available in WSL.

✔️ I see a client version

kubectl is available inside Ubuntu. Your two command-line tools are ready for cluster authentication.

ⓧ Command not found

kubectl is missing from Ubuntu. Azure CLI can install the Kubernetes client locally.

Install kubectl through Azure CLI by running this command:

az aks install-cli

What does this installation do?

Azure CLI downloads kubectl locally. This gives your WSL shell the client needed to communicate with AKS.

Confirm that kubectl is available by running the client check again:

kubectl version --client

What confirms kubectl is ready?

A client version response confirms that kubectl is installed inside Ubuntu. Cluster access comes next.

kubectl installation failing?

Confirm that the Azure CLI check succeeded before retrying the installation. A network restriction can also prevent the client download.

help me troubleshoot az aks install-cli inside Ubuntu on WSL

Authenticate and connect to the cluster

Azure authentication identifies your subscription. A kubeconfig context then directs kubectl to the intended cluster.

The sign-in command may open a browser. Complete the displayed authentication flow with the Azure account that can access your sandbox cluster.

Start Azure CLI authentication by running this command:

az login

What does Azure sign-in establish?

This authenticates Azure CLI for your local Ubuntu session. The returned account information shows the subscriptions available to that identity.

Complete the authentication flow in the browser.
Return to the Ubuntu terminal after authentication succeeds.
Select the subscription that contains your AKS cluster by running this command:

az account set --subscription "[[AZURE_SUBSCRIPTION="your Azure subscription name or ID"]]"

Why select the subscription explicitly?

Azure CLI can access more than one subscription. Selecting the intended subscription prevents later commands from targeting an account that does not contain your cluster.

Confirm the active subscription by running this command:

az account show --output table

What should the account table confirm?

The table displays the current subscription. Its name or ID should match the subscription that contains your approved AKS cluster.

Define the target cluster variables by running these assignments:

CLUSTER_NAME="[[CLUSTER_NAME="your AKS cluster name"]]"
AKS_RG="[[AKS_RG="your AKS resource group name"]]"

What do these variables hold?

The CLUSTER_NAME variable stores the existing AKS cluster name. The AKS_RG variable stores its Azure resource group.

Later commands reuse these values. This keeps every cluster action aimed at the same sandbox.

Retrieve credentials for the target AKS cluster by running this command:

az aks get-credentials --resource-group $AKS_RG --name $CLUSTER_NAME

What do the downloaded credentials change?

Azure CLI downloads the cluster connection details. It merges them into your local kubeconfig before setting the AKS cluster as the active kubectl context.

Display the active kubectl context by running this command:

kubectl config current-context

Why check the active context?

The current context controls which Kubernetes cluster receives each kubectl command. Its output should identify the AKS cluster stored in CLUSTER_NAME.

Before the final check, predict whether every node in the target cluster will report Ready.
Verify the Azure CLI version and cluster nodes by running these commands:

az version
kubectl get nodes

What does the final check prove?

The first command proves that Azure CLI is available inside WSL. The second command proves that kubectl can authenticate to the active AKS context.

Azure CLI should report 2.90.0 or later. Every target node should show Ready in the status column.

That is the connection layer locked in. Your authenticated WSL shell now reaches the safe cluster you selected for the incident.

Connection Checkpoint

Your existing managed-identity AKS cluster is in resource group $AKS_RG under $CLUSTER_NAME. Azure CLI is authenticated to the intended subscription.

kubectl credentials are configured for the active context. All nodes report Ready.

Nodes not reporting Ready?

Confirm that the active context names your intended cluster. Check the subscription plus both cluster variables before retrieving credentials again.

help me diagnose why kubectl cannot reach my intended AKS cluster

Your WSL path into AKS is ready. Next, you will turn on the monitoring baseline that captures the incident evidence.

Enable the AKS Monitoring Baseline

Your Ubuntu shell now has authenticated access to the intended Azure Kubernetes Service cluster. Every target node reports a ready state.

Current cluster state only captures the present moment. Retained metrics expose CPU pressure over time. Logs preserve workload activity from the same incident. This step connects both evidence streams to Azure Monitor.

In this step, get ready to:

Confirm every Azure monitoring prerequisite.
Onboard the existing cluster to Azure Monitor.
Verify the monitoring agents plus the portal telemetry.

Confirm the monitoring prerequisites

Monitoring onboarding requires a managed identity on the existing cluster. Your account also needs at least Contributor access to change the cluster.

The Azure CLI can inspect both prerequisites before collection begins. This prevents an incompatible identity or insufficient access from interrupting onboarding.

Check the cluster identity type by running this command in the WSL terminal from earlier:

az aks show --name $CLUSTER_NAME --resource-group $AKS_RG --query identity.type --output tsv

What does this identity check show?

The command reads the identity configuration of the cluster stored in CLUSTER_NAME. The filtered output shows only the identity type.

You should see SystemAssigned or UserAssigned. Either value confirms that the cluster uses managed identity authentication.

Managed identity missing?

Stop before onboarding if the command returns an empty value. Monitoring onboarding requires a managed-identity AKS cluster.

Ask the cluster owner to confirm the identity configuration. Help me interpret the AKS identity result.

Azure role assignments define which cluster changes your signed-in identity can make. The next check includes assignments inherited from parent scopes through Azure RBAC.

List role assignments that apply to the AKS resource by running these commands:

AKS_ID=$(az aks show --resource-group $AKS_RG --name $CLUSTER_NAME --query id --output tsv)
az role assignment list --all --scope $AKS_ID --include-inherited --output table

What does this access check show?

The first command stores the cluster resource ID in AKS_ID. The second command lists role assignments at that cluster scope.

Inherited assignments cover access granted through the resource group or subscription. This reveals Contributor access that may sit above the cluster itself.

Find an assignment for your signed-in account or one of its groups.
Confirm the assignment provides at least Contributor access.

Cannot confirm Contributor access?

Ask your Azure administrator to confirm your effective role at the cluster or a parent scope. Continue after they confirm Contributor-level onboarding access.

Help me interpret my effective AKS role assignments.

Azure resource providers expose the service APIs used during monitoring onboarding. The cluster subscription needs four providers registered before the add-ons can create or connect supporting resources.

Provider registration can take several minutes. A quiet terminal during this process does not mean the request has stalled.

az provider register --namespace Microsoft.ContainerService
az provider register --namespace Microsoft.Insights
az provider register --namespace Microsoft.AlertsManagement
az provider register --namespace Microsoft.Monitor

Why register these providers?

Microsoft.ContainerService exposes the AKS resource APIs.
Microsoft.Insights exposes Azure monitoring capabilities.
Microsoft.AlertsManagement supports Azure Monitor alerting resources.
Microsoft.Monitor supports the managed monitoring workspace path.

Check the registration state of all four providers by running these commands:

az provider show --namespace Microsoft.ContainerService --query registrationState
az provider show --namespace Microsoft.Insights --query registrationState
az provider show --namespace Microsoft.AlertsManagement --query registrationState
az provider show --namespace Microsoft.Monitor --query registrationState

What does this verification prove?

Each line asks Azure for the current registration state of one required provider. Four completed registrations make the monitoring APIs available to the onboarding commands.

You should see Registered four times.

Provider registration still in progress?

Wait a few minutes if a provider is still registering. Run the four checks again after Azure finishes the subscription update.

A permission failure needs help from a subscription administrator. Help me diagnose this provider registration problem.

Microsoft requires the aks-preview extension to be absent before this onboarding path runs. The installed extension list determines whether you need one cleanup action.

List the Azure CLI extensions installed in WSL by running this command:

az extension list --output table

What does this extension check show?

The table lists extensions installed in your current Azure CLI environment. The monitoring prerequisite is satisfied when aks-preview is absent.

✔️ aks-preview is not listed

The extension prerequisite is already satisfied. Your cluster is ready for monitoring onboarding.

ⓧ aks-preview is listed

The installed preview extension must be removed before monitoring onboarding. This keeps the next commands on the supported Azure CLI path.

Remove the installed extension by running this command:

az extension remove --name aks-preview

What does this command change?

The command removes the Azure CLI extension from your WSL environment. It does not delete the AKS cluster or its workloads.

Confirm the extension is absent by running the list command again:

az extension list --output table

What should the table show?

The refreshed table should no longer contain aks-preview. This completes the final monitoring prerequisite.

Enable metrics and container logging

Managed Prometheus records changing resource pressure. Container insights collects container telemetry through the monitoring add-on.

Why managed monitoring?

A self-hosted Prometheus stack adds installation work plus ongoing maintenance. Azure's managed onboarding path keeps your attention on incident detection.

Prometheus supplies the workload metrics. Container insights adds container evidence through a Log Analytics workspace.

Know what starts billing

These Azure Monitor services use consumption-based billing. Prometheus charges can depend on samples ingested plus samples queried.

Container insights can incur Log Analytics ingestion plus retention charges. The cleanup section shows how to stop collection when you finish.

The first onboarding update can take several minutes while Azure connects a workspace and deploys the metrics components. Let the command finish before starting the logging add-on.

Enable managed Prometheus on the existing cluster by running this command:

az aks update --enable-azure-monitor-metrics --name $CLUSTER_NAME --resource-group $AKS_RG

What does this update create?

The update enables Prometheus collection on the existing cluster. Azure uses an applicable Azure Monitor workspace or creates one in the cluster region when no workspace is specified.

The onboarding process also deploys the node metrics agent. That agent captures the CPU evidence used during the incident investigation.

Azure CLI returns the updated cluster record when the operation completes. The in-cluster agent check in the final substep proves that collection is deployed.

Managed Prometheus onboarding failed?

Confirm that the identity check returned a managed identity type. Confirm that all four providers reached their registered state.

A permissions error needs Contributor access on the target cluster. Help me diagnose the managed Prometheus onboarding failure.

The metrics stream is now enabled. The second onboarding command adds container logs plus workload telemetry to the same incident-response baseline.

Enable the Container insights monitoring add-on by running this command:

az aks enable-addons --addon monitoring --name $CLUSTER_NAME --resource-group $AKS_RG

What does the monitoring add-on provide?

The add-on deploys the agents that collect container telemetry. Azure connects the cluster to an applicable Log Analytics workspace.

Azure can use or create a default workspace for the cluster region when no workspace is specified. Review that workspace before deleting monitoring resources later.

The completed command confirms that the monitoring add-on is enabled. The next checks prove that its agents are ready inside the cluster.

Container insights onboarding failed?

Confirm that the Azure CLI session still targets the intended subscription. Confirm that AKS_RG contains the target cluster.

Wait for any provider registration to finish before retrying. Help me diagnose the Container insights onboarding failure.

Verify the agents and telemetry

Successful onboarding needs evidence from inside the cluster plus evidence in the portal. Agent readiness is immediate proof while retained telemetry begins flowing to Azure Monitor.

Before you run these checks, consider whether every monitoring agent should already match the cluster's requested replica count.

Verify the metrics and logging agents by running these commands:

kubectl get ds ama-metrics-node --namespace=kube-system
kubectl get ds ama-logs --namespace=kube-system
kubectl get deployment ama-logs-rs --namespace=kube-system

What do these components collect?

The ama-metrics-node DaemonSet runs the metrics agent across Linux nodes.
The ama-logs DaemonSet runs the container logging agent across Linux nodes.
The ama-logs-rs Deployment supports the Container insights logging solution.

For both DaemonSets you should see matching DESIRED, CURRENT, and READY counts. The ama-logs-rs Deployment should report every requested replica as ready.

Agent counts do not match?

Allow more time when the components were deployed moments ago. Run the three verification commands again to check their latest readiness.

A component that remains unready needs investigation in kube-system. Help me diagnose the unready monitoring agent.

Azure Monitor telemetry can take about 10 minutes to begin appearing after onboarding. That delay is normal while the first metrics plus container records reach their workspaces.

Open the Azure portal in your browser.
Enter $CLUSTER_NAME in the portal search bar.
Select the matching AKS cluster resource.
Select Monitor in the cluster's Monitoring area.
Select Nodes to confirm the target nodes have started reporting telemetry.
Select Controllers to confirm controller telemetry is beginning to populate.
Select Containers to confirm container telemetry is beginning to populate.

You should see the cluster's Nodes, Controllers, and Containers data beginning to populate. This proves the portal can receive evidence from the agents you verified in kube-system.

Monitor view still empty?

Wait until the initial onboarding period has passed. Refresh the Monitor view after the first telemetry reaches Azure.

Recheck all three in-cluster components if the view remains empty. Help me trace the missing AKS telemetry.

That's the monitoring baseline established: your cluster now sends retained metrics plus container telemetry to Azure Monitor. Next up, you will create a controlled CPU incident and capture the pressure from WSL plus the portal.

Create and Detect a CPU Incident

Your WSL shell is connected to the intended AKS cluster. Azure Monitor is collecting workload telemetry.

A healthy dashboard only proves that monitoring works during quiet periods. In this step, you will put one application replica under sustained demand to capture evidence of a controlled CPU incident.

In this step, get ready to:

Deploy the official CPU-intensive sample with one ready application replica.
Record a quiet CPU baseline in WSL.
Generate sustained traffic to capture CPU-pressure evidence in WSL and Azure Monitor.

Deploy the CPU sample

The official Kubernetes sample packages a CPU-intensive application inside a Deployment. A Service gives the load generator a stable HTTP destination.

Create the php-apache Deployment and Service by applying the official manifest:

kubectl apply -f https://k8s.io/examples/application/php-apache.yaml

What does this command create?

The php-apache Deployment runs the CPU-intensive sample container.
The application Pod carries the run=php-apache label.
The php-apache Service sends HTTP traffic on port 80 to Pods with that label.

Inspect the new Deployment and Service by running these commands:

kubectl get deployment php-apache
kubectl get service php-apache

What should you see?

The Deployment table should list php-apache with 1/1 ready replicas.
The Service table should list php-apache as the HTTP route for the sample.

Good progress. Your cluster now has a single application replica that can receive controlled traffic.

Sample workload not ready?

Confirm that your active Kubernetes context still points to the approved sandbox cluster. A Deployment that remains unready can also indicate insufficient node capacity or an image-pull problem.

Use the Pod events shown by Kubernetes to identify the failing stage. Help me diagnose why the php-apache Deployment is not ready.

Record a quiet baseline

A baseline records normal resource use before the incident begins. The later reading becomes meaningful because you can compare it with this quiet state.

Measure the current CPU usage of the sample Pod by running:

kubectl top pod -l run=php-apache

Metrics and Logs Answer Different Questions

A metric is a numeric measurement captured over time. kubectl top gives you a current CPU and memory reading from the cluster's Metrics Server.

A log records an event or message from a workload. Azure Monitor retains telemetry so you can investigate how resource use changed across the incident period.

Record the CPU value from the php-apache row here: your quiet CPU value.

CPU Metrics Not Available?

The AKS-managed metrics-server supplies the data used by kubectl top. Wait briefly if the sample Pod has only just become ready.

If the command still returns no readings, confirm that the Pod remains ready. Help me diagnose missing kubectl top metrics for php-apache.

Azure Monitor can take about 10 minutes to display newly collected monitoring data. A short delay here reflects telemetry ingestion.

Return to the AKS cluster resource from earlier in the Azure portal.
Select Monitoring from the cluster menu.
Select Dashboards with Grafana.
Open the Kubernetes | Compute Resources | Cluster dashboard.
Select the workload namespace from the CPU Quota table.
Open the Kubernetes | Compute Resources | Namespace (Pods) dashboard.
Locate the php-apache workload to confirm that its quiet CPU trend is visible.

Generate sustained load

The next command deliberately raises demand against the sample Service in your approved sandbox. Its terminal stays occupied until you press Ctrl+C.

Open a second WSL terminal through the same Ubuntu environment you used earlier.
Before you run the load generator, predict whether one replica will remain near its quiet CPU baseline.
Start the temporary load-generator Pod by running:

kubectl run -i --tty load-generator --rm --image=busybox:1.28 --restart=Never -- /bin/sh -c "while sleep 0.01; do wget -q -O- http://php-apache; done"

How Does the Load Generator Work?

The command starts an interactive load-generator Pod from the busybox:1.28 image.
The request loop repeatedly calls http://php-apache through the Service.
The --restart=Never setting keeps the workload as a standalone Pod.
The --rm flag removes the temporary Pod when its interactive command ends.

Keep the second WSL terminal running during the investigation.
Switch back to the first WSL terminal.
Before you check the workload, predict whether its CPU reading will remain close to your quiet CPU value.
Measure the application Pod while the load generator runs:

kubectl top pod -l run=php-apache

What Does This Reading Prove?

The label selector isolates the application Pod from other workloads in the namespace. Its live CPU reading lets you compare the incident directly with your recorded baseline.

You should see active CPU use above your quiet CPU value. The workload still has one ready replica, so that single Pod carries the sustained demand.

CPU Use Still Looks Quiet?

Confirm that the second WSL terminal is still running the interactive request loop. Wait briefly for Metrics Server to capture a fresh reading.

If the load-generator Pod ended unexpectedly, review its terminal output for a Service connectivity or image-pull problem. Help me diagnose why the load generator is not raising php-apache CPU use.

The command-line reading captures the incident immediately. Azure Monitor can take about 10 minutes to turn the same telemetry into a visible trend.

Keep the load-generator terminal running while Azure Monitor receives the CPU samples.
Return to the Kubernetes | Compute Resources | Namespace (Pods) dashboard from earlier.
Before you refresh the dashboard, predict whether the php-apache CPU trend will stay flat or rise above its quiet period.
Refresh the dashboard after the telemetry interval.
Select the php-apache workload to isolate its CPU trend.

You should see the workload CPU trend rise during the same period as the sustained requests. The WSL reading and Azure Monitor trend now show the same controlled incident.

That is the incident captured. You have proved that the monitoring baseline can detect pressure on a real workload.

Your single replica is visibly carrying sustained demand. Next, you will convert that CPU signal into an automatic capacity response.

Automate Recovery with HPA

The controlled incident proved that one php-apache replica can buckle under sustained CPU pressure. The command-line reading plus the Azure Monitor trend now capture the same bottleneck.

Recovery needs a control loop that reacts while demand is live. A Horizontal Pod Autoscaler (HPA) uses CPU utilization to add application replicas.

You will stop the load after scale-out. The final check proves capacity returns toward its baseline.

In this step, get ready to:

Create an HPA with a 50% CPU target across a one-to-ten replica range.
Observe the php-apache Deployment scale above one replica.
Verify recovery after the load stops.

Create the autoscaling policy

An HPA is a Kubernetes control loop that compares measured CPU utilization with a target. Its replica limits define how far the response can move.

Why Autoscaling Instead of Manual Scaling?

Manual scaling depends on an operator noticing pressure before adding capacity.

The HPA evaluates live CPU demand continuously. This observe-decide-act loop keeps recovery tied to current workload conditions.

Confirm the second WSL terminal is still occupied by the load-generator command.
Apply the autoscaling thresholds by running this command:

kubectl autoscale deployment php-apache --cpu=50% --min=1 --max=10

How Does This Policy Respond?

The --cpu=50% setting establishes the average CPU utilization target.
The --min=1 setting preserves one application replica during quiet periods.
The --max=10 setting allows the response to grow as high as ten replicas.

You should see a confirmation that the autoscaler was created for php-apache.

Autoscaler Creation Failed?

The HPA needs the sample Deployment in the active namespace. It also needs healthy Metrics Server data.

Confirm the php-apache Deployment is available by running:

kubectl get deployment php-apache

This check confirms whether the autoscaler has a Deployment to control.

help me diagnose why the php-apache HPA could not be created.

Watch the workload scale out

The policy is active while the load generator continues sending requests. Watching the HPA reveals how measured CPU demand becomes extra replicas.

Switch back to the first WSL terminal.
Before you run the watch command, consider whether the replica count will stay at one or rise.
Watch the HPA react by running this command:

kubectl get hpa php-apache --watch

What Does the Watch Show?

The --watch flag keeps the command connected to replica updates.
The CPU target compares current utilization with the configured 50% threshold.
The replica count shows the capacity decision made by the HPA.

The watch continues to occupy this terminal until you stop it with Ctrl+C.

You should see the current CPU target climb above 50%. You should then see the replica count rise above one.

You have just watched the HPA convert observed CPU pressure into extra application capacity.

Press Ctrl+C in the first WSL terminal to stop the watch.
Switch back to the Azure portal tab from earlier.
Select Dashboards with Grafana under Monitoring.
Open Kubernetes | Compute Resources | Namespace (Pods) for the workload namespace.
Compare the pod CPU trend with the replica growth from your terminal.

You should see CPU pressure distributed across multiple php-apache pods. This dashboard evidence matches the HPA scale-out you observed in WSL.

Replicas Staying at One?

Confirm the load-generator command still occupies the second WSL terminal.
Allow the HPA to receive another metrics update before comparing the replica count.
Refresh the dashboard after the command-line CPU target rises above 50%.

help me diagnose why the php-apache HPA is not scaling above one replica.

Stop the load and verify recovery

Stopping the request stream removes the CPU demand signal. The HPA then uses lower utilization readings to decide when capacity can shrink.

Switch back to the second WSL terminal.
Press Ctrl+C to stop the load generator.

The interactive command exits. Its --rm behavior removes the temporary load-generator Pod.

The recovery check can take a stabilization period, so give the HPA time to confirm the lower demand.

Before you check recovery, consider which value should fall first: CPU demand or replica count.
Switch back to the first WSL terminal.
Watch the recovery by running this command:

kubectl get hpa php-apache --watch

What Proves Recovery?

The HPA continues comparing current CPU utilization with its 50% target. Lower demand eventually produces a smaller desired replica count.

The configured minimum keeps one replica available after scale-down.

You should see CPU utilization fall first. The Deployment then trends back toward one replica after the stabilization period.

Press Ctrl+C to stop the HPA watch.
Confirm the Deployment recovery by running this command:

kubectl get deployment php-apache

What Does This Final Check Prove?

The Deployment status shows the application replica count after demand has fallen. Its ready state confirms that scale-down preserved an available application instance.

You should see the ready replica count trend back toward one. The php-apache Service remains available throughout the recovery.

Workload Not Scaling Back Down?

Verify that no second WSL terminal is still running the load-generator command.
Run the HPA watch command above again through the stabilization period.
Confirm the current CPU utilization is falling below the target.

help me diagnose why the php-apache HPA is not scaling back toward one replica.

The full response loop is now working from CPU pressure through scale-out to recovery.

Secret mission

Secret Mission: Prove the Autoscaling Ceiling Is Too Low

An autoscaler can follow its policy perfectly while the workload remains short of capacity. In this optional challenge, you will temporarily cap the php-apache HPA at two replicas. You will prove the ceiling is binding before restoring the safer one-to-ten policy.

Clean Up Your Resources

Your existing AKS cluster can continue generating charges after this exercise. Azure Monitor can also keep accruing consumption-based costs.

Cost warning

Azure Monitor managed service for Prometheus charges are based on ingested samples plus queried samples. Container insights can incur Log Analytics ingestion plus retention charges.

Your existing AKS charges also continue while the cluster runs.

Choose the Pause or Delete option if you no longer need the exercise workload.
Delete only dedicated monitoring workspaces that no other cluster or resource uses.

The temporary Pod/load-generator has already been removed because both interactive load tests ended with Ctrl+C.

Resources you used:

The Deployment/php-apache workload in your active Kubernetes namespace.
The Service/php-apache network endpoint in your active namespace.
The HorizontalPodAutoscaler/php-apache policy with a minimum of one replica plus a maximum of ten replicas.
Managed Prometheus collection enabled on $CLUSTER_NAME.
Container insights collection enabled on $CLUSTER_NAME.
The applicable Azure Monitor workspace associated with $CLUSTER_NAME.
The applicable Log Analytics workspace associated with $CLUSTER_NAME.

Keep everything running

No action is needed. This option preserves the monitored workload plus its automatic recovery policy.

Keep the php-apache sample resources available for future incident drills.
Retain both monitoring add-ons to continue collecting cluster telemetry.
Preserve the associated workspaces to keep historical monitoring data.
Keep Pod/load-generator absent so the exercise does not continue creating CPU pressure.
Review your Azure Monitor cost data regularly while collection remains enabled.

Pause - I'll come back to this later

Remove the demonstration workload to stop its resource usage. Monitoring stays enabled for your next AKS exercise.

Remove the exercise workload from your active namespace by running these commands in WSL Ubuntu:

kubectl delete hpa php-apache
kubectl delete deployment php-apache
kubectl delete service php-apache

What does this cleanup remove?

The first command removes the automatic scaling policy.
The second command removes the CPU-intensive application workload.
The third command removes the internal network endpoint.

Confirm each command reports that its named resource was deleted.

Cleanup commands rejected?

Check that kubectl still uses the intended AKS context.
Confirm that your Azure CLI session still has access to the cluster.
Ask for help with the cleanup commands:

Managed Prometheus plus Container insights remain enabled. The associated monitoring workspaces also remain available.

Delete - I don't want to use this again

This option removes the sample workload. It also stops new managed Prometheus plus Container insights collection from the cluster.

Remove the exercise workload from your active namespace by running these commands:

kubectl delete hpa php-apache
kubectl delete deployment php-apache
kubectl delete service php-apache

What does this cleanup remove?

These commands remove the autoscaler plus the sample application resources. Your existing AKS cluster remains intact.

Confirm each command reports that its named resource was deleted.

Kubernetes resources still present?

Check that kubectl still points to the AKS context used throughout this project.
Confirm that the resources exist in your active namespace.
Ask for help removing the sample resources:

Edit this managed Prometheus disable template before running it:

az aks update --disable-azure-monitor-metrics -n <cluster-name> -g <cluster-resource-group>

What does this command remove?

This command removes the managed Prometheus metrics add-on from the AKS cluster. Previously collected workspace data remains stored.

Replace <cluster-name> with the value stored in CLUSTER_NAME.
Replace <cluster-resource-group> with the value stored in AKS_RG.
Run the edited command in WSL Ubuntu.
Confirm the cluster update completes successfully.

Metrics collection still enabled?

Check that both placeholders were replaced with the intended cluster values.
Confirm that your Azure CLI session uses the intended subscription.
Ask for help disabling managed Prometheus:

Edit this Container insights disable template before running it:

az aks disable-addons -a monitoring -n <cluster-name> -g <cluster-resource-group>

What does this command remove?

This command disables the Container insights monitoring add-on. Stored Log Analytics data remains available after collection stops.

Replace <cluster-name> with the value stored in CLUSTER_NAME.
Replace <cluster-resource-group> with the value stored in AKS_RG.
Run the edited command in WSL Ubuntu.
Confirm the add-on disable operation completes successfully.

Log collection still enabled?

Check that both placeholders match the target AKS cluster.
Confirm that your account still has Contributor access for the cluster.
Ask for help disabling Container insights:

Protect shared workspaces

Disabling collection leaves historical data in the associated workspaces. A workspace may also serve another cluster or Azure resource.

Review the Azure Monitor workspace associated with $CLUSTER_NAME to determine whether another cluster uses it.
Review the associated Log Analytics workspace to determine whether another resource sends data to it.
Keep each workspace when another resource uses it.
Keep each workspace when its ownership is uncertain.
Delete a workspace through its Azure resource page only when it was created solely for $CLUSTER_NAME.

Nice Work!

You did it! You turned an existing AKS cluster into an observable incident-response environment. You traced controlled CPU pressure from detection through automated recovery.

You've learned how to:

Enabled an Azure Monitor baseline with Azure Monitor managed service for Prometheus metrics plus Container insights logs. Verified the monitoring agents in kube-system.
Generated a controlled CPU incident against php-apache. Captured incident evidence in WSL plus the Azure Monitor dashboard.
Configured a Horizontal Pod Autoscaler with a 50% CPU target across one to ten replicas. Watched it add capacity under load. Confirmed the workload returned toward one replica after demand stopped.
Completed the Secret Mission by capping autoscaling at two replicas. Diagnosed the ScalingLimited condition. Restored the safer one-to-ten replica policy.

Ready to quiz yourself?