Upgrade an AKS Cluster Safely

Safely upgrade an AKS cluster with prechecks, rolling nodes, and validation.

Introduction

30 Second Summary

Routine software updates can become late-night incidents when hidden problems surface. For teams supporting critical services, a careful sequence keeps the change controlled.

In this project, you will safely upgrade an Azure Kubernetes Service (AKS) cluster from Kubernetes 1.35 to 1.36. You will prove the workflow against a running sample workload.

What You'll Build

Picture comparing your before-and-after terminal evidence as your cluster moves to 1.36 with every demo pod still healthy.

By the end of this project, you'll have:

A live cluster upgrade that moves AKS from 1.35 to 1.36 while your sample workload stays healthy.
A pre-upgrade evidence pack with a kubent deprecated API scan plus a baseline of node versions and pod health.
A control-plane-first workflow that exposes the blocked node-pool shortcut before completing a rolling upgrade in the correct order.
Secret Mission: Configure automatic patch upgrades to run during a weekend maintenance window.

Are there any prerequisites?

This project assumes that Terraform, Azure CLI, Helm, and VS Code are installed on Windows. You also need kubectl plus an active Azure Pay-As-You-Go subscription.

Before We Start

Before the hands-on work begins, this step gives you a clear picture of the production-style AKS upgrade workflow you are about to practise. That shared plan keeps each later decision focused on protecting workload health.

Provision the AKS Cluster

You already have the production upgrade sequence mapped out. Now you need a real Azure Kubernetes Service cluster on Kubernetes 1.35 so you can practise that sequence safely.

You will use Terraform to create the cluster plus its supporting network. You will also connect kubectl to the cluster before installing the deprecation scanner used later.

In this step, get ready to:

Define the Azure provider plus the AKS networking resources.
Apply the Terraform configuration for a Kubernetes 1.35 cluster.
Connect kubectl to the cluster and install kubent 0.7.3.

Create the Terraform configuration

The Azure CLI session supplies the identity that Terraform uses for provisioning. Signing in first keeps authentication errors separate from infrastructure errors.

Press the Windows key to open Windows search.
Type PowerShell into the search box.
Press Enter to open PowerShell.
Sign in to Azure by running this command:

az login

What does this command do?

The command starts an interactive Azure sign-in. Your browser may open so you can complete authentication with the account linked to your Pay-As-You-Go subscription.

Complete the browser sign-in if Azure requests it.
Return to PowerShell after your subscription details appear in the terminal.

Having trouble signing in?

Check that the browser uses the account linked to your Azure subscription. A different Microsoft account can leave Terraform without access to the subscription.

Ask for help: Help me troubleshoot an Azure CLI sign-in problem on Windows.

Create the aks-upgrade folder on your Desktop by running these commands:

New-Item -Path "$HOME\Desktop" -Name "aks-upgrade" -ItemType "Directory"
Set-Location -Path "$HOME\Desktop\aks-upgrade"

What do these commands do?

The first command creates the aks-upgrade project folder on your Desktop.
The second command moves your PowerShell session into that folder.

Your PowerShell prompt should now end with Desktop\aks-upgrade.

Does the folder already exist?

An existing aks-upgrade folder can cause the creation command to report an error. Use the existing empty folder only if it belongs to this project.

Ask for help: Help me check whether my existing aks-upgrade folder is safe to use.

Press the Windows key to open Windows search.
Type Visual Studio Code into the search box.
Press Enter to open Visual Studio Code.
Select File from the top menu.
Select Open Folder.
Choose the Desktop\aks-upgrade folder.
Create main.tf inside the aks-upgrade folder.

The first part of main.tf pins AzureRM to 4.81.0. The legacy registration mode lets this provider generation register the Azure resource providers required by the configuration.

Add the provider configuration plus the resource group to main.tf by pasting this code:

# Pin the AzureRM provider used by this project
terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "= 4.81.0"
    }
  }
}

# Let Terraform register the Azure resource providers it needs
provider "azurerm" {
  features {}
  resource_provider_registrations = "legacy"
}

# Keep the upgrade lab resources together
resource "azurerm_resource_group" "main" {
  name     = "aks-upgrade-rg"
  location = "eastus"
}

What does this code do?

The required_providers block locks the AzureRM provider to 4.81.0.
The provider block sets resource_provider_registrations = "legacy" for automatic provider registration.
The resource group becomes the container for the cluster plus its supporting network.

Save main.tf.
Initialize the Terraform project from the existing PowerShell window by running:

terraform init

What did Terraform initialize?

Terraform reads the provider requirement from main.tf. It downloads the pinned AzureRM provider into the local project.

You should see output confirming that initialization completed successfully.

Seeing an initialization error?

Confirm that PowerShell is still inside Desktop\aks-upgrade. Terraform must run from the folder containing main.tf.

Ask for help: Help me debug terraform init for this AzureRM project.

The next resources form the virtual network used by the system node pool. A network security group stays attached to the AKS subnet so the network remains Terraform-managed.

Add the network resources below the resource group in main.tf by pasting this code:

# Create the private network for the AKS nodes
resource "azurerm_virtual_network" "main" {
  name                = "aks-upgrade-vnet"
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name
  address_space       = ["10.0.0.0/16"]
}

# Reserve an address range for the system node pool
resource "azurerm_subnet" "aks" {
  name                 = "aks-subnet"
  resource_group_name  = azurerm_resource_group.main.name
  virtual_network_name = azurerm_virtual_network.main.name
  address_prefixes     = ["10.0.1.0/24"]
}

# Create the security boundary for the AKS subnet
resource "azurerm_network_security_group" "aks" {
  name                = "aks-upgrade-nsg"
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name
}

# Attach the security group to the AKS subnet
resource "azurerm_subnet_network_security_group_association" "aks" {
  subnet_id                 = azurerm_subnet.aks.id
  network_security_group_id = azurerm_network_security_group.aks.id
}

What does this network code do?

The virtual network reserves the private 10.0.0.0/16 address space.
The AKS subnet gives the system node pool the smaller 10.0.1.0/24 range.
The association attaches the network security group to that subnet.

Save main.tf.
Preview the network resources by running:

terraform plan

What should the plan contain?

The plan should preview the resource group plus the virtual network resources. Terraform has not created them at this point.

Does the network plan fail?

Check each Terraform reference for matching resource labels such as azurerm_resource_group.main and azurerm_subnet.aks.

Ask for help: Help me debug the Terraform network plan for my AKS cluster.

The AKS resource pins both layers to Kubernetes 1.35. The system pool contains one Standard_D2s_v7 node attached to the Terraform-managed subnet.

Add the cluster resource below the network resources in main.tf by pasting this code:

# Create the Kubernetes 1.35 control plane
resource "azurerm_kubernetes_cluster" "main" {
  name                = "aks-upgrade-cluster"
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name
  dns_prefix          = "aks-upgrade"
  kubernetes_version  = "1.35"

  # Run one system node on the same Kubernetes minor version
  default_node_pool {
    name                 = "system"
    node_count           = 1
    vm_size              = "Standard_D2s_v7"
    os_sku               = "AzureLinux"
    vnet_subnet_id       = azurerm_subnet.aks.id
    orchestrator_version = "1.35"
  }

  # Give AKS a managed identity for Azure operations
  identity {
    type = "SystemAssigned"
  }

  # Use Azure networking for the cluster
  network_profile {
    network_plugin = "azure"
  }
}

What does this cluster code do?

The cluster-level kubernetes_version setting creates a 1.35 control plane.
The default_node_pool block creates one system node at the same minor version.
The subnet reference places that node inside the Terraform-managed network.
The managed identity lets AKS perform operations against Azure resources.

Save main.tf.

✔️ Awesome, I've got everything!

Your provider settings, network resources, and AKS cluster definition are now saved in main.tf.

ⓧ I'd like to double check the full code

Compare your complete main.tf file with this reference.

# Pin the AzureRM provider used by this project
terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "= 4.81.0"
    }
  }
}

# Let Terraform register the Azure resource providers it needs
provider "azurerm" {
  features {}
  resource_provider_registrations = "legacy"
}

# Keep the upgrade lab resources together
resource "azurerm_resource_group" "main" {
  name     = "aks-upgrade-rg"
  location = "eastus"
}

# Create the private network for the AKS nodes
resource "azurerm_virtual_network" "main" {
  name                = "aks-upgrade-vnet"
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name
  address_space       = ["10.0.0.0/16"]
}

# Reserve an address range for the system node pool
resource "azurerm_subnet" "aks" {
  name                 = "aks-subnet"
  resource_group_name  = azurerm_resource_group.main.name
  virtual_network_name = azurerm_virtual_network.main.name
  address_prefixes     = ["10.0.1.0/24"]
}

# Create the security boundary for the AKS subnet
resource "azurerm_network_security_group" "aks" {
  name                = "aks-upgrade-nsg"
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name
}

# Attach the security group to the AKS subnet
resource "azurerm_subnet_network_security_group_association" "aks" {
  subnet_id                 = azurerm_subnet.aks.id
  network_security_group_id = azurerm_network_security_group.aks.id
}

# Create the Kubernetes 1.35 control plane
resource "azurerm_kubernetes_cluster" "main" {
  name                = "aks-upgrade-cluster"
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name
  dns_prefix          = "aks-upgrade"
  kubernetes_version  = "1.35"

  # Run one system node on the same Kubernetes minor version
  default_node_pool {
    name                 = "system"
    node_count           = 1
    vm_size              = "Standard_D2s_v7"
    os_sku               = "AzureLinux"
    vnet_subnet_id       = azurerm_subnet.aks.id
    orchestrator_version = "1.35"
  }

  # Give AKS a managed identity for Azure operations
  identity {
    type = "SystemAssigned"
  }

  # Use Azure networking for the cluster
  network_profile {
    network_plugin = "azure"
  }
}

Apply the cluster and connect kubectl

A Terraform plan shows the exact resources that Azure is about to receive. Reviewing it gives you a final checkpoint before billable infrastructure is created.

Preview the complete infrastructure plan by running:

terraform plan

What should the complete plan contain?

The plan should include aks-upgrade-rg plus its virtual network resources. It should also include aks-upgrade-cluster with one system node.

Applying this plan creates billable Azure resources. The cleanup section removes the entire lab when you finish.

Apply the reviewed Terraform plan by running:

terraform apply

What happens during the apply?

Terraform sends the resource definitions to Azure. Follow the terminal confirmation prompt to approve the plan.

Expect the AKS cluster to take several minutes to provision. The changing status messages show that Azure is still working.

You should see Terraform report that the apply completed. That is the infrastructure backbone in place for every upgrade exercise that follows.

Did the apply fail?

Check the first Azure error in the output. Subscription permissions, regional capacity, or an unavailable Kubernetes version can stop provisioning before later resources are created.

Ask for help: Help me diagnose my failed AKS Terraform apply.

Terraform creates the cluster in Azure. Your local kubeconfig still needs the cluster credentials before kubectl can send requests to it.

Merge the AKS credentials into your local kubeconfig by running:

az aks get-credentials --resource-group aks-upgrade-rg --name aks-upgrade-cluster

What does this command do?

The Azure CLI retrieves access details for aks-upgrade-cluster. It merges them into the kubeconfig used by kubectl.

You should see confirmation that the cluster context was added or merged.

Could not retrieve the credentials?

Confirm that the resource group is named aks-upgrade-rg. Confirm that the cluster is named aks-upgrade-cluster.

Ask for help: Help me connect kubectl to my Terraform-managed AKS cluster.

Install kubent and verify the environment

kubent scans Kubernetes resources for API versions that can disappear during an upgrade. Version 0.7.3 includes deprecation rules through Kubernetes 1.32.

Check whether kubent is already available by running:

kubent --version

How should you use the result?

The output identifies whether the expected release is already installed. Choose the tab that matches what PowerShell shows.

✔️ I see version 0.7.3

kubent 0.7.3 is already available through your PATH. You can continue to the final check.

ⓧ I see an older version

The older executable needs to be replaced with kubent 0.7.3 before the scan step.

Download kubent-0.7.3-windows-amd64.tar.gz from the official release.
Extract the downloaded archive in your Downloads folder.
Replace the older kubent.exe file in its existing PATH folder with the extracted executable.
Close PowerShell.
Start a new PowerShell window through Windows search.

Still seeing the older release?

A PowerShell window opened before the replacement can keep using an older PATH state. Close every PowerShell window before checking again.

Ask for help: Help me find why Windows still runs an older kubent version.

ⓧ Command not found

Windows cannot find a kubent executable through PATH. You will give the extracted executable a permanent folder before adding that folder to your user environment.

Download kubent-0.7.3-windows-amd64.tar.gz from the official release.
Extract the downloaded archive in your Downloads folder.
Create the C:\Tools folder in File Explorer.
Create the C:\Tools\kubent folder inside it.
Move the extracted kubent.exe file into C:\Tools\kubent.

Why use a permanent folder?

Files in Downloads are easy to move or delete. A stable tools folder gives the PATH entry a reliable destination.

Press the Windows key to open Windows search.
Search for environment variables.
Open the result for editing environment variables for your account.
Select the Path entry under your user variables.
Add C:\Tools\kubent as a new entry.
Confirm the environment variable changes.
Close PowerShell.
Start a new PowerShell window through Windows search.

Is kubent still unavailable?

Confirm that kubent.exe sits directly inside C:\Tools\kubent. A nested extraction folder leaves the executable outside the PATH location.

Ask for help: Help me fix the kubent PATH entry on Windows.

Before you run the final check, what version and node status do you expect to see?

Verify kubent plus the connected AKS node by running these commands:

kubent --version
kubectl get nodes -o wide

What should you compare?

The kubent output should identify version 0.7.3.
The node table should contain one node with status Ready.
The Kubernetes version column should show 1.35.x.
The node pool behind that row uses Standard_D2s_v7.

Your local tools now reach a live Kubernetes 1.35 cluster. The single system node is ready for a workload.

Is the node missing or not ready?

A missing node can mean the wrong kubeconfig context is active. A node that is still starting may need another short wait before it reaches Ready.

Ask for help: Help me diagnose my AKS node status.

Your AKS lab is provisioned and connected. Next, you will deploy a real workload that can reveal whether the upgrade preserves application health.

Deploy the Sample Workload

Your AKS cluster is healthy at Kubernetes 1.35.x. An empty cluster cannot show whether an upgrade protects real workloads.

This step uses Helm to deploy the NGINX Ingress Controller into demo. The resulting pods give you a workload to monitor during the upgrade.

In this step, get ready to:

Add the official ingress-nginx chart repository to Helm.
Deploy the ingress controller into the demo namespace.
Verify the pods, deployment, and public service.

Add the chart repository and deploy the release

A Helm repository is a catalog of installable charts. Adding the official repository lets Helm locate the ingress controller chart.

Add the official ingress-nginx repository by running this command:

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

What does this command do?

The command maps the local repository name ingress-nginx to the official chart source. Helm can now resolve the ingress-nginx/ingress-nginx chart.

You'll see confirmation that the ingress-nginx repository was added.

Repository Not Added?

Check that your Windows machine can reach GitHub.
Confirm the repository URL matches the command above.

Ask for help: Why can't Helm add the ingress-nginx repository?

A Helm release is a named installation of a chart. This release creates the demo namespace plus the ingress controller resources inside it.

Install the ingress controller release by running this command:

helm install ingress-nginx ingress-nginx/ingress-nginx --namespace demo --create-namespace

What does this command do?

Helm creates a release named ingress-nginx from the official chart.
The namespace option places the release inside demo.
The namespace creation option creates demo as part of the installation.
The chart creates a controller service with type LoadBalancer so Azure can assign a public address.

Good progress. Your cluster now has a live workload that can expose disruption during the later upgrade.

Release Not Deployed?

Confirm that kubectl still connects to aks-upgrade-cluster.
Check that the ingress-nginx repository was added successfully.

Ask for help: Why did my ingress-nginx Helm installation fail?

Verify the workload health

Kubernetes exposes separate health signals for the deployment, pods, and service. Together they prove the workload is ready for upgrade testing.

The external IP is the slowest part of this check. Azure usually assigns it within a few minutes.

Before you run these checks, predict whether every readiness signal will pass.

Check the deployment, pods, and service by running these commands:

kubectl get deployments -n demo
kubectl get pods -n demo
kubectl get svc -n demo

What Should You Compare?

The deployment result shows whether the desired replica count matches the available replica count.
The pod result shows each controller pod's READY count plus its STATUS.
The service result shows the service TYPE plus its assigned EXTERNAL-IP.

You'll see every controller pod in Running state with READY set to 1/1.

The deployment's desired replica count matches its available replica count.

The ingress-nginx-controller service has type LoadBalancer with an assigned external IP.

Still Waiting for a Healthy Workload?

Wait a few minutes if the external IP has not been assigned.
Repeat the service check above after the wait.
Read the pod status to identify any controller pod that has not reached Running.

Ask for help: Why are my ingress controller pods or LoadBalancer service not ready?

Your demo workload is live with healthy pods plus a public service. Next, you'll scan for deprecated APIs and capture the baseline for your upgrade.

Scan and Baseline the Cluster

Your deployed workload gives this AKS upgrade something real to protect. Changing the cluster without a baseline would leave you guessing whether a problem existed before the upgrade.

You will use Azure CLI to confirm the supported target. You will scan for deprecated APIs before recording the current Kubernetes versions and workload health.

In this step, get ready to:

Confirm that Kubernetes 1.36.x is an available upgrade target.
Review the cluster for deprecated API usage with kubent.
Capture the node versions, control plane version, workload health, pod count, and LoadBalancer external IP.

Check the available upgrade target

AKS only offers upgrade targets that follow its supported version path. Checking the available targets confirms that the cluster can move from 1.35.x to the next minor version.

Check the upgrade targets for aks-upgrade-cluster by running this command in the PowerShell terminal from earlier:

az aks get-upgrades --resource-group aks-upgrade-rg --name aks-upgrade-cluster --output table

You should see the current control plane on 1.35.x. The available upgrades should include 1.36.x.

What does this check prove?

The upgrade table reflects the paths currently offered for this cluster. Seeing 1.36.x confirms that the next minor version is available before you change anything.

Can't see version 1.36?

Check that the resource group is aks-upgrade-rg.
Check that the cluster name is aks-upgrade-cluster.
Confirm that your Azure CLI session still uses the subscription containing the cluster.

Ask for help: Help me diagnose why Kubernetes 1.36 is missing from my AKS upgrade targets.

Scan for deprecated APIs

Removed APIs can break workloads when the API server stops accepting their old definitions. A pre-upgrade scan gives you a chance to find those definitions before the cluster changes.

Know the scan boundary

Version 0.7.3 includes deprecation rules through Kubernetes 1.32. A clean result means the tool found no deprecated APIs within that coverage.

Your cluster is already on 1.35.x. Keep the coverage limit beside the report so nobody treats the scan as proof about later deprecations.

Scan the connected cluster for deprecated API usage by running:

kubent

How to read the scan

Review every reported resource as a potential upgrade blocker. Record the affected workload so its manifest can be updated before an incompatible upgrade.

A clean scan is also valid evidence. Label it with the 1.32 coverage limit when you keep it with the baseline.

Good, you now have a deprecated-API report tied to the cluster before its version changes.

Having trouble running kubent?

Confirm that the kubent executable remains available on your PATH.
Confirm that kubectl still connects to aks-upgrade-cluster.

Ask for help: Help me troubleshoot my kubent pre-upgrade scan.

Capture the pre-upgrade baseline

A baseline turns the upgrade into a before-and-after comparison. It records the versions and workload signals that must remain healthy when the upgrade finishes.

Before you capture the first baseline value, do you expect the node version to match the control plane version? Keep your prediction in mind.

Capture the current node status and version by running:

kubectl get nodes -o wide

You should see one healthy node in the system node pool. Its Kubernetes version should be 1.35.x.

Record the node version in your baseline: your system node version.

Capture pod health across every namespace by running:

kubectl get pods --all-namespaces

You should see the workloads across every namespace. The nginx ingress controller pods in demo should remain healthy with their containers ready.

Count the pod rows belonging to the demo namespace.
Record the count in your baseline: your demo pod count.

Capture the current AKS control plane details by running:

az aks show --resource-group aks-upgrade-rg --name aks-upgrade-cluster --output table

You should see the control plane on Kubernetes 1.35.x. This value should match the node version you recorded.

Record the control plane version in your baseline: your control plane version.

Capture the current service details in the demo namespace by running:

kubectl get svc -n demo

You should see ingress-nginx-controller with its assigned external IP. This is the address you compare after the upgrade.

Record the assigned address in your baseline: your demo external IP.

Missing part of the baseline?

Check that your current kubectl connection points to aks-upgrade-cluster if the node or pod commands cannot reach the cluster.
Check the demo namespace if the nginx ingress controller rows are missing.
Confirm that your Azure CLI session still uses the subscription containing aks-upgrade-rg if the control plane command fails.

Ask for help: Help me complete my AKS pre-upgrade baseline.

That's your safety net in place. You now have the supported target, scan report, current versions, workload health, pod count, and service address ready for comparison.

Your pre-upgrade evidence is captured. Next, you will begin the cluster upgrade with a clear picture of what healthy looks like.

Upgrade the Control Plane

Your pre-upgrade baseline shows that your AKS cluster is healthy on Kubernetes 1.35.x.

The order between the control plane upgrade and the node pool upgrade determines whether Azure accepts the change. In this step, you'll test the tempting shortcut before applying the accepted sequence.

In this step, get ready to:

Test a node-pool-first upgrade against the existing 1.35.x control plane.
Apply the upgrade sequence that AKS accepts.
Confirm the version split between the control plane and the system node pool.

Attempt the node pool upgrade first

The system node pool currently matches the control plane at 1.35.x. Trying 1.36 here tests whether the node pool can move ahead on its own.

Before you run this, do you think AKS accepts the newer node pool version?

Attempt the node-pool-first upgrade by running this command:

az aks nodepool upgrade --resource-group aks-upgrade-rg --cluster-name aks-upgrade-cluster --name system --kubernetes-version 1.36 --no-wait

AKS Rejects the Version Order

You'll see an error containing NodePoolMcVersionIncompatible. The requested 1.36 node pool would be newer than the 1.35.x control plane.

That rejection is useful evidence. It proves that the system node pool cannot move ahead of the control plane.

Keep the failed command visible in your terminal for the checkpoint below.

Seeing a Different Result?

Compare the resource group in your command with aks-upgrade-rg.
Compare the cluster name in your command with aks-upgrade-cluster.
Check that the requested node pool version is 1.36.

Ask for help: My node-pool-first attempt did not show the expected incompatibility error.

Upgrade only the control plane

The failed attempt exposes the AKS version gate. The control plane defines the highest version that a node pool can use.

The Azure CLI pauses for confirmation after the next command starts. A quiet terminal for a few minutes is normal while Azure updates the managed control plane.

Start the control-plane-only upgrade by running this command:
Approve the confirmation prompt when it appears.

az aks upgrade --resource-group aks-upgrade-rg --name aks-upgrade-cluster --kubernetes-version 1.36 --control-plane-only

What Does This Command Do?

The --kubernetes-version 1.36 option selects the next supported minor version.
The --control-plane-only option confines this upgrade to the managed control plane.
The system node pool stays on 1.35.x until you upgrade it separately.

When the operation finishes, the terminal returns control to you. The cluster is then ready for separate version checks.

Control Plane Upgrade Stuck?

Give the operation a few minutes before treating the quiet terminal as a problem. Control plane upgrades take time to complete.

Ask for help: Help me diagnose an AKS control plane upgrade that has not completed.

Verify the version split

Before you check, which version do you expect to see on each cluster layer now?

Check the control plane version by running this command:

az aks show --resource-group aks-upgrade-rg --name aks-upgrade-cluster --output table

What Should You Compare?

This check reads the cluster-level properties. Find the KubernetesVersion value in the table.

You'll see 1.36.x for the control plane. This confirms that the first half of the upgrade is complete.

Check the node pool version separately by running this command:

az aks nodepool list --resource-group aks-upgrade-rg --cluster-name aks-upgrade-cluster --query "[].{Name:name,Version:orchestratorVersion}" --output table

Why Check the Node Pool Separately?

The query reduces the output to the node pool name and its orchestrator version. This makes the temporary version split easy to spot.

You'll see system at 1.35.x. The control plane is now one minor version ahead of the node pool.

Versions Do Not Match the Checkpoint?

Wait a few minutes before repeating the control plane check if it still shows 1.35.x.
Confirm both checks target aks-upgrade-cluster in aks-upgrade-rg.

Ask for help: My AKS control plane and node pool versions do not match the expected checkpoint.

You have proven the control-plane-first rule with a real AKS rejection. Next, you'll upgrade the system node pool and compare the healthy workload against your baseline.

Upgrade Node Pools and Validate

Your AKS control plane now runs Kubernetes 1.36.x. The system node pool still runs Kubernetes 1.35.x, which leaves the cluster upgrade incomplete.

Your workload is still running on the older node. A rolling upgrade replaces that node in stages before you compare the finished cluster against your pre-upgrade baseline.

In this step, get ready to:

Upgrade the system node pool to Kubernetes 1.36.x.
Verify the Kubernetes 1.36.x replacement node.
Compare the post-upgrade workload health against your baseline.

Upgrade the system node pool

A rolling upgrade swaps node capacity in stages to preserve workload availability. This operation can take several minutes, so a quiet terminal does not mean it has stalled.

Upgrade the system node pool by running this command:

az aks nodepool upgrade --resource-group aks-upgrade-rg --cluster-name aks-upgrade-cluster --name system --kubernetes-version 1.36

How Does the Rolling Upgrade Work?

AKS adds replacement capacity according to the node pool surge settings.
AKS cordons the old node to prevent new workloads from landing there.
AKS drains the old node so its workloads move to the replacement.

Wait for the command to finish before checking the replacement node.

Before you check, which Kubernetes version do you expect the replacement node to report? Hold that prediction for the result.

Inspect the replacement node by running this command:

kubectl get nodes -o wide

What Should You See?

You should see one replacement node with status Ready. Its Kubernetes version should show 1.36.x.

That result confirms the workload node has caught up with the control plane.

Still Seeing the Older Node Version?

Allow the rolling upgrade more time if the replacement node is still joining the cluster.
Confirm the upgrade targeted the system node pool on aks-upgrade-cluster.

Ask for help: Help me diagnose why my AKS replacement node still reports Kubernetes 1.35.x.

Validate the demo workload

Draining reschedules the nginx ingress controller onto the replacement node. A healthy result preserves the pod count recorded in your pre-upgrade baseline.

Before you inspect the workload, what pod health do you expect after the move? Keep your prediction in mind as you run the check.

Inspect the pods in the demo namespace by running this command:

kubectl get pods -n demo

What Should You See?

Every demo pod should report Running with READY 1/1. The number of pod rows should match your pre-upgrade baseline.

No pod should report CrashLoopBackOff or Error.

Compare the number of demo pods with the count in your pre-upgrade baseline.
Confirm every pod has a ready count of 1/1.
Confirm every pod has a status of Running.

Is a Demo Pod Unhealthy?

Wait briefly if a replacement pod is still starting after the drain.
Compare the unhealthy pod name with the nginx ingress controller pods in your baseline.
Check that every listed pod belongs to the demo namespace.

Ask for help: Help me investigate an unhealthy demo pod after the AKS node pool upgrade.

Compare the final state with your baseline

The pod check proves the workload recovered on the replacement node. Your final comparison confirms the control plane version first.

Before you run the final checks, do you expect the control plane version and service address to match the target state? Compare your prediction with each result.

Check the control plane version by running this command:

az aks show --resource-group aks-upgrade-rg --name aks-upgrade-cluster --output table

Check the Control Plane

The Kubernetes version in the cluster output should show 1.36.x. This confirms the control plane remains on the upgraded version.

Check the node pool version by running this command:

az aks nodepool list --resource-group aks-upgrade-rg --cluster-name aks-upgrade-cluster --query "[].{Name:name,Version:orchestratorVersion}" --output table

Check the Node Pool

The system row should show version 1.36.x. The matching control plane version confirms the required upgrade order is complete.

Inspect the demo service address by running this command:

kubectl get svc -n demo

Check the Service Address

The ingress-nginx-controller service should still use type LoadBalancer. Its external IP should match the address in your pre-upgrade baseline.

An unchanged address confirms the node replacement preserved the existing service endpoint.

Compare the current external IP with the address in your pre-upgrade baseline.
Save the node output with the workload output as your post-upgrade evidence.

Does the Service Address Look Different?

Confirm you are comparing the ingress-nginx-controller row with the same service in your baseline.
Confirm the service still reports type LoadBalancer.

Ask for help: Help me compare my demo LoadBalancer service with the pre-upgrade baseline.

That is the full upgrade complete: the control plane and system node pool now run Kubernetes 1.36.x. Your nginx ingress controller stayed healthy through the rolling replacement.

Secret mission

Configure Auto-Upgrade with a Maintenance Window

Manual upgrades give you control, but repeating them across every environment does not scale. Configure patch-level auto-upgrades with a weekend maintenance window so AKS can apply updates during planned hours.

Clean Up Your Resources

The Standard_D2s_v7 node costs approximately $0.10 per hour. The Azure Load Balancer is billed separately.

Keeping the cluster active also lets the patch auto-upgrade channel apply future patch updates. Choose Pause or Delete if you have finished experimenting.

Nice Work!

That upgrade is complete! Your AKS cluster now runs Kubernetes 1.36.

What you learned:

Provision a starting cluster at 1.35 with Terraform. Deploy the nginx ingress controller with Helm as a live workload.
Run a deprecated API scan with kubent 0.7.3. Capture a pre-upgrade baseline for the nodes. Record workload health. Record the service external IP.
Trigger the NodePoolMcVersionIncompatible error through a node-pool-first test. Upgrade the control plane before the node pool. Complete a rolling upgrade. Compare the final cluster with the baseline to prove the workload stayed healthy.
Secret Mission: Set the auto-upgrade channel to patch. Schedule a planned maintenance window for weekends from 02:00 to 06:00 UTC.

Ready to quiz yourself?