Monitor Application Gateway Failures

Monitor and resolve Application Gateway outages with Azure Monitor evidence.

Introduction

30 Second Summary

A website can run smoothly for days before failing at the moment someone needs it. A single error page proves that something broke without showing where the problem started or whether recovery is complete.

In this project, you will build an observable web delivery path in Microsoft Azure using Application Gateway, IIS, and Azure Monitor. You will inject a controlled backend outage before tracing the failure through every monitoring layer from the public endpoint to the resource-specific logs.

What You'll Build

Picture a controlled IIS outage leaving a complete trail from the public 502 response to a resolved email alert.

By the end of this project, you'll have:

  • A healthy-to-failure demonstration that moves from the custom Project 06 Healthy page to a controlled 502 response when you stop IIS remotely.
  • Proactive incident detection that turns an unhealthy backend into fired and resolved email notifications through UnhealthyHostCount.
  • A GitHub-ready incident report that correlates backend health with FailedRequests metrics and AGWAccessLogs evidence from Log Analytics.
  • An optional Secret Mission: Calculate detection and recovery intervals from your incident timeline before recommending a production-oriented alert threshold.

Are there any prerequisites?

Familiarity from Projects 01-05 or equivalent experience with Bicep, Azure CLI, virtual networks, network security groups, Windows VMs, IIS, and Application Gateway is expected.

An Azure subscription with remaining credits is required because Application Gateway and the Windows VM incur usage-based charges.

Before We Start

Operational skill comes from tracing a failure through every layer of a system. Before any hands-on work begins, commit to an observable Application Gateway incident lab where you stop IIS to create a controlled backend failure and collect evidence from the outage through recovery.

Prepare the Repository and Validate Azure Access

A reliable monitoring lab needs a known local baseline before Azure creates any paid resources. Unknown tool versions or the wrong subscription can make later evidence impossible to trust.

This step prepares the complete repository in Visual Studio Code. You will also verify the Azure CLI toolchain plus VM availability in Denmark East.

In this step, get ready to:
  • Create the repository structure with the supplied Project 06 files.
  • Verify Azure access plus the pinned command-line tools.
  • Confirm an unrestricted B-series VM size in Denmark East.
Build the repository structure

The repository separates infrastructure definitions from test scripts. It also gives every incident artifact a predictable home.

  • Press the Windows key to open Windows search.
  • Type Visual Studio Code into the search bar.
  • Press Enter to open Visual Studio Code.
  • Use the folder picker to create a new repository folder on your Desktop.
  • Record its name here: your Project 06 repository folder name.
  • Create infra, scripts, queries, docs, plus evidence inside the repository folder.
  • Leave evidence empty for the screenshots plus command output you collect later.
  • Open Visual Studio Code's terminal panel.
  • Initialize the open repository folder with Git by running this command:
git init

What does this command do?

The command creates a hidden .git directory inside the open folder. Git can now track changes to the supplied project files.

You should see a message confirming that Git initialized the repository. The exact folder path depends on the repository name you chose.

  • Add the supplied source to infra/foundation.bicep.
  • Add the supplied source to infra/monitoring.bicep.
  • Add the supplied source to scripts/Test-Gateway.ps1.
  • Add the supplied source to queries/incident-investigation.kql.
  • Add the supplied template to docs/incident-report.md.
  • Add the supplied template to docs/NEXT-PROJECT-BRIEF-entry.md.
  • Save every supplied file.

✔️ Awesome, I've got everything!

Great. Confirm that the Explorer sidebar shows five folders plus all seven supplied files.

ⓧ I'd like to double check the full code

Compare every supplied file with these complete references. Keep the file paths plus capitalization exactly as shown.

targetScope = 'resourceGroup'

@description('Azure region for all regional resources.')
param location string = 'denmarkeast'

@description('Size of the IIS backend VM. Verify subscription availability before deployment.')
param vmSize string = 'Standard_B2s'

@description('Local administrator username for the Windows VM.')
param adminUsername string = 'azureuser'

@secure()
@description('Local administrator password for the Windows VM. Supply interactively.')
param adminPassword string

var vnetName = 'vnet-az104-p06'
var appGatewaySubnetName = 'snet-appgw'
var backendSubnetName = 'snet-backend'
var backendNsgName = 'nsg-p06-backend'
var backendPublicIpName = 'pip-p06-backend'
var backendNicName = 'nic-p06-iis'
var vmName = 'vm-p06-iis'
var vmOsDiskName = 'osdisk-p06-iis'
var appGatewayPublicIpName = 'pip-p06-appgw'
var appGatewayName = 'agw-az104-p06'
var gatewayIpConfigName = 'appGatewayIpConfig'
var frontendIpConfigName = 'frontendPublicIp'
var frontendPortName = 'frontendPort80'
var backendPoolName = 'backendPoolIis'
var backendHttpSettingsName = 'backendHttpSettings80'
var listenerName = 'listenerHttp80'
var routingRuleName = 'ruleHttp80'
var probeName = 'probeIisRoot'
var backendPrivateIp = '10.0.2.4'
var iisInstallCommand = 'powershell -ExecutionPolicy Unrestricted -Command "Install-WindowsFeature -Name Web-Server -IncludeManagementTools; Set-Content -Path C:\\inetpub\\wwwroot\\iisstart.htm -Value ''<html><body><h1>Project 06 Healthy</h1><p>Traffic reached IIS through Application Gateway.</p></body></html>''"'

resource backendNsg 'Microsoft.Network/networkSecurityGroups@2024-05-01' = {
  name: backendNsgName
  location: location
  properties: {
    securityRules: [
      {
        name: 'AllowApplicationGatewayHttp'
        properties: {
          access: 'Allow'
          direction: 'Inbound'
          priority: 100
          protocol: 'Tcp'
          sourceAddressPrefix: '10.0.1.0/24'
          sourcePortRange: '*'
          destinationAddressPrefix: backendPrivateIp
          destinationPortRange: '80'
        }
      }
    ]
  }
}

resource vnet 'Microsoft.Network/virtualNetworks@2024-05-01' = {
  name: vnetName
  location: location
  properties: {
    addressSpace: {
      addressPrefixes: [
        '10.0.0.0/16'
      ]
    }
    subnets: [
      {
        name: appGatewaySubnetName
        properties: {
          addressPrefix: '10.0.1.0/24'
        }
      }
      {
        name: backendSubnetName
        properties: {
          addressPrefix: '10.0.2.0/24'
        }
      }
    ]
  }
}

resource backendPublicIp 'Microsoft.Network/publicIPAddresses@2024-05-01' = {
  name: backendPublicIpName
  location: location
  sku: {
    name: 'Standard'
    tier: 'Regional'
  }
  properties: {
    publicIPAddressVersion: 'IPv4'
    publicIPAllocationMethod: 'Static'
  }
}

resource backendNic 'Microsoft.Network/networkInterfaces@2024-05-01' = {
  name: backendNicName
  location: location
  properties: {
    enableAcceleratedNetworking: false
    enableIPForwarding: false
    networkSecurityGroup: {
      id: backendNsg.id
    }
    ipConfigurations: [
      {
        name: 'ipconfig1'
        properties: {
          primary: true
          privateIPAddress: backendPrivateIp
          privateIPAddressVersion: 'IPv4'
          privateIPAllocationMethod: 'Static'
          publicIPAddress: {
            id: backendPublicIp.id
          }
          subnet: {
            id: resourceId('Microsoft.Network/virtualNetworks/subnets', vnetName, backendSubnetName)
          }
        }
      }
    ]
  }
  dependsOn: [
    vnet
  ]
}

resource backendVm 'Microsoft.Compute/virtualMachines@2024-03-01' = {
  name: vmName
  location: location
  properties: {
    hardwareProfile: {
      vmSize: vmSize
    }
    networkProfile: {
      networkInterfaces: [
        {
          id: backendNic.id
          properties: {
            primary: true
            deleteOption: 'Detach'
          }
        }
      ]
    }
    osProfile: {
      computerName: 'p06iis'
      adminUsername: adminUsername
      adminPassword: adminPassword
      windowsConfiguration: {
        provisionVMAgent: true
        enableAutomaticUpdates: true
      }
    }
    storageProfile: {
      imageReference: {
        publisher: 'MicrosoftWindowsServer'
        offer: 'WindowsServer'
        sku: '2022-datacenter-azure-edition'
        version: 'latest'
      }
      osDisk: {
        name: vmOsDiskName
        createOption: 'FromImage'
        caching: 'ReadWrite'
        deleteOption: 'Delete'
        managedDisk: {
          storageAccountType: 'StandardSSD_LRS'
        }
      }
    }
    diagnosticsProfile: {
      bootDiagnostics: {
        enabled: true
      }
    }
  }
}

resource iisExtension 'Microsoft.Compute/virtualMachines/extensions@2024-03-01' = {
  name: 'install-iis'
  parent: backendVm
  location: location
  properties: {
    publisher: 'Microsoft.Compute'
    type: 'CustomScriptExtension'
    typeHandlerVersion: '1.10'
    autoUpgradeMinorVersion: true
    protectedSettings: {
      commandToExecute: iisInstallCommand
    }
  }
}

resource appGatewayPublicIp 'Microsoft.Network/publicIPAddresses@2024-05-01' = {
  name: appGatewayPublicIpName
  location: location
  sku: {
    name: 'Standard'
    tier: 'Regional'
  }
  properties: {
    publicIPAddressVersion: 'IPv4'
    publicIPAllocationMethod: 'Static'
  }
}

resource appGateway 'Microsoft.Network/applicationGateways@2024-05-01' = {
  name: appGatewayName
  location: location
  properties: {
    sku: {
      name: 'Standard_v2'
      tier: 'Standard_v2'
      capacity: 1
    }
    gatewayIPConfigurations: [
      {
        name: gatewayIpConfigName
        properties: {
          subnet: {
            id: resourceId('Microsoft.Network/virtualNetworks/subnets', vnetName, appGatewaySubnetName)
          }
        }
      }
    ]
    frontendIPConfigurations: [
      {
        name: frontendIpConfigName
        properties: {
          publicIPAddress: {
            id: appGatewayPublicIp.id
          }
        }
      }
    ]
    frontendPorts: [
      {
        name: frontendPortName
        properties: {
          port: 80
        }
      }
    ]
    backendAddressPools: [
      {
        name: backendPoolName
        properties: {
          backendAddresses: [
            {
              ipAddress: backendPrivateIp
            }
          ]
        }
      }
    ]
    probes: [
      {
        name: probeName
        properties: {
          protocol: 'Http'
          host: '127.0.0.1'
          path: '/'
          interval: 15
          timeout: 10
          unhealthyThreshold: 2
          minServers: 0
          pickHostNameFromBackendHttpSettings: false
          match: {
            body: ''
            statusCodes: [
              '200-399'
            ]
          }
        }
      }
    ]
    backendHttpSettingsCollection: [
      {
        name: backendHttpSettingsName
        properties: {
          cookieBasedAffinity: 'Disabled'
          port: 80
          protocol: 'Http'
          requestTimeout: 20
          pickHostNameFromBackendAddress: false
          probe: {
            id: resourceId('Microsoft.Network/applicationGateways/probes', appGatewayName, probeName)
          }
        }
      }
    ]
    httpListeners: [
      {
        name: listenerName
        properties: {
          frontendIPConfiguration: {
            id: resourceId('Microsoft.Network/applicationGateways/frontendIPConfigurations', appGatewayName, frontendIpConfigName)
          }
          frontendPort: {
            id: resourceId('Microsoft.Network/applicationGateways/frontendPorts', appGatewayName, frontendPortName)
          }
          protocol: 'Http'
          requireServerNameIndication: false
        }
      }
    ]
    requestRoutingRules: [
      {
        name: routingRuleName
        properties: {
          priority: 100
          ruleType: 'Basic'
          httpListener: {
            id: resourceId('Microsoft.Network/applicationGateways/httpListeners', appGatewayName, listenerName)
          }
          backendAddressPool: {
            id: resourceId('Microsoft.Network/applicationGateways/backendAddressPools', appGatewayName, backendPoolName)
          }
          backendHttpSettings: {
            id: resourceId('Microsoft.Network/applicationGateways/backendHttpSettingsCollection', appGatewayName, backendHttpSettingsName)
          }
        }
      }
    ]
    enableHttp2: false
  }
  dependsOn: [
    iisExtension
  ]
}

output applicationGatewayName string = appGateway.name
output applicationGatewayId string = appGateway.id
output applicationGatewayPublicIpName string = appGatewayPublicIp.name
output virtualMachineName string = backendVm.name

What does the foundation template define?

This template defines the Denmark East network plus its protected IIS backend. It also defines the public Standard_v2 Application Gateway path used by the incident lab.

targetScope = 'resourceGroup'

@description('Azure region for the Log Analytics workspace.')
param location string = 'denmarkeast'

@description('Existing Application Gateway name.')
param applicationGatewayName string = 'agw-az104-p06'

@description('Email address that receives Azure Monitor alerts. Supply interactively.')
param alertEmail string

var workspaceName = 'law-p06-${uniqueString(subscription().id, resourceGroup().id)}'
var actionGroupName = 'ag-p06-operations'
var metricAlertName = 'alert-p06-unhealthy-host'

resource applicationGateway 'Microsoft.Network/applicationGateways@2024-05-01' existing = {
  name: applicationGatewayName
}

resource workspace 'Microsoft.OperationalInsights/workspaces@2023-09-01' = {
  name: workspaceName
  location: location
  properties: {
    sku: {
      name: 'PerGB2018'
    }
    retentionInDays: 30
    publicNetworkAccessForIngestion: 'Enabled'
    publicNetworkAccessForQuery: 'Enabled'
  }
}

resource actionGroup 'Microsoft.Insights/actionGroups@2023-01-01' = {
  name: actionGroupName
  location: 'global'
  properties: {
    groupShortName: 'P06Ops'
    enabled: true
    emailReceivers: [
      {
        name: 'Project06Operator'
        emailAddress: alertEmail
        useCommonAlertSchema: true
      }
    ]
  }
}

resource diagnosticSetting 'Microsoft.Insights/diagnosticSettings@2021-05-01-preview' = {
  name: 'send-appgw-access-to-law'
  scope: applicationGateway
  properties: {
    workspaceId: workspace.id
    logAnalyticsDestinationType: 'Dedicated'
    logs: [
      {
        category: 'ApplicationGatewayAccessLog'
        enabled: true
      }
    ]
  }
}

resource unhealthyHostAlert 'Microsoft.Insights/metricAlerts@2018-03-01' = {
  name: metricAlertName
  location: 'global'
  properties: {
    description: 'Project 06: Application Gateway has one or more unhealthy backend hosts.'
    severity: 2
    enabled: true
    scopes: [
      applicationGateway.id
    ]
    evaluationFrequency: 'PT1M'
    windowSize: 'PT1M'
    autoMitigate: true
    criteria: {
      'odata.type': 'Microsoft.Azure.Monitor.SingleResourceMultipleMetricCriteria'
      allOf: [
        {
          name: 'UnhealthyBackendDetected'
          criterionType: 'StaticThresholdCriterion'
          metricName: 'UnhealthyHostCount'
          metricNamespace: 'Microsoft.Network/applicationGateways'
          timeAggregation: 'Average'
          operator: 'GreaterThan'
          threshold: 0
          dimensions: []
          skipMetricValidation: false
        }
      ]
    }
    actions: [
      {
        actionGroupId: actionGroup.id
      }
    ]
  }
}

output workspaceName string = workspace.name
output actionGroupName string = actionGroup.name
output metricAlertName string = unhealthyHostAlert.name

What does the monitoring template define?

This template connects Application Gateway access logs to a resource-specific Log Analytics table. It also defines the email Action Group plus the stateful unhealthy-backend alert.

[CmdletBinding()]
param(
    [Parameter(Mandatory = $true)]
    [string]$GatewayIp,

    [int]$Count = 6,

    [int]$DelaySeconds = 10
)

$uri = "http://$GatewayIp/"

1..$Count | ForEach-Object {
    $attempt = $_

    try {
        $response = Invoke-WebRequest -Uri $uri -TimeoutSec 15 -UseBasicParsing

        [PSCustomObject]@{
            Attempt    = $attempt
            StatusCode = [int]$response.StatusCode
            Result     = 'Success'
        }
    }
    catch {
        $statusCode = if ($null -ne $_.Exception.Response) {
            [int]$_.Exception.Response.StatusCode
        }
        else {
            'NoResponse'
        }

        [PSCustomObject]@{
            Attempt    = $attempt
            StatusCode = $statusCode
            Result     = $_.Exception.Message
        }
    }

    if ($attempt -lt $Count) {
        Start-Sleep -Seconds $DelaySeconds
    }
}

What does the gateway test report?

The script sends repeated requests to the gateway public IP. Each result records its attempt number plus the observed HTTP status.

AGWAccessLogs
| where TimeGenerated > ago(30m)
| where HttpStatus between (500 .. 599)
| project TimeGenerated, HttpStatus, ServerStatus, ErrorInfo, BackendPoolName, BackendSettingName, ServerRouted, RequestUri, InstanceId
| order by TimeGenerated desc

What evidence does the query return?

The query isolates recent Application Gateway requests with 5xx statuses. Its projected fields connect each failed request to backend-routing evidence.

# Project 06 Incident Report

Incident: Controlled IIS backend outage behind Azure Application Gateway
Date:
Resource group: rg-az104-p06-monitoring
Application Gateway: agw-az104-p06
Backend VM: vm-p06-iis

## Executive summary

Summarize the user-visible symptom, affected layer, detection method, root cause, corrective action, and final result.

## Timeline

| Event | UTC timestamp | Evidence |
|---|---|---|
| Healthy baseline confirmed | | |
| IIS stopped | | |
| First failed public request | | |
| Backend marked unhealthy | | |
| Alert fired | | |
| IIS restored | | |
| Public request returned 200 | | |
| Alert resolved | | |

## Symptom

Record the public HTTP status and what a user would experience.

## Evidence by layer

| Layer | Evidence | Conclusion |
|---|---|---|
| Client | | |
| Application Gateway frontend | | |
| Routing and probe | | |
| Backend service | | |
| Azure Monitor metric | | |
| Alert notification | | |
| Resource-specific access log | | |

## Root cause

State the deliberately introduced cause and explain why the correlated evidence identifies the backend service layer.

## Resolution

Record the command used to restore the service and the observed recovery sequence.

## Validation

- Endpoint returned HTTP 200:
- Backend health returned Healthy:
- Alert condition returned Resolved:
- Access logs show post-recovery success:

## Operational measurements

- Detection interval:
- Recovery interval:
- Alert resolution interval:

## Threshold recommendation

State whether the lab threshold should remain unchanged or be adapted for a production pool with multiple backends. Support the recommendation with evidence.

## Lessons learned

List the monitoring, troubleshooting, and cost-control lessons that transfer to production operations.

How is the incident report organized?

The template captures the timeline plus evidence from each delivery layer. It also reserves space for operational measurements plus a threshold recommendation.

## Project 06: Monitor and Respond to Application Gateway Failures with Azure Monitor

Status: Complete
Region: Denmark East
Deployment: Bicep through Azure CLI in Windows PowerShell

### Objective

Detect, investigate, and resolve an Application Gateway backend outage by correlating user symptoms with backend health, Azure Monitor metrics, a stateful alert, and resource-specific access logs.

### What I built

- A Standard_v2 Application Gateway routing HTTP traffic to one IIS backend VM
- A custom health probe and a controlled IIS failure scenario
- A Log Analytics workspace with resource-specific Application Gateway access logs
- A stateful metric alert on UnhealthyHostCount
- An email Action Group using the common alert schema
- KQL and PowerShell evidence for a complete incident timeline

### Troubleshooting story

I first stopped IIS before deploying monitoring and observed that a 502 response did not identify the failing layer. After adding diagnostics and alerting, I repeated the outage, confirmed the backend was unhealthy, observed UnhealthyHostCount and FailedRequests, received the alert, queried AGWAccessLogs, restored IIS, and verified both endpoint and alert resolution.

### Evidence

- Healthy and failed endpoint tests
- Backend health output
- Fired and resolved alert emails
- UnhealthyHostCount and FailedRequests output
- AGWAccessLogs query results
- Completed incident report with detection and recovery intervals

### Skills demonstrated

Azure Monitor, Log Analytics, KQL, Application Gateway health probes, metric alerts, Action Groups, Bicep, Azure CLI, controlled failure injection, evidence correlation, incident response, and Azure cost cleanup.

### Cleanup decision

Record whether the resource group was kept, paused, or deleted after evidence collection.

Repository: Add the GitHub repository URL here.

What does the project brief capture?

The brief turns the finished lab into a copy-ready summary. Its placeholders cover the objective plus the troubleshooting story plus the final repository link.

  • Confirm that evidence contains no files.
  • Confirm that the Explorer sidebar shows both files under infra.
  • Confirm that the Explorer sidebar shows one file under scripts.
  • Confirm that the Explorer sidebar shows one file under queries.
  • Confirm that the Explorer sidebar shows both files under docs.

Missing a supplied file?

Check the file path plus capitalization against the comparison tab. A file placed beside its required folder will break later relative paths.

Ask for help: Help me compare my Project 06 repository tree with the required infra, scripts, queries, docs, and evidence structure.

Verify Azure access and tool versions

Azure CLI controls the subscription used by every later deployment. The pinned versions keep the commands plus Bicep behavior consistent with this project.

  • Check the installed Azure CLI version by running:
az --version

What should you inspect?

Find the Azure CLI version near the top of the output. The required version for this project is 2.91.0.

✔️ I see version 2.91.0

Your Azure CLI version matches the project pin. Continue with the Bicep check below.

ⓧ I see an older version

Update Azure CLI to the pinned release before continuing. The installer may ask you to accept package agreements.

  • Install Azure CLI 2.91.0 through WinGet by running:
winget install --exact --id Microsoft.AzureCLI --version 2.91.0

What does this command install?

WinGet selects the official Azure CLI package by its exact package identifier. The version option pins the installation to 2.91.0.

  • Close the active terminal after the installer completes.
  • Open a new Windows PowerShell terminal.
  • Rerun the version check from above.

ⓧ Command not found

Azure CLI is missing from Windows. WinGet can install the pinned release.

  • Install Azure CLI 2.91.0 by running:
winget install --exact --id Microsoft.AzureCLI --version 2.91.0

What happens after installation?

The package adds the az command to Windows. A new terminal session loads the updated command path.

  • Close the active terminal after the installer completes.
  • Open a new Windows PowerShell terminal.
  • Rerun the version check from above.
  • Check the Azure CLI-managed Bicep version by running:
az bicep version

What should the result contain?

The result identifies the Bicep CLI managed by Azure CLI. This project requires version 0.48.1.

✔️ I see version 0.48.1

Your Bicep CLI matches the project pin. Continue with the extension check.

ⓧ I see another version

Install the pinned Bicep release through Azure CLI. This keeps the managed Bicep instance separate from any manual installation.

  • Install Bicep CLI v0.48.1 by running:
az bicep install --version v0.48.1

What does this command change?

Azure CLI downloads the pinned Bicep binary into its managed location. Later deployment commands use this managed version.

  • Rerun the Bicep version check from above.

ⓧ Bicep is not installed

Azure CLI can install its managed Bicep CLI directly. The download may take a moment on the first run.

  • Install Bicep CLI v0.48.1 by running:
az bicep install --version v0.48.1

What does Azure CLI manage?

Azure CLI stores its own Bicep binary. The az bicep commands use that managed installation.

  • Rerun the Bicep version check from above.
  • Check the installed Log Analytics extension version by running:
az extension show --name log-analytics --query version --output tsv

Why check this extension?

The extension supplies the Log Analytics query command used later in the incident. The required version is 1.0.0b2.

✔️ I see version 1.0.0b2

The query extension matches the project pin. Your local Azure toolchain is ready for authentication.

ⓧ I see another version

Install the exact extension version required by this project. Azure CLI replaces the mismatched local extension.

  • Install the pinned Log Analytics extension by running:
az extension add --name log-analytics --version 1.0.0b2

What does the extension add?

The extension adds Azure CLI support for querying Log Analytics workspaces. Pinning it prevents command behavior from drifting during the lab.

  • Rerun the extension version check from above.

ⓧ The extension is missing

The failed version lookup confirms that the extension is absent. Install the pinned version before continuing.

  • Install the Log Analytics extension by running:
az extension add --name log-analytics --version 1.0.0b2

What becomes available?

Azure CLI gains the Log Analytics query command used to retrieve Application Gateway access logs later.

  • Rerun the extension version check from above.

The next action opens a browser-based sign-in flow. Use the Azure account that owns the credits intended for this paid lab.

  • Start the interactive Azure sign-in by running:
az login

What happens during sign-in?

Azure CLI opens a browser for authentication. After sign-in, the terminal lists the subscriptions available to your account.

  • Complete the browser sign-in with the account that has Azure credits.
  • Return to the Windows PowerShell terminal.
  • Inspect the active subscription by running:
az account show

Why verify the active subscription?

Azure CLI sends later deployments to the active subscription. Confirming it now prevents paid resources from landing in the wrong account.

  • Confirm that the displayed subscription is the one intended for Project 06.
  • Record the intended subscription here: your intended Azure subscription name or ID.

Need to switch subscriptions?

Use the following command only when the active subscription is wrong. The value can be the subscription name or its ID.

az account set --subscription "[[AZURE_SUBSCRIPTION="your intended Azure subscription name or ID"]]"

This command changes the default subscription used by later Azure CLI commands.

  • Rerun the account check after switching:
az account show

The output should now identify the subscription you recorded above.

Having trouble with the toolchain?

Close plus reopen Windows PowerShell after installing Azure CLI. An existing terminal may still use its old command path.

Ask for help: Help me verify Azure CLI 2.91.0, Bicep CLI 0.48.1, and log-analytics extension 1.0.0b2 on Windows.

Check Denmark East VM capacity

A valid VM size can still be unavailable to a particular subscription. This preflight checks the subscription restriction before the paid deployment begins.

Before you run the check, do you expect the Restrictions column to allow Standard_B2s for your subscription?

  • Check Standard_B2s availability in Denmark East by running:
az vm list-skus --location denmarkeast --size Standard_B2s --all --output table

What does the preflight prove?

The output combines the VM SKU with restrictions applied to the active subscription. A NotAvailableForSubscription restriction means the default size cannot be deployed from that subscription.

✔️ Standard_B2s is unrestricted

The default VM size is available for your subscription in Denmark East. Record it for the deployment step.

  • Record the deployable size here: your unrestricted B-series VM size.

ⓧ Standard_B2s is restricted

Your subscription cannot deploy the default size in Denmark East. List the regional VM sizes before choosing another B-series option.

  • List VM sizes plus their subscription restrictions by running:
az vm list-skus --location denmarkeast --resource-type virtualMachines --all --output table

How should you choose a replacement?

Use the Restrictions column to find a B-series size without a subscription block. The selected value becomes the vmSize deployment parameter later.

  • Choose an unrestricted B-series row from the output.
  • Record the deployable size here: your unrestricted B-series VM size.

Your preflight now identifies a deployable VM size for the active subscription. No Azure resources have been created yet.

Your repository plus Azure toolchain are ready. Next, you will deploy the healthy Application Gateway baseline plus confirm the custom IIS page returns HTTP 200.

Deploy a Healthy Gateway Baseline

Your local checks are complete. The project can now move from prepared files to a live request path.

Using Bicep, you'll deploy an Application Gateway in front of an IIS backend. The public endpoint gives you a known-good baseline for every failure signal you investigate later.

In this step, get ready to:
  • Create the Denmark East resource group.
  • Validate the foundation template before deploying it.
  • Establish a healthy public gateway baseline with HTTP 200 evidence.
Validate the foundation deployment

An Azure resource group provides one boundary for the lab resources. Template validation checks the planned deployment before the paid infrastructure starts running.

  • Switch back to the Windows PowerShell terminal from the previous step.
  • Create the Denmark East resource group by running these commands:
$resourceGroup = 'rg-az104-p06-monitoring'
az group create --name $resourceGroup --location denmarkeast

What did these commands create?

  • The $resourceGroup variable stores the resource group name for later commands.
  • The create command establishes rg-az104-p06-monitoring in denmarkeast.
  • This resource group becomes the deployment boundary for the complete incident lab.

You should see rg-az104-p06-monitoring as the resource group name. Its location should be denmarkeast.

Resource group creation failed?

  • Confirm the terminal still uses the Azure subscription selected in the previous step.
  • Check that the resource group name is exactly rg-az104-p06-monitoring.

Ask for help: Why is my Azure resource group command failing?

The template defaults to Standard_B2s. The vmSize parameter carries your unrestricted selection from the earlier SKU check.

Azure CLI asks for the secure adminPassword after validation starts. This keeps the credential out of the command itself.

  • Set the deployment size to Standard_B2s.
  • Store a strong temporary administrator password in your password manager.
  • Validate infra\foundation.bicep by running this command:
az deployment group validate --resource-group $resourceGroup --template-file .\infra\foundation.bicep --parameters vmSize='[[VM_SIZE="Standard_B2s"]]'

What does validation check?

  • The --template-file value points Azure CLI to the existing foundation template.
  • The vmSize parameter supplies the B-series size approved by your earlier preflight.
  • The secure prompt supplies adminPassword without placing the password in shell history.
  • Validation checks the resource definitions without creating the infrastructure.

The command should finish without a validation error. Your password remains absent from the visible command.

Foundation validation failed?

  • Confirm PowerShell is inside the repository folder containing infra\foundation.bicep.
  • Confirm Standard_B2s matches the unrestricted B-series size from your preflight.
  • Re-enter the administrator password if the secure parameter prompt rejects it.

Ask for help: Help me diagnose my foundation Bicep validation failure.

Deploy the healthy request path

Successful validation proves that Azure accepts the planned foundation. The deployment command now creates the network path from the public frontend to the IIS service.

Paid resources start here

This deployment creates usage-billed Azure resources. Keep the lab running only while you collect the project evidence.

The final cleanup section removes the resource group when you finish. That cleanup stops the main ongoing charges.

Provisioning can take several minutes while Azure creates the VM and Application Gateway. A quiet stretch during this command is expected.

Azure CLI asks for adminPassword again. Use the same password stored in your password manager.

  • Deploy the foundation as p06-foundation by running this command:
az deployment group create --resource-group $resourceGroup --name p06-foundation --template-file .\infra\foundation.bicep --parameters vmSize='[[VM_SIZE="Standard_B2s"]]'

What did Bicep deploy?

  • The network layer creates vnet-az104-p06 with snet-appgw at 10.0.1.0/24. It also creates snet-backend at 10.0.2.0/24.
  • The security layer creates nsg-p06-backend. Its AllowApplicationGatewayHttp rule permits TCP port 80 from the gateway subnet to 10.0.2.4.
  • The compute layer creates pip-p06-backend, nic-p06-iis, vm-p06-iis, and osdisk-p06-iis.
  • The delivery layer creates pip-p06-appgw with agw-az104-p06. The gateway routes port 80 traffic to the IIS backend at 10.0.2.4.

The deployment should finish with a provisioning state of Succeeded. The VM uses azureuser as its default administrator username.

Foundation deployment failed?

  • Check the failed resource entry in the deployment output before rerunning the command.
  • Confirm the selected vmSize remains available in Denmark East for your subscription.
  • Confirm the secure password meets the Windows VM password requirements without sharing the password.

Ask for help: Help me troubleshoot the failed p06-foundation deployment.

The backend VM needs explicit outbound connectivity so its extension can install IIS. The Standard public IP on nic-p06-iis provides that path.

Why use a backend public IP?

The public IP gives this single-VM lab an explicit outbound route. The backend NSG still limits inbound HTTP traffic to the Application Gateway subnet.

A NAT Gateway is a stronger production choice for shared outbound connectivity. The VM public IP keeps this monitoring lab smaller.

Prove the gateway is healthy

A completed deployment only proves that Azure created the resources. The endpoint test proves that a real HTTP request crosses the gateway and reaches the custom IIS page.

  • Retrieve the public gateway IP into $gatewayIp by running these commands:
$gatewayIp = az network public-ip show --resource-group $resourceGroup --name pip-p06-appgw --query ipAddress --output tsv
$gatewayIp

What does this lookup do?

  • The Azure CLI lookup reads the static address from pip-p06-appgw.
  • PowerShell stores that address in $gatewayIp for the endpoint test.
  • The final line prints the address so you can confirm it was retrieved.

You should see one public IPv4 address in the terminal. That address belongs to the Application Gateway frontend.

Gateway IP is empty?

  • Confirm the foundation deployment finished with Succeeded.
  • Confirm the public IP resource name is pip-p06-appgw.
  • Confirm $resourceGroup still contains rg-az104-p06-monitoring.

Ask for help: Why is my Application Gateway public IP lookup returning no value?

  • Predict which status the known-good gateway route should return.
  • Test the public endpoint once by running:
.\scripts\Test-Gateway.ps1 -GatewayIp $gatewayIp -Count 1

What does the test prove?

  • The script sends one HTTP request to the address stored in $gatewayIp.
  • The StatusCode field shows whether the gateway served the request successfully.
  • The Result field records the request outcome for your evidence.

You should see StatusCode set to 200. The Result value should be Success.

Endpoint test is not healthy?

  • Rerun the test after the custom probe has marked the backend healthy.
  • Confirm $gatewayIp contains the frontend address from pip-p06-appgw.
  • Check that vm-p06-iis finished provisioning its IIS extension.

Ask for help: Why is Test-Gateway.ps1 not returning HTTP 200?

  • Open the gateway page in your default browser by running:
Start-Process -FilePath "http://$gatewayIp/"

What does this command open?

PowerShell builds the URL from $gatewayIp. Your default browser opens the Application Gateway public endpoint.

You should see the heading Project 06 Healthy above a message confirming that traffic reached IIS through Application Gateway. You now have a visible healthy baseline for the incident lab.

Healthy page is missing?

  • Refresh the browser after the endpoint script returns 200.
  • Confirm the browser address begins with http://.
  • Confirm the page request uses the public address stored in $gatewayIp.

Ask for help: Why does the gateway test succeed while the healthy page is missing?

  • Press Windows+Shift+S to open the screenshot overlay.
  • Drag over the browser area containing the public URL and the healthy page.
  • Select the capture notification to open Snipping Tool.
  • Select the save control.
  • Choose the repository's evidence folder.
  • Enter healthy-gateway-page.png as the filename.
  • Select Save.

The saved file should now exist at evidence\healthy-gateway-page.png.

Your public gateway now serves a proven healthy IIS baseline. Next, you'll stop IIS and observe how little a public failure reveals without monitoring.

Inject and Restore the Blind-Spot Failure

Your healthy Azure Application Gateway proved that the delivery path works when every layer is available. That baseline cannot show how much one HTTP symptom reveals during an outage.

A controlled IIS outage lets you experience that uncertainty without opening an RDP session. The contrast between endpoint behavior and backend health exposes the monitoring blind spot.

In this step, get ready to:
  • Stop IIS remotely to create a controlled backend outage.
  • Compare the failed endpoint with backend health evidence.
  • Prove the endpoint returns to healthy service after the investigation.
Stop IIS remotely

VM Run Command executes a Windows PowerShell command inside the VM through your existing Azure CLI session. You can control the web service without exposing remote desktop access.

This action deliberately interrupts the lab endpoint. The VM plus its deployed resources remain intact.

  • Set the resource variables plus stop IIS remotely by running these commands:
$vmName = 'vm-p06-iis'
$appGatewayName = 'agw-az104-p06'
az vm run-command invoke --resource-group $resourceGroup --name $vmName --command-id RunPowerShellScript --scripts "NET STOP w3svc"

What do these commands do?

  • The $vmName variable identifies the Windows VM that hosts the backend website.
  • The $appGatewayName variable identifies the gateway whose backend health you inspect next.
  • VM Run Command sends the final service instruction into the VM through the Azure control plane.
  • The service instruction stops the World Wide Web Publishing Service that serves the IIS page.

Azure CLI reports that the remote execution completed. IIS is now unavailable to the gateway probe.

Before you test the endpoint, do you think all three requests will still return 200?

  • Generate three endpoint checks by running:
.\scripts\Test-Gateway.ps1 -GatewayIp $gatewayIp -Count 3 -DelaySeconds 10

What does this test do?

  • The script sends requests to the public address stored in $gatewayIp.
  • The Count value produces three attempts.
  • The DelaySeconds value leaves ten seconds between attempts so the gateway probe can update backend state.

You should see at least one attempt report StatusCode 502 once the backend becomes unhealthy. This shortfall is intentional.

The 502 proves that the public request failed after the gateway lost its healthy backend. The symptom alone cannot identify whether the fault sits with the client, gateway, network, probe, or backend service.

Still seeing only 200 responses?

  • Confirm that the earlier VM Run Command completed successfully.
  • Rerun the gateway test command above to give the health probe another evaluation cycle.

Ask for help: Why is my gateway still returning 200 after I stopped IIS?

Trace the blind spot

Backend health is the gateway-side view of its configured pool. It can reveal whether the custom health probe accepts the backend at 10.0.2.4.

  • Inspect the gateway backend health by running:
az network application-gateway show-backend-health --resource-group $resourceGroup --name $appGatewayName --output jsonc

What does backend health add?

  • The command asks the gateway for its current probe assessment.
  • The JSON output connects the failed public endpoint to the backend pool containing 10.0.2.4.
  • This evidence narrows the incident to the path between the gateway probe and the backend service.

You should see backend 10.0.2.4 with health Unhealthy. This result adds gateway-side evidence that the endpoint test could not provide.

  • Switch back to Visual Studio Code from earlier.
  • Select docs/incident-report.md in the Explorer sidebar.
  • Record the IIS stop under Timeline.
  • Record the first HTTP 502 under Timeline.
  • Record the Unhealthy backend state under Timeline.
  • Describe the public failure in the Symptom section.
  • Note the still-unproven client, gateway, network, probe, plus backend layers in Evidence by layer.
  • Save docs/incident-report.md.

You have captured the blind spot clearly. The endpoint proved an outage while backend health supplied the first layer-specific clue.

No Azure Monitor detector or diagnostic log trail exists yet. Your report therefore separates confirmed evidence from the layers that remain unproven.

Restore the healthy baseline

Starting IIS removes the deliberately introduced cause. Application Gateway must then accept healthy probe responses before public traffic succeeds again.

  • Restore IIS remotely by running:
az vm run-command invoke --resource-group $resourceGroup --name $vmName --command-id RunPowerShellScript --scripts "NET START w3svc"

What does this command restore?

  • VM Run Command sends the service-start instruction to vm-p06-iis.
  • Starting the publishing service makes the custom IIS page available on port 80 again.
  • The gateway resumes routing only after its probe recognizes the backend as healthy.

Azure CLI reports that the remote execution completed. The backend service is available for the next probe check.

Before you run the final endpoint check, do you expect recovery to appear immediately or only after the probe accepts IIS again?

  • Test the recovered public endpoint by running:
.\scripts\Test-Gateway.ps1 -GatewayIp $gatewayIp -Count 1

What does this check prove?

  • A successful request proves that public traffic can reach IIS through the gateway again.
  • The single attempt keeps this recovery check focused on the current endpoint state.

You should see StatusCode 200 with a successful result. The public delivery path is working again.

Endpoint still returning 502?

  • Confirm that the service-start VM Run Command completed successfully.
  • Rerun the endpoint test above until the probe accepts the backend.

Ask for help: Why is my gateway still returning 502 after I started IIS?

  • Switch back to docs/incident-report.md.
  • Record the IIS restore under Timeline.
  • Record the returned HTTP 200 under Timeline.
  • Record the service-start command in the Resolution section.
  • Save docs/incident-report.md.
  • Confirm the gateway has recovered its backend by running:
az network application-gateway show-backend-health --resource-group $resourceGroup --name $appGatewayName --output jsonc

What completes the recovery evidence?

  • The endpoint test proves that the user-facing path returns HTTP 200.
  • The backend health output proves that the gateway probe recognizes 10.0.2.4 as Healthy again.
  • Together these observations close the controlled blind-spot incident.

You should now see backend 10.0.2.4 with health Healthy. No monitoring resources have been attached to the gateway.

Backend still marked Unhealthy?

  • Confirm that the endpoint test returns HTTP 200.
  • Rerun the backend health command after the probe completes another check.

Ask for help: Why is my backend health still Unhealthy after the endpoint recovered?

That controlled failure is closed. Your endpoint is healthy again, so the next step can add diagnostics plus proactive alerting to the same known failure path.

Deploy Diagnostics and Alerting

The first outage exposed a blind spot in your Application Gateway. The 502 response proved that users were affected.

Backend health pointed to the server side. No monitoring system recorded the incident or contacted you.

In this step, you will deploy Azure Monitor resources from Bicep. A metric alert will detect unhealthy backend hosts. Log Analytics will preserve request evidence.

An Action Group will send the alert to a verified email receiver. This gives the next incident a detector plus an investigation trail.

In this step, get ready to:
  • Deploy the monitoring resources from infra/monitoring.bicep.
  • Verify the Action Group email receiver.
  • Confirm healthy request evidence in AGWAccessLogs.
Validate and deploy the monitoring template

The infra/monitoring.bicep file already defines the detector plus its investigation layer. Validation checks the template against Azure before any monitoring resource is created.

Both deployment commands prompt for alertEmail. Azure CLI keeps the address out of the command itself.

  • Prepare an email inbox for the alertEmail prompt.
  • Validate the monitoring template by running:
az deployment group validate --resource-group $resourceGroup --template-file .\infra\monitoring.bicep --parameters applicationGatewayName=$appGatewayName

What does validation check?

Azure CLI checks the template syntax plus the resource definitions. It also confirms that required deployment parameters can be resolved.

Validation does not create the workspace or alerting resources. A successful result means the template is ready for deployment.

Template validation failed?

  • Check that your PowerShell terminal is still inside the Project 06 repository.
  • Confirm that infra/monitoring.bicep is present in the Visual Studio Code Explorer.
  • Confirm that $appGatewayName still contains agw-az104-p06.

Ask for help: Help me diagnose why my Project 06 monitoring Bicep template failed Azure deployment validation.

This deployment adds a Log Analytics workspace plus alert notifications that can incur usage charges. The cleanup section removes the entire resource group after you collect your evidence.

Expect the deployment to take a few minutes while Azure creates the workspace plus the alerting resources. A quiet terminal during provisioning is normal.

  • Deploy the monitoring resources by running:
az deployment group create --resource-group $resourceGroup --name p06-monitoring --template-file .\infra\monitoring.bicep --parameters applicationGatewayName=$appGatewayName

What did the deployment create?

  • The workspace uses the PerGB2018 SKU with 30 days of retention.
  • The send-appgw-access-to-law diagnostic setting sends ApplicationGatewayAccessLog records to a resource-specific table.
  • The ag-p06-operations Action Group contains the Project06Operator email receiver.
  • The alert-p06-unhealthy-host rule evaluates average UnhealthyHostCount every minute.

Before checking the deployment, what state do you expect after Azure finishes provisioning every monitoring resource?

  • Check the deployment state by running:
az deployment group show --resource-group $resourceGroup --name p06-monitoring --query properties.provisioningState --output tsv

What should you see?

You should see Succeeded. This confirms that Azure created the workspace, diagnostic setting, Action Group, and metric alert.

That is the monitoring foundation deployed. Your healthy gateway now has both a proactive detector and a request-level evidence store.

Deployment did not succeed?

  • Confirm that the active Azure subscription contains rg-az104-p06-monitoring.
  • Check that the email address supplied for alertEmail is valid.
  • Review the deployment details for the first resource that Azure could not create.

Ask for help: Help me identify the failed resource in my p06-monitoring deployment.

✔️ Awesome, the deployment succeeded

Your monitoring resources are deployed. Keep the PowerShell session open for the workspace checks later in this step.

ⓧ I'd like to double check the full code

Compare your existing infra/monitoring.bicep file with the complete monitoring template below.

targetScope = 'resourceGroup'

@description('Azure region for the Log Analytics workspace.')
param location string = 'denmarkeast'

@description('Existing Application Gateway name.')
param applicationGatewayName string = 'agw-az104-p06'

@description('Email address that receives Azure Monitor alerts. Supply interactively.')
param alertEmail string

var workspaceName = 'law-p06-${uniqueString(subscription().id, resourceGroup().id)}'
var actionGroupName = 'ag-p06-operations'
var metricAlertName = 'alert-p06-unhealthy-host'

resource applicationGateway 'Microsoft.Network/applicationGateways@2024-05-01' existing = {
  name: applicationGatewayName
}

resource workspace 'Microsoft.OperationalInsights/workspaces@2023-09-01' = {
  name: workspaceName
  location: location
  properties: {
    sku: {
      name: 'PerGB2018'
    }
    retentionInDays: 30
    publicNetworkAccessForIngestion: 'Enabled'
    publicNetworkAccessForQuery: 'Enabled'
  }
}

resource actionGroup 'Microsoft.Insights/actionGroups@2023-01-01' = {
  name: actionGroupName
  location: 'global'
  properties: {
    groupShortName: 'P06Ops'
    enabled: true
    emailReceivers: [
      {
        name: 'Project06Operator'
        emailAddress: alertEmail
        useCommonAlertSchema: true
      }
    ]
  }
}

resource diagnosticSetting 'Microsoft.Insights/diagnosticSettings@2021-05-01-preview' = {
  name: 'send-appgw-access-to-law'
  scope: applicationGateway
  properties: {
    workspaceId: workspace.id
    logAnalyticsDestinationType: 'Dedicated'
    logs: [
      {
        category: 'ApplicationGatewayAccessLog'
        enabled: true
      }
    ]
  }
}

resource unhealthyHostAlert 'Microsoft.Insights/metricAlerts@2018-03-01' = {
  name: metricAlertName
  location: 'global'
  properties: {
    description: 'Project 06: Application Gateway has one or more unhealthy backend hosts.'
    severity: 2
    enabled: true
    scopes: [
      applicationGateway.id
    ]
    evaluationFrequency: 'PT1M'
    windowSize: 'PT1M'
    autoMitigate: true
    criteria: {
      'odata.type': 'Microsoft.Azure.Monitor.SingleResourceMultipleMetricCriteria'
      allOf: [
        {
          name: 'UnhealthyBackendDetected'
          criterionType: 'StaticThresholdCriterion'
          metricName: 'UnhealthyHostCount'
          metricNamespace: 'Microsoft.Network/applicationGateways'
          timeAggregation: 'Average'
          operator: 'GreaterThan'
          threshold: 0
          dimensions: []
          skipMetricValidation: false
        }
      ]
    }
    actions: [
      {
        actionGroupId: actionGroup.id
      }
    ]
  }
}

output workspaceName string = workspace.name
output actionGroupName string = actionGroup.name
output metricAlertName string = unhealthyHostAlert.name

How the template fits together

  • The workspace stores Application Gateway access records for investigation.
  • The diagnostic setting selects the resource-specific AGWAccessLogs destination.
  • The Action Group defines the verified notification receiver.
  • The stateful metric alert fires above zero unhealthy hosts. It automatically resolves after the condition clears.
Verify the email receiver

A new Action Group email receiver must complete one-time passcode verification within 30 minutes. An unverified receiver cannot deliver the alert notifications used in the next incident.

The verification step confirms that Project06Operator can receive the common alert schema emails configured by the template.

  • Check the inbox for the email address supplied to alertEmail.
  • Find the verification message from Microsoft Azure.
  • Follow the verification link in the message.
  • Enter the one-time passcode on the verification page.
  • Confirm that the receiver status is verified.

The confirmation page should show that the receiver is verified. Your Action Group can now deliver fired and resolved notifications.

Why receiver verification matters

The metric alert can evaluate successfully even when its receiver is unverified. Verification closes that notification gap before you inject the next failure.

Generate and query healthy access logs

The diagnostic setting writes resource-specific access logs to AGWAccessLogs. A healthy request gives you a known-good row before the next outage produces failed-request evidence.

  • Generate a healthy request through the gateway by running:
.\scripts\Test-Gateway.ps1 -GatewayIp $gatewayIp -Count 1

What did the request prove?

The script sends one request to the public gateway address. A 200 result proves that IIS remains healthy while the new diagnostic setting captures traffic.

  • Load the deployed workspace name plus its customer ID into your PowerShell session by running:
$workspaceName = az deployment group show --resource-group $resourceGroup --name p06-monitoring --query properties.outputs.workspaceName.value --output tsv
$workspaceId = az monitor log-analytics workspace show --resource-group $resourceGroup --workspace-name $workspaceName --query customerId --output tsv

What do these variables hold?

  • The $workspaceName variable holds the deterministic workspace name returned by the deployment.
  • The $workspaceId variable holds the workspace customer ID required by the query command.

Application Gateway access logs are collected every 60 seconds. The first query can be early, so one empty result does not mean the diagnostic setting failed.

  • Wait through one access-log collection cycle.

Before you run the query, which HTTP status do you expect the healthy request to produce in the access-log table?

  • Query the newest healthy access-log records by running:
az monitor log-analytics query --workspace $workspaceId --analytics-query "AGWAccessLogs | where TimeGenerated > ago(15m) | project TimeGenerated, HttpStatus, RequestUri | top 5 by TimeGenerated desc" --timespan PT30M --output table

What should you see?

You should see at least one AGWAccessLogs row with HttpStatus set to 200. The row proves that resource-specific access logging is working.

The timestamp identifies when Azure recorded the request. The request URI should show the gateway path that reached the healthy IIS backend.

No access-log rows yet?

  • Wait through another 60-second collection cycle.
  • Run the same Log Analytics query again.
  • Confirm that the gateway test returned 200 before the query.
  • Confirm that $workspaceId contains the customer ID for the deployed workspace.

Ask for help: Help me troubleshoot why AGWAccessLogs has no healthy request rows.

Your gateway now has a verified notification path plus queryable request evidence. Next, you will repeat the outage and test whether these signals detect the failure through recovery.

Correlate the Incident and Recover

Your healthy Application Gateway proved that public traffic can reach the backend. The same backend failure now needs to produce evidence that identifies the affected layer.

This time, Azure Monitor must detect the outage. Log Analytics must preserve the failed request trail.

You will restore IIS after capturing the outage evidence. You will then prove recovery across the endpoint, backend health, alert state, and access logs.

In this step, get ready to:
  • Reproduce the controlled IIS outage under active monitoring.
  • Correlate backend health with platform metrics, Action Group notifications, and access logs.
  • Prove recovery through a completed incident report.
Reproduce the monitored outage

A timestamp anchors every signal to the same incident window. Capture the failure start before stopping the backend service.

  • In Visual Studio Code, return to docs/incident-report.md.
  • Record the current UTC timestamp in the IIS stopped row of the timeline.
  • Stop IIS remotely through VM Run Command by running:
az vm run-command invoke --resource-group $resourceGroup --name $vmName --command-id RunPowerShellScript --scripts "NET STOP w3svc"

What does this command prove?

VM Run Command sends NET STOP w3svc to the Windows VM through Azure. This isolates the failure to the IIS service without requiring an RDP connection.

  • Generate repeated public failure evidence by running:
.\scripts\Test-Gateway.ps1 -GatewayIp $gatewayIp -Count 6 -DelaySeconds 10

What does this test capture?

The script sends six requests through the public gateway. The delay gives the custom health probe time to mark the backend unhealthy.

Each result records the attempt number. It also records the status code and request result.

You should see failed requests with HTTP status 502 once Application Gateway stops routing traffic to the unhealthy backend.

  • Save the failed test output as evidence/second-incident-endpoint-results.
  • Inspect the backend state by running:
az network application-gateway show-backend-health --resource-group $resourceGroup --name $appGatewayName --output jsonc

Why check backend health?

The public 502 confirms a user-visible failure. Backend health narrows the incident to the gateway pool member at 10.0.2.4.

You should see the backend reported as Unhealthy. This connects the public symptom to the backend layer.

  • Save the unhealthy backend output in evidence/backend-health.json.
  • Complete the failed-request and unhealthy-backend rows in docs/incident-report.md.

Still seeing successful requests?

  • Confirm that VM Run Command reported a successful execution of the IIS stop command.
  • Rerun the six-request test after the health probe has completed another check.
  • Confirm that $gatewayIp still contains the public IP for pip-p06-appgw.

Ask for help: My gateway test still returns successful requests after I stopped IIS. Help me verify the VM command result, gateway IP, and backend health.

Correlate metrics, alerts, and access logs

Platform metrics show when the gateway detects an unhealthy backend. The failed-request trail in AGWAccessLogs supplies request-level evidence for the same incident window.

  • Store the Application Gateway resource ID by running:
$appGatewayId = az network application-gateway show --resource-group $resourceGroup --name $appGatewayName --query id --output tsv

Why store the resource ID?

Azure Monitor metric queries target the full resource ID. Saving it in $appGatewayId keeps the next two queries tied to the same Application Gateway.

  • Query the unhealthy backend count by running:
az monitor metrics list --resource $appGatewayId --metrics UnhealthyHostCount --aggregation Average --interval 1m --output jsonc

What does UnhealthyHostCount show?

The UnhealthyHostCount metric records how many backend hosts Application Gateway considers unhealthy. The alert evaluates its Average value over each one-minute window.

You should see a nonzero value during the controlled outage. That value is the detector that drives the stateful metric alert.

  • Save the metric output as evidence/metrics-unhealthy-host-count.json.
  • Query the failed request total by running:
az monitor metrics list --resource $appGatewayId --metrics FailedRequests --aggregation Total --interval 1m --output jsonc

What does FailedRequests add?

The FailedRequests metric counts requests that failed during each one-minute interval. Its Total aggregation shows the incident's request impact.

  • Save the metric output as evidence/metrics-failed-requests.json.
  • Check the inbox for the verified Project06Operator receiver.
  • Save the fired severity-2 notification as evidence/action-group-fired-email.

The fired email proves that alert-p06-unhealthy-host crossed its threshold. The notification also proves that ag-p06-operations reached the verified receiver.

  • Load the investigation KQL query into PowerShell by running:
$incidentQuery = Get-Content -Path .\queries\incident-investigation.kql -Raw

What does this command load?

The -Raw option loads the entire query as one string. PowerShell stores that string in $incidentQuery for the Log Analytics request.

  • Run the investigation query against the workspace by running:
az monitor log-analytics query --workspace $workspaceId --analytics-query $incidentQuery --timespan PT1H --output table

How does this query correlate the incident?

The query filters AGWAccessLogs for 5xx requests. It returns the timestamp and HTTP status for each failure.

It also returns the server status, error information, backend pool, backend setting, routed server, request URI, and gateway instance ID. Those fields connect each failed request to the gateway's routing path.

Access-log collection runs every 60 seconds. End-to-end ingestion can take longer, so a quiet first query does not mean the diagnostic setting failed.

  • Rerun the investigation query until failed 5xx rows appear.
  • Save the failed-request table as evidence/agw-access-logs-failures.

No failed access-log rows?

  • Confirm that $workspaceId still contains the customer ID for the Project 06 workspace.
  • Confirm that the failed gateway requests occurred within the query's one-hour timespan.
  • Wait for another collection cycle before rerunning the same query.

Ask for help: My AGWAccessLogs query returns no 5xx rows after the outage. Help me check the workspace ID, query window, diagnostic setting, and ingestion timing.

Restore IIS and prove recovery

Recovery requires more than restarting the service. The endpoint, backend health, and stateful alert must all return to healthy states.

  • Restore IIS through VM Run Command by running:
az vm run-command invoke --resource-group $resourceGroup --name $vmName --command-id RunPowerShellScript --scripts "NET START w3svc"

What does this command restore?

The command starts the World Wide Web Publishing Service on vm-p06-iis. IIS can now serve the custom Project 06 page to the gateway probe.

Before you run the endpoint test, decide whether you expect immediate success or another probe cycle first.

  • Test the recovered public endpoint by running:
.\scripts\Test-Gateway.ps1 -GatewayIp $gatewayIp -Count 1

What does the recovery test prove?

A successful request proves that Application Gateway has resumed routing traffic to IIS. The expected status is 200.

  • Repeat the endpoint test if the first request still reports a failure.

You should eventually see status 200 with a successful result. That closes the user-visible part of the incident.

  • Append the successful recovery output to evidence/second-incident-endpoint-results.
  • Verify the recovered backend state by running:
az network application-gateway show-backend-health --resource-group $resourceGroup --name $appGatewayName --output jsonc

What confirms backend recovery?

The backend health output should now report 10.0.2.4 as Healthy. This proves that the gateway probe can reach IIS again.

  • Append the healthy backend output to evidence/backend-health.json.

Alert resolution takes at least three consecutive clear evaluations. Each evaluation runs once per minute, so the delay is expected.

  • Wait for the resolved notification from alert-p06-unhealthy-host.
  • Save the resolved notification as evidence/action-group-resolved-email.

Recovery is taking longer than expected?

  • Confirm that the endpoint test returns 200 before waiting for alert resolution.
  • Confirm that backend health reports Healthy.
  • Allow three consecutive one-minute evaluations after the condition clears.

Ask for help: My endpoint recovered, but the stateful alert has not resolved. Help me compare backend health, the alert evaluation schedule, and the three-clear-evaluation requirement.

Complete the incident narrative

The completed incident report should preserve the healthy baseline, failure timestamps, public 502 evidence, unhealthy backend state, fired alert time, restoration time, recovered 200 result, and resolved alert time.

Its evidence table should connect the client symptom to gateway behavior. It should also connect the health metrics, alert notification, and access-log records to the same backend outage.

The project brief should describe the controlled failure before monitoring. It should also describe the repeated incident after monitoring was active.

  • Complete docs/incident-report.md using the captured incident evidence.
  • Complete docs/NEXT-PROJECT-BRIEF-entry.md using the finished incident narrative.

✔️ My incident package is complete

Your evidence now traces the controlled outage from the first failed request through alert resolution. Keep the cleanup decision placeholder for the resource cleanup section.

ⓧ I want to double check the evidence

Confirm that every incident signal has a saved artifact:

  • Confirm evidence/action-group-fired-email contains the fired notification.
  • Confirm evidence/action-group-resolved-email contains the resolved notification.
  • Confirm evidence/agw-access-logs-failures contains the failed 5xx rows.
  • Confirm evidence/backend-health.json contains the unhealthy and healthy backend states.
  • Confirm evidence/metrics-failed-requests.json contains the failed-request metric output.
  • Confirm evidence/metrics-unhealthy-host-count.json contains the unhealthy-host metric output.
  • Confirm evidence/second-incident-endpoint-results contains the failed and recovered endpoint tests.
  • Confirm evidence/healthy-gateway-page.png still shows Project 06 Healthy.
Check the documentation
  • Confirm docs/incident-report.md contains the timeline, evidence correlation, root cause, recovery sequence, and validation results.
  • Confirm docs/NEXT-PROJECT-BRIEF-entry.md contains the build summary, troubleshooting story, evidence, demonstrated skills, and cleanup decision placeholder.

Strong work. Your incident package now proves detection, investigation, correlation, recovery, and stateful alert resolution.

Secret mission

Measure Detection and Recovery

Your recovered incident contains more than proof of failure. Turn its timestamps into detection, recovery, and alert-resolution intervals. Then use the evidence to recommend a production threshold for a multi-backend pool.

Clean Up Your Resources

Clean Up Your Resources

Your incident evidence is complete. Choose whether to keep the lab running, pause it for later, or delete the Azure resource group. The live Application Gateway and virtual machine continue to incur usage-based charges until you pause or delete them.

Cost warning

Application Gateway Standard_v2 and the Windows VM are the main cost drivers. Log Analytics ingestion can also be billable. Azure Monitor alert time series can also be billable.

Pausing stops Application Gateway charges and VM compute charges. Disks, networking resources, retained logs, and other persistent resources can continue to incur charges. Pause and Delete are the safer choices when you have finished collecting evidence.

Resources you used:

  • Resource group rg-az104-p06-monitoring in Denmark East.
  • VNet vnet-az104-p06 with snet-appgw and snet-backend.
  • NSG nsg-p06-backend with the Application Gateway HTTP rule.
  • Two Standard static public IP resources named pip-p06-appgw and pip-p06-backend.
  • Windows Server VM vm-p06-iis with NIC nic-p06-iis, OS disk osdisk-p06-iis, and extension install-iis.
  • Standard_v2 Application Gateway agw-az104-p06 with its listener, routing rule, backend pool, and health probe.
  • Log Analytics workspace stored in $workspaceName with the diagnostic setting send-appgw-access-to-law.
  • Action Group ag-p06-operations with its verified email receiver.
  • Stateful metric alert alert-p06-unhealthy-host.

Keep everything running

No action is needed. Choose this option if you want to keep demonstrating the live gateway or collect more monitoring evidence.

  • The Application Gateway continues routing public HTTP traffic to the IIS backend.
  • The VM continues serving the custom Project 06 Healthy page.
  • The Log Analytics workspace continues retaining Application Gateway access logs.
  • Your local repository keeps the Bicep templates, investigation query, incident report, portfolio entry, and saved evidence.
  • Ongoing Azure charges continue until you pause or delete the lab.

Pause - I'll come back to this later

Shut down the billable compute while keeping the Azure configuration and collected logs. Persistent resources remain available for your return.

  • Stop the Application Gateway and deallocate the VM by running these commands:
$resourceGroup = 'rg-az104-p06-monitoring'

az network application-gateway stop --resource-group $resourceGroup --name $appGatewayName
az vm deallocate --resource-group $resourceGroup --name $vmName

What Does Pausing Change?

The first command stops the Application Gateway. A stopped Application Gateway does not incur gateway charges.

The second command deallocates the VM. Azure stops VM compute billing while the OS disk, public IPs, workspace, and other persistent resources remain.

  • Wait for both Azure CLI commands to complete.

The live endpoint stops responding after the gateway shuts down. Your configuration and incident evidence remain available.

Starting the gateway can take several minutes, so a quiet wait after the command is normal.

  • Resume the VM and Application Gateway later by running these commands:
az vm start --resource-group $resourceGroup --name $vmName
az network application-gateway start --resource-group $resourceGroup --name $appGatewayName

What Does Resuming Change?

The first command starts the IIS VM. The second command starts the Application Gateway after the backend is available.

Usage-based charges resume when these resources start running again.

  • Return to the gateway public IP stored in $gatewayIp after both commands complete.

You should see the custom Project 06 Healthy page again.

Delete - I don't want to use this again

Deleting the resource group is permanent, which can feel drastic. Your repository stays intact because this command targets only the Azure resource group.

  • Confirm that docs/incident-report.md, docs/NEXT-PROJECT-BRIEF-entry.md, and the evidence files are committed to your repository.
  • Record your deletion choice in the cleanup decision section of docs/NEXT-PROJECT-BRIEF-entry.md.
  • Delete all Azure lab resources by running this command:
$resourceGroup = 'rg-az104-p06-monitoring'

az group delete --name $resourceGroup --yes --no-wait

What Does This Command Remove?

Deleting rg-az104-p06-monitoring removes every Azure resource listed above. This includes the gateway, VM, networking resources, workspace, diagnostic setting, Action Group, and alert rule.

The --yes flag skips the confirmation prompt. The --no-wait flag returns control to PowerShell while Azure continues the deletion.

This cleanup can take several minutes while Azure removes the compute and networking dependencies.

  • Give Azure several minutes to remove the resource group.
  • Return to the resource groups list in the Azure portal.
  • Confirm that rg-az104-p06-monitoring no longer appears.

Still See the Resource Group?

The command uses --no-wait, so Azure can continue deleting resources after PowerShell returns.

  • Wait several more minutes before checking the resource groups list again.
  • Refresh the resource groups list to check the latest deletion state.
  • Open the remaining resource group to identify any resource that Azure could not remove.

Help me investigate an Azure resource group deletion that is taking longer than expected.

That is the cost-heavy lab cleared from Azure. Your incident report and evidence remain ready for your Project 06 repository.

Nice Work!

Nice Work!

Strong finish! You built an observable Microsoft Azure delivery path with a recovered IIS backend.

You learned how to:

  • Deployed a healthy delivery path with a Standard_v2 Azure Application Gateway routing public HTTP traffic to a Windows Server backend. Defined the environment in Bicep for repeatable deployment.
  • Reproduced a controlled outage with an HTTP 502 symptom before monitoring. Repeated the failure after adding Azure Monitor diagnostics. Correlated the endpoint symptom with backend health. Confirmed the failure with UnhealthyHostCount plus FailedRequests. Queried AGWAccessLogs in Log Analytics with KQL for request-level evidence.
  • Connected a stateful metric alert to a verified email Action Group. Captured the fired notification. Restored IIS. Confirmed endpoint recovery. Captured the resolved notification. Prepared the completed incident report for GitHub. Completed the Project 06 portfolio entry.
  • Secret Mission: Measured the detection interval from the incident timeline. Calculated the recovery interval. Calculated the alert-resolution interval. Evaluated UnhealthyHostCount threshold options for a multi-backend production pool. Supported the recommendation with incident evidence.

Ready to quiz yourself?