Part 9: Troubleshoot Azure Networks

Introduction

30 Second Summary

When something online stops responding, every cause can feel like the same blank wall. Guessing at settings can erase the clue that points to the real fault.

In this project, you will build a private incident lab with two Azure Virtual Machines. You will use Azure Network Watcher evidence to isolate each failure to the correct network layer.

What You'll Build

You will watch the same private SSH path move from Reachable to Unreachable, then use diagnostic evidence to restore it after each fault.

By the end of this project, you'll have:

• A repeatable known-good baseline that proves TCP port 22 works by private IP or private FQDN.
• A layered troubleshooting workflow that distinguishes an NSG denial from a blackhole route or a DNS failure.
• Diagnostic evidence from Connection troubleshoot, IP flow verify, Next hop, effective routes, and effective security rules that shows why each path failed.
• Secret Mission: Triage a wrong-answer DNS incident by proving each healthy layer before correcting the A record.

Are there any prerequisites?

You need an active Azure subscription with permission to create and delete resources. Basic Azure portal experience is enough because this lab starts fresh.

Before We Start

Before any hands-on work in Azure, this checkpoint locks in the connection you are investigating. You will troubleshoot private TCP 22 connectivity from nw-source-vm to nw-target-vm against a known-good baseline before introducing faults.

Deploy the Isolated Troubleshooting Lab

Older Azure resources can carry stale routes or settings from earlier projects. Those differences would make every diagnosis harder to trust.

This step creates a fresh two-VM environment with a controlled known-good baseline. Every later failure starts from the same private network topology.

In this step, get ready to:

• Record whether the older NextWork-Networking and NextWork-LoadBalancer resource groups are present.
• Deploy the complete azuredeploy.json template into a new NextWork-Troubleshooting resource group.
• Verify the private VM topology and confirm that Network Watcher is enabled for the lab region.

Record the older lab state

The isolated lab must remain separate from any resources left by earlier projects. This inventory is read-only so unfamiliar resources stay untouched.

• Return to the Azure portal search bar.
• Enter Resource groups in the search bar.
• Select Resource groups from the results.
• Search the resource-group list for NextWork-Networking.
• Record whether NextWork-Networking is present or absent in your notes.
• Search the resource-group list for NextWork-LoadBalancer.
• Record whether NextWork-LoadBalancer is present or absent in your notes.
• Leave both resource groups unchanged if either one exists.

Why record the older groups?

Their presence does not affect this project because the new lab uses its own resource group. Recording their state keeps your incident notes accurate without risking someone else's resources.

Deploy the ARM template

An Azure Resource Manager template describes the complete lab as one deployment. It gives both VMs the same topology that every later diagnostic expects.

Why use one template?

A single template creates the network as one repeatable unit. This removes manual setup differences from the troubleshooting exercise.

The portal still shows every resource after deployment. You can inspect each network interface, route table, security group, and DNS record during the incident.

📣 Before you deploy

This deployment creates two billable Standard_B1s Linux VMs with managed OS disks. Running VMs incur compute charges.

Managed disks and the private DNS zone can also incur charges. The cleanup section shows you how to stop or delete the lab when you finish.

The template creates no public IP addresses. Every connection in this lab stays on the private virtual network.

• Return to the Azure portal search bar.
• Search for Deploy a custom template.
• Select Deploy a custom template from the results.
• Select Build your own template in the editor.
• Use the tabs below to retrieve the complete azuredeploy.json template:

✔️ Awesome, I've got everything!

Great. Keep the complete template unchanged so the deployed topology matches the troubleshooting checks.

ⓧ I'd like to double check the full code

• Copy this complete template without changing any values:

Copy the template in two parts

• Copy Part 1 into the blank template editor:

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "location": {
      "type": "string",
      "defaultValue": "[resourceGroup().location]",
      "metadata": {
        "description": "Location for the regional lab resources."
      }
    },
    "adminUsername": {
      "type": "string",
      "defaultValue": "nextworkadmin",
      "metadata": {
        "description": "Administrator username for both Linux VMs."
      }
    },
    "adminPassword": {
      "type": "securestring",
      "metadata": {
        "description": "Temporary administrator password for both Linux VMs."
      }
    }
  },
  "variables": {
    "vnetName": "nw-troubleshooting-vnet",
    "sourceSubnetName": "source-subnet",
    "targetSubnetName": "target-subnet",
    "sourceNsgName": "nw-source-nsg",
    "targetNsgName": "nw-target-nsg",
    "routeTableName": "nw-source-rt",
    "sourceNicName": "nw-source-nic",
    "targetNicName": "nw-target-nic",
    "sourceVmName": "nw-source-vm",
    "targetVmName": "nw-target-vm",
    "privateDnsZoneName": "nextwork.internal",
    "dnsLinkName": "nw-vnet-link",
    "sourcePrivateIp": "10.90.1.4",
    "targetPrivateIp": "10.90.2.4",
    "vmSize": "Standard_B1s",
    "imageReference": {
      "publisher": "Canonical",
      "offer": "0001-com-ubuntu-server-jammy",
      "sku": "22_04-lts-gen2",
      "version": "latest"
    }
  },
  "resources": [
    {
      "type": "Microsoft.Network/networkSecurityGroups",
      "apiVersion": "2025-07-01",
      "name": "[variables('sourceNsgName')]",
      "location": "[parameters('location')]",
      "properties": {
        "securityRules": []
      }
    },
    {
      "type": "Microsoft.Network/networkSecurityGroups",
      "apiVersion": "2025-07-01",
      "name": "[variables('targetNsgName')]",
      "location": "[parameters('location')]",
      "properties": {
        "securityRules": []
      }
    },
    {
      "type": "Microsoft.Network/routeTables",
      "apiVersion": "2025-07-01",
      "name": "[variables('routeTableName')]",
      "location": "[parameters('location')]",
      "properties": {
        "disableBgpRoutePropagation": false,
        "routes": []
      }
    },
    {
      "type": "Microsoft.Network/virtualNetworks",
      "apiVersion": "2025-07-01",
      "name": "[variables('vnetName')]",
      "location": "[parameters('location')]",
      "dependsOn": [
        "[resourceId('Microsoft.Network/networkSecurityGroups', variables('sourceNsgName'))]",
        "[resourceId('Microsoft.Network/networkSecurityGroups', variables('targetNsgName'))]",
        "[resourceId('Microsoft.Network/routeTables', variables('routeTableName'))]"
      ],
      "properties": {
        "addressSpace": {
          "addressPrefixes": [
            "10.90.0.0/16"
          ]
        },
        "subnets": [
          {
            "name": "[variables('sourceSubnetName')]",
            "properties": {
              "addressPrefix": "10.90.1.0/24",
              "defaultOutboundAccess": false,
              "networkSecurityGroup": {
                "id": "[resourceId('Microsoft.Network/networkSecurityGroups', variables('sourceNsgName'))]"
              },
              "routeTable": {
                "id": "[resourceId('Microsoft.Network/routeTables', variables('routeTableName'))]"
              }
            }
          },
          {
            "name": "[variables('targetSubnetName')]",
            "properties": {
              "addressPrefix": "10.90.2.0/24",
              "defaultOutboundAccess": false,
              "networkSecurityGroup": {
                "id": "[resourceId('Microsoft.Network/networkSecurityGroups', variables('targetNsgName'))]"
              }
            }
          }
        ]
      }
    },
    {
      "type": "Microsoft.Network/networkInterfaces",
      "apiVersion": "2025-07-01",
      "name": "[variables('sourceNicName')]",
      "location": "[parameters('location')]",
      "dependsOn": [
        "[resourceId('Microsoft.Network/virtualNetworks', variables('vnetName'))]"
      ],
      "properties": {
        "enableAcceleratedNetworking": false,
        "enableIPForwarding": false,
        "ipConfigurations": [
          {
            "name": "ipconfig1",
            "properties": {
              "primary": true,
              "privateIPAddressVersion": "IPv4",
              "privateIPAllocationMethod": "Static",
              "privateIPAddress": "[variables('sourcePrivateIp')]",
              "subnet": {
                "id": "[resourceId('Microsoft.Network/virtualNetworks/subnets', variables('vnetName'), variables('sourceSubnetName'))]"
              }
            }
          }
        ]
      }
    },

• Paste Part 2 on the next line after the final comma in Part 1:

    {
      "type": "Microsoft.Network/networkInterfaces",
      "apiVersion": "2025-07-01",
      "name": "[variables('targetNicName')]",
      "location": "[parameters('location')]",
      "dependsOn": [
        "[resourceId('Microsoft.Network/virtualNetworks', variables('vnetName'))]"
      ],
      "properties": {
        "enableAcceleratedNetworking": false,
        "enableIPForwarding": false,
        "ipConfigurations": [
          {
            "name": "ipconfig1",
            "properties": {
              "primary": true,
              "privateIPAddressVersion": "IPv4",
              "privateIPAllocationMethod": "Static",
              "privateIPAddress": "[variables('targetPrivateIp')]",
              "subnet": {
                "id": "[resourceId('Microsoft.Network/virtualNetworks/subnets', variables('vnetName'), variables('targetSubnetName'))]"
              }
            }
          }
        ]
      }
    },
    {
      "type": "Microsoft.Compute/virtualMachines",
      "apiVersion": "2026-03-01",
      "name": "[variables('sourceVmName')]",
      "location": "[parameters('location')]",
      "dependsOn": [
        "[resourceId('Microsoft.Network/networkInterfaces', variables('sourceNicName'))]"
      ],
      "properties": {
        "hardwareProfile": {
          "vmSize": "[variables('vmSize')]"
        },
        "storageProfile": {
          "imageReference": "[variables('imageReference')]",
          "osDisk": {
            "name": "nw-source-osdisk",
            "caching": "ReadWrite",
            "createOption": "FromImage",
            "deleteOption": "Delete",
            "managedDisk": {
              "storageAccountType": "Standard_LRS"
            }
          }
        },
        "osProfile": {
          "computerName": "[variables('sourceVmName')]",
          "adminUsername": "[parameters('adminUsername')]",
          "adminPassword": "[parameters('adminPassword')]",
          "linuxConfiguration": {
            "disablePasswordAuthentication": false,
            "provisionVMAgent": true
          }
        },
        "networkProfile": {
          "networkInterfaces": [
            {
              "id": "[resourceId('Microsoft.Network/networkInterfaces', variables('sourceNicName'))]",
              "properties": {
                "primary": true,
                "deleteOption": "Delete"
              }
            }
          ]
        },
        "diagnosticsProfile": {
          "bootDiagnostics": {
            "enabled": true
          }
        }
      }
    },
    {
      "type": "Microsoft.Compute/virtualMachines",
      "apiVersion": "2026-03-01",
      "name": "[variables('targetVmName')]",
      "location": "[parameters('location')]",
      "dependsOn": [
        "[resourceId('Microsoft.Network/networkInterfaces', variables('targetNicName'))]"
      ],
      "properties": {
        "hardwareProfile": {
          "vmSize": "[variables('vmSize')]"
        },
        "storageProfile": {
          "imageReference": "[variables('imageReference')]",
          "osDisk": {
            "name": "nw-target-osdisk",
            "caching": "ReadWrite",
            "createOption": "FromImage",
            "deleteOption": "Delete",
            "managedDisk": {
              "storageAccountType": "Standard_LRS"
            }
          }
        },
        "osProfile": {
          "computerName": "[variables('targetVmName')]",
          "adminUsername": "[parameters('adminUsername')]",
          "adminPassword": "[parameters('adminPassword')]",
          "linuxConfiguration": {
            "disablePasswordAuthentication": false,
            "provisionVMAgent": true
          }
        },
        "networkProfile": {
          "networkInterfaces": [
            {
              "id": "[resourceId('Microsoft.Network/networkInterfaces', variables('targetNicName'))]",
              "properties": {
                "primary": true,
                "deleteOption": "Delete"
              }
            }
          ]
        },
        "diagnosticsProfile": {
          "bootDiagnostics": {
            "enabled": true
          }
        }
      }
    },
    {
      "type": "Microsoft.Network/privateDnsZones",
      "apiVersion": "2024-06-01",
      "name": "[variables('privateDnsZoneName')]",
      "location": "global",
      "properties": {}
    },
    {
      "type": "Microsoft.Network/privateDnsZones/virtualNetworkLinks",
      "apiVersion": "2024-06-01",
      "name": "[format('{0}/{1}', variables('privateDnsZoneName'), variables('dnsLinkName'))]",
      "location": "global",
      "dependsOn": [
        "[resourceId('Microsoft.Network/privateDnsZones', variables('privateDnsZoneName'))]",
        "[resourceId('Microsoft.Network/virtualNetworks', variables('vnetName'))]"
      ],
      "properties": {
        "registrationEnabled": false,
        "virtualNetwork": {
          "id": "[resourceId('Microsoft.Network/virtualNetworks', variables('vnetName'))]"
        }
      }
    },
    {
      "type": "Microsoft.Network/privateDnsZones/A",
      "apiVersion": "2024-06-01",
      "name": "[format('{0}/{1}', variables('privateDnsZoneName'), 'target')]",
      "dependsOn": [
        "[resourceId('Microsoft.Network/privateDnsZones', variables('privateDnsZoneName'))]"
      ],
      "properties": {
        "ttl": 30,
        "aRecords": [
          {
            "ipv4Address": "[variables('targetPrivateIp')]"
          }
        ]
      }
    }
  ],
  "outputs": {
    "sourcePrivateIp": {
      "type": "string",
      "value": "[variables('sourcePrivateIp')]"
    },
    "targetPrivateIp": {
      "type": "string",
      "value": "[variables('targetPrivateIp')]"
    },
    "targetFqdn": {
      "type": "string",
      "value": "[format('target.{0}', variables('privateDnsZoneName'))]"
    }
  }
}

💡 What does this template create?

• The Azure Virtual Network uses 10.90.0.0/16 with separate source and target subnets.
• The two private Ubuntu VMs use static private IP addresses and have no public IP addresses.
• The source subnet receives an empty route table for the routing incident later in the project.
• The Private DNS A record maps target.nextwork.internal to 10.90.2.4.

• Replace the blank template in the editor with the complete azuredeploy.json template.
• Select Save.

Azure parses the template and opens a deployment form containing the template parameters.

• Create NextWork-Troubleshooting with the Resource group control.
• Choose a Location where Standard_B1s is available.

The selected resource group and location now isolate every regional lab resource from the older projects.

• Keep nextworkadmin as the adminUsername value.
• Enter your compliant temporary password in the adminPassword field.

The password remains a secure deployment parameter. It does not appear in the template or the deployment outputs.

• Select Review + create.

Azure validates the resource names, template structure, subscription permissions, and regional availability.

• Select Create after validation succeeds.

Provisioning can take several minutes while Azure creates both VMs and their network resources. A changing deployment status during this wait is expected.

• Wait until the portal reports that the deployment completed successfully.

That's the main setup complete. Your isolated network now exists as one reproducible deployment.

Deployment validation failed?

• Choose another region if the selected region cannot provide Standard_B1s capacity.
• Re-enter your temporary password if Azure reports that it does not meet the Linux password requirements.
• Confirm that your subscription permits resource creation if authorization validation fails.

Ask for help: Help me diagnose why my NextWork-Troubleshooting ARM template deployment failed without changing the template topology.

Verify the deployed baseline

A successful deployment status confirms that Azure accepted the template. The baseline check confirms that the actual resources match the private topology required by the later incidents.

Before you check, do you expect both VMs to have private addresses only?

• Return to Resource groups in the Azure portal.
• Open NextWork-Troubleshooting.
• Confirm the resource list includes nw-troubleshooting-vnet, nw-source-vm, nw-target-vm, nw-source-nic, and nw-target-nic.
• Confirm the resource list includes nw-source-nsg, nw-target-nsg, nw-source-rt, and nextwork.internal.

Expected Network Baseline

• The nw-troubleshooting-vnet address space is 10.90.0.0/16.
• The source-subnet prefix is 10.90.1.0/24.
• The source subnet uses nw-source-nsg and nw-source-rt.
• The target-subnet prefix is 10.90.2.0/24.
• The target subnet uses nw-target-nsg.
• The nw-source-rt route table contains no custom routes.

• Open nw-source-vm from the resource group.
• Confirm its status is Running.
• Confirm its size is Standard_B1s.
• Confirm it runs Ubuntu 22.04.
• Open the networking details for nw-source-vm.
• Confirm the attached NIC is nw-source-nic.
• Confirm the static private IP is 10.90.1.4.
• Confirm no public IP is assigned.

The source VM is running entirely on the private source subnet.

• Return to NextWork-Troubleshooting.
• Open nw-target-vm.
• Confirm its status is Running.
• Confirm its size is Standard_B1s.
• Confirm it runs Ubuntu 22.04.
• Open the networking details for nw-target-vm.
• Confirm the attached NIC is nw-target-nic.
• Confirm the static private IP is 10.90.2.4.

• Confirm no public IP is assigned to nw-target-vm.
• Return to NextWork-Troubleshooting.
• Open the nextwork.internal private DNS zone.
• Select Recordsets.
• Confirm the target A record resolves to 10.90.2.4.
• Confirm the A record TTL is 30.
• Confirm the zone contains the nw-vnet-link virtual network link.

The DNS record now gives the target VM a stable private name for the FQDN baseline.

• Return to the Azure portal search bar.
• Search for Network Watcher.
• Select Network Watcher.
• Confirm Network Watcher lists an instance for the lab region.

You now have verified evidence that both private VMs are running with their expected static IP addresses. Network Watcher is ready to test the path between them.

Your isolated lab is ready. Next up, you'll prove that TCP port 22 works by private IP and private FQDN before introducing the first fault.

Prove the Known-Good Path

The isolated lab is live. Both private Azure Virtual Machines are running inside the same controlled network.

A successful baseline gives every later failure a control result. It shows what healthy connectivity looks like before you introduce a fault.

You'll use Azure Network Watcher to test the target by private IP. You'll repeat the test with its private FQDN.

In this step, get ready to:

• Run a TCP port 22 diagnostic from nw-source-vm to nw-target-vm.
• Capture the allowed NSG path plus the virtual-network route evidence.
• Confirm the private FQDN reaches the same target.

Test the private IP path

Connection troubleshoot starts with reachability. Its NSG diagnostic identifies the security decision.

Its Next hop diagnostic identifies the selected route. Together, these views create a layered control result.

• Return to the Azure portal from the previous step.
• Enter Network Watcher in the portal search bar.
• Select Network Watcher from the search results.
• Select Connection troubleshoot under the diagnostic tools.

Why Network Watcher Instead of Guest Commands?

Guest commands can mix operating-system behavior with Azure network behavior. That makes the source of a failure harder to isolate.

Network Watcher keeps this baseline focused on the Azure path. It also avoids logging in to either VM.

• Set the Source type to Virtual machine.
• Select nw-source-vm as the source virtual machine.
• Set the destination type to Select a virtual machine.
• Select nw-target-vm as the destination virtual machine.
• Set the protocol to TCP.
• Enter 22 as the destination port.

The selected endpoints represent a connection from 10.90.1.4 to 10.90.2.4.

• Select Connectivity in the diagnostics list.
• Select NSG diagnostic in the diagnostics list.
• Select Next hop in the diagnostics list.

Before you run the diagnostic, predict whether this untouched private path is reachable.

• Start the private IP check by selecting Run diagnostic tests.

You should see Reachable for the connection from 10.90.1.4 to 10.90.2.4.

The NSG diagnostic shows an allowed path. The Next hop evidence identifies the system route through VirtualNetwork.

Seeing Unreachable?

• Confirm that nw-source-vm still shows Running.
• Confirm that nw-target-vm still shows Running.
• Check that the source is nw-source-vm.
• Check that the destination is nw-target-vm.
• Rerun the diagnostic after correcting the endpoint settings.

Ask for help: Help me diagnose an unexpected Network Watcher result.

Capture the baseline evidence

A baseline is useful when you can compare it with later failures. Your capture should preserve the status plus the diagnostic evidence.

• Expand the Connectivity result.
• Expand the NSG diagnostic result.
• Expand the Next hop result.
• Capture a screenshot that includes Reachable plus the allowed NSG path plus the VirtualNetwork system route.

That's your control result secured. You now have a record of the healthy private IP path.

Test the private FQDN path

A private FQDN makes Azure Private DNS resolve the name before Network Watcher checks the path. The network check then uses that resolved IP.

• Return to the Connection troubleshoot form above the results.
• Keep nw-source-vm as the source virtual machine.
• Set the destination type to Specify manually.
• Enter target.nextwork.internal as the destination.
• Keep the protocol set to TCP.
• Keep the destination port set to 22.

The inputs now mirror the first test except for the destination format.

• Keep Connectivity selected.
• Keep NSG diagnostic selected.
• Keep Next hop selected.

Before you run this check, picture the private address you expect the name to resolve to.

• Start the FQDN check by selecting Run diagnostic tests.

You should see Reachable. The resolved destination should be 10.90.2.4.

• Capture the FQDN result with Reachable plus the resolved destination visible.

You now have both baseline results. The service name reaches the same healthy TCP path as the private IP.

FQDN Still Unreachable?

• Check that the destination is exactly target.nextwork.internal.
• Return to the nextwork.internal Private DNS zone from earlier.
• Confirm that the target A record still maps to 10.90.2.4.
• Leave the NSGs unchanged while the direct-IP baseline remains Reachable.
• Leave the routes unchanged while the direct-IP baseline remains Reachable.

Ask for help: Help me compare my private IP result with my FQDN result.

Your known-good baseline now covers direct private-IP connectivity plus private name resolution. Next, you'll use this control result to isolate a security-rule failure.

Diagnose an NSG Failure

The baseline you captured in Azure Network Watcher proves that TCP port 22 reaches the target by private IP. The FQDN check proves that DNS also resolves correctly.

A network security group can stop that healthy path while routing and DNS remain unchanged. This controlled incident lets you identify filtering from diagnostic evidence.

In this step, get ready to:

• Introduce a targeted inbound NSG deny.
• Trace the blocked packet across three security views.
• Remove the custom rule to restore the known-good path.

Create an inbound deny rule

NSG rules are evaluated by priority. A lower number takes precedence over a higher number.

• Enter nw-target-nsg in the Azure portal search bar.
• Select nw-target-nsg from the search results.
• Select Inbound security rules from the left menu.
• Select + Add above the rules list.

Why Priority 100?

Priority 100 gives this custom rule precedence over rules with higher numbers. The first matching rule decides whether the packet is allowed or denied.

• Set Source to IP Addresses.
• Enter 10.90.1.0/24 as the source address.
• Keep the source port set to *.
• Set Destination to IP Addresses.
• Enter 10.90.2.4 as the destination address.
• Enter 22 as the destination port.

What Traffic Does This Match?

The source prefix covers traffic from the source subnet. The destination values narrow the rule to TCP port 22 on the target VM.

• Set Protocol to TCP.
• Set Action to Deny.
• Enter 100 for Priority.
• Enter Deny-SSH-From-Source for the rule name.
• Select Add to create the rule.

You've set up a clean experiment: Deny-SSH-From-Source is the only new variable in the lab.

• Switch back to Connection troubleshoot in Network Watcher.
• Select nw-source-vm as the source virtual machine.
• Select nw-target-vm as the destination virtual machine.
• Set the protocol to TCP.
• Enter 22 as the destination port.
• Select Connectivity as a diagnostic test.
• Select NSG diagnostic as a diagnostic test.
• Select Next hop as a diagnostic test.

Before you run the diagnostic, consider which result a priority 100 inbound deny should produce.

• Select Run diagnostic tests.

You'll see Unreachable with a NetworkSecurityRule issue. That failure is intentional because it proves the new security rule changed the connection.

Still Seeing Reachable?

• Confirm that Deny-SSH-From-Source has priority 100 with action Deny.
• Confirm that the source is 10.90.1.0/24.
• Confirm that the destination is 10.90.2.4 on port 22.

Ask for help: My Connection troubleshoot result is still Reachable after I added Deny-SSH-From-Source. Can you help me check the NSG rule values?

Trace the deny to one rule

Connection troubleshoot identifies the failing layer. IP flow verify evaluates one packet against the rules applied to the target VM.

• Switch back to Network Watcher from earlier.
• Select IP flow verify under the diagnostic tools.
• Set Target resource to Virtual machine.
• Select nw-target-vm as the virtual machine.
• Select nw-target-nic as the network interface.

The packet details now need to describe the same inbound SSH flow that Connection troubleshoot tested.

• Set Protocol to TCP.
• Set Direction to Inbound.
• Enter 10.90.2.4 as the local IP address.
• Enter 22 as the local port.
• Enter 10.90.1.4 as the remote IP address.
• Enter 60000 as the remote port.
• Select Verify IP flow.

You'll see Access denied with Deny-SSH-From-Source named as the matching rule. That's the rule-level proof connecting the failed packet to the NSG change.

Seeing Access Allowed?

• Confirm that the packet direction is Inbound.
• Confirm that the local endpoint is 10.90.2.4 on port 22.
• Confirm that Deny-SSH-From-Source still appears in nw-target-nsg.

Ask for help: IP flow verify returns Access allowed for my inbound TCP test. Can you help me compare the packet details with Deny-SSH-From-Source?

The effective security rules view aggregates the rules applied through the target subnet or network interface. It shows where the matching deny sits in the target VM's complete security policy.

• Select Effective security rules under Network Watcher diagnostic tools.
• Select your Azure subscription.
• Select NextWork-Troubleshooting as the resource group.
• Select nw-target-vm as the virtual machine.
• Confirm that the network interface is nw-target-nic.
• Locate Deny-SSH-From-Source among the aggregated inbound rules.

You now have three connected pieces of evidence. Connection troubleshoot identifies the security layer. IP flow verify names the matching rule. Effective security rules confirms that the rule applies to the target interface.

Delete the rule and prove recovery

Deleting this custom rule affects only the isolated lab rule you created. The default rules remain in place.

• Switch back to nw-target-nsg from earlier.
• Select Inbound security rules.
• Select Deny-SSH-From-Source.
• Select Delete.
• Select Yes to confirm.
• Wait until Deny-SSH-From-Source no longer appears in the inbound rules list.

The temporary deny is gone. The final diagnostic checks whether that single change restores the original path.

• Switch back to Connection troubleshoot from earlier.
• Select nw-source-vm as the source virtual machine.
• Select nw-target-vm as the destination virtual machine.
• Set the protocol to TCP.
• Enter 22 as the destination port.
• Select Connectivity as a diagnostic test.
• Select NSG diagnostic as a diagnostic test.
• Select Next hop as a diagnostic test.

Before you rerun the test, predict whether removing the only custom deny restores the baseline.

• Select Run diagnostic tests.

You'll see Reachable from nw-source-vm to nw-target-vm on TCP port 22. You proved the incident was caused by one NSG rule and restored the lab to its known-good state.

Still Seeing Unreachable?

• Confirm that Deny-SSH-From-Source is absent from nw-target-nsg.
• Rerun the diagnostic after the deleted rule disappears from the inbound rules list.
• Confirm that the selected destination is nw-target-vm on port 22.

Ask for help: Connection troubleshoot is still Unreachable after I deleted Deny-SSH-From-Source. Can you help me verify the restored NSG path?

The lab is back at its known-good baseline. Next, you'll create a routing fault that produces a different diagnostic signature.

Diagnose a Blackhole Route

Your earlier Connection troubleshoot result proved that an inbound Azure network security group could stop TCP 22 while the route stayed healthy. Removing Deny-SSH-From-Source returned the connection to Reachable.

This incident examines a different layer. A destination-specific user-defined route gives you a controlled way to inspect Azure's forwarding decision.

In this step, get ready to:

• Trigger a controlled routing incident in nw-source-rt.
• Correlate Next hop with Effective routes.
• Close the incident with a verified recovery.

Create the routing incident

A route scoped to one destination keeps this experiment focused on the target VM. You will change only the route table associated with the source subnet.

• Use the Azure portal search bar to find nw-source-rt.
• Open nw-source-rt from the search results.
• Select Routes from the left menu.
• Select + Add.
• Enter Blackhole-Target for Route name.
• Enter 10.90.2.4/32 for Address prefix.
• Select None for Next hop type.
• Select OK.

The route table now lists Blackhole-Target with destination prefix 10.90.2.4/32.

Before the diagnostic, consider whether the baseline TCP 22 path stays reachable with the new route in place.

• Return to Network Watcher from earlier.
• Select Connection troubleshoot.

The baseline uses nw-source-vm as the source. Its destination is 10.90.2.4.

• Select nw-source-vm as the source virtual machine.
• Select Specify manually as the destination type.
• Enter 10.90.2.4 as the destination.
• Select TCP as the protocol.
• Enter 22 as the destination port.
• Keep the previous diagnostic selections.
• Select Run diagnostic tests.

You will see Unreachable. The issue evidence points to UserDefinedRoute instead of NetworkSecurityRule.

Why Is This Result Different?

The Blackhole-Target route matches 10.90.2.4. Its next hop is None.

Azure drops traffic assigned to None. The packet never reaches nw-target-vm.

🙋‍♀️ Still Seeing Reachable?

• Confirm that Blackhole-Target appears under Routes in nw-source-rt.
• Check that the destination prefix is exactly 10.90.2.4/32.
• Check that the next hop type is None.

Ask for help: Help me find why Connection troubleshoot still reports Reachable after I added Blackhole-Target.

Trace the route with diagnostic evidence

Connection troubleshoot identifies the failing layer. Next hop identifies the forwarding action selected for one packet.

The Effective routes view exposes the route entry behind that action.

• Select Next hop under Diagnostic tools in Network Watcher.
• Select the subscription containing NextWork-Troubleshooting.
• Select NextWork-Troubleshooting for Resource group.
• Select nw-source-vm for Virtual machine.
• Select nw-source-nic for Network interface.
• Enter 10.90.1.4 for Source IP address.
• Enter 10.90.2.4 for Destination IP address.
• Run the Next hop check.

You will see a next hop type of None. The route ID points to nw-source-rt.

• Return to nw-source-vm in NextWork-Troubleshooting.
• Select Networking.
• Select nw-source-nic.
• Select Effective routes.
• Locate the active 10.90.2.4/32 user-defined route.

You will find the active route for 10.90.2.4/32 associated with nw-source-rt.

How the Evidence Fits Together

Connection troubleshoot reports that delivery failed at the routing layer. Next hop names the forwarding action as None.

Effective routes shows 10.90.2.4/32 among the source NIC's active routes. These views form one chain of evidence back to Blackhole-Target.

Remove the faulty route

A diagnosis is complete when removing the suspected cause restores the forwarding decision. The connection test must also return to its baseline.

• Return to nw-source-rt.
• Select Routes.
• Find Blackhole-Target.
• Select the ellipsis for Blackhole-Target.
• Select Delete.
• Select Yes.
• Wait until Blackhole-Target no longer appears under Routes.

Before rerunning Next hop, predict which forwarding path Azure selects now that the custom route is gone.

• Return to the Next hop check from earlier.
• Keep the previous packet details unchanged.
• Run the Next hop check again.

You will see the VirtualNetwork next hop again. This is the virtual-network system path.

Before the final diagnostic, predict whether TCP 22 now follows the restored path successfully.

• Return to Connection troubleshoot from earlier.
• Keep the previous baseline configuration unchanged.
• Select Run diagnostic tests.

You will see Reachable from nw-source-vm to 10.90.2.4 on TCP port 22.

That closes the routing incident. You identified the winning route before removing the single faulty change.

Still Seeing None or Unreachable?

• Confirm that Blackhole-Target no longer appears under Routes in nw-source-rt.
• Verify that the final Connection troubleshoot test targets 10.90.2.4 on TCP port 22.
• Rerun the checks after Azure finishes applying the route deletion.

Ask for help: Help me diagnose why Next hop or Connection troubleshoot has not returned to the known-good result.

Your lab is back on its known-good virtual-network path. Next, you will isolate a name-resolution failure while direct-IP connectivity stays healthy.

Diagnose a DNS Failure

Your route investigation restored the direct TCP 22 path to 10.90.2.4. Now you can use that healthy IP result to isolate a failure in DNS name resolution.

Removing the target A record breaks the FQDN lookup. The destination IP path remains reachable, which separates name resolution from packet delivery.

You'll use Azure Network Watcher to compare both paths. Then you'll recover connectivity by changing only the layer supported by your evidence.

In this step, get ready to:

• Delete the target A record to create a controlled DNS failure.
• Compare the FQDN result with the direct-IP control.
• Recover FQDN connectivity by changing only the faulty layer.

Delete the target A record

Azure Private DNS maps target.nextwork.internal to the target VM's private address. Deleting that one mapping creates a controlled incident without changing the healthy security or routing configuration.

• Use the Azure portal search bar to search for nextwork.internal.
• Select the nextwork.internal Private DNS zone from the results.
• Select Recordsets from the zone menu.
• Select the trash icon beside the target record set.
• Verify that the record-set name shows target.
• Select Yes to delete the record set.
• Confirm that target no longer appears under Recordsets.

The missing record gives you a controlled test case. Connection troubleshoot can now show which diagnostic layer reacts to that change.

• Return to Network Watcher from earlier.
• Select Connection troubleshoot.
• Choose nw-source-vm as the source virtual machine.
• Set the destination type to Specify manually.
• Enter target.nextwork.internal as the destination.
• Select TCP as the protocol.
• Enter 22 in Destination port.

Before you run the test, which diagnostic layer do you expect Network Watcher to flag?

• Select Run diagnostic tests.

You'll see Unreachable with a DNSResolution issue. This shortfall is intentional because the FQDN no longer has an A record.

Not Seeing DNSResolution?

• Return to Recordsets to confirm that target is absent.
• Check that the manual destination is exactly target.nextwork.internal.
• Rerun the test with protocol TCP and destination port 22.

Ask for help: Why is my FQDN test showing a different result?

Compare the FQDN and direct-IP paths

A direct-IP test bypasses the missing name-to-address mapping. Its result tells you whether the underlying TCP path still works.

• Keep nw-source-vm as the source virtual machine.
• Replace the destination with 10.90.2.4.
• Keep TCP as the protocol.
• Keep 22 as the destination port.

Before you rerun the diagnostic, do you expect the direct-IP result to match the failed FQDN result?

• Select Run diagnostic tests.

You'll see Reachable for 10.90.2.4 on TCP port 22. The packet path still works without DNS.

How Does the Control Isolate DNS?

The direct-IP result uses the same source VM. It also uses the same destination VM and TCP port.

That comparison keeps NSG filtering and routing constant. The failed FQDN test isolates the changed layer to name resolution.

That comparison pays off: your direct-IP control proves the network path remains healthy. You now have evidence for a DNS-only fix.

Restore the target A record

The paired tests point to one missing name-to-address mapping. Recreating that mapping is the smallest evidence-backed fix.

• Return to the nextwork.internal Private DNS zone.
• Select Recordsets.
• Select + Add.
• Enter target in the Name field.
• Select A as the record Type.
• Enter 30 in the TTL field.
• Enter 10.90.2.4 as the IP address.
• Select OK to create the record set.

You'll see target listed with TTL 30 and IP address 10.90.2.4.

The restored record needs one final FQDN test. This check proves that changing only DNS recovers the original connection.

• Return to Connection troubleshoot in Network Watcher.
• Choose nw-source-vm as the source virtual machine.
• Set the destination type to Specify manually.
• Enter target.nextwork.internal as the destination.
• Select TCP as the protocol.
• Enter 22 in Destination port.

Before you run the final check, do you expect the restored FQDN path to match the healthy direct-IP control?

• Select Run diagnostic tests.

You'll see Reachable for target.nextwork.internal on TCP port 22. The resolved destination now matches 10.90.2.4 again.

Still Seeing DNSResolution?

Resolvers can briefly retain the earlier negative answer after the record is recreated. That cached response can outlive the portal change for a short time.

• Wait briefly before running the same FQDN diagnostic again.
• Leave the healthy NSG configuration unchanged.
• Leave the healthy route table unchanged.

Ask for help: Why does my restored DNS record still show a DNSResolution issue?

You now have both direct-IP and FQDN checks returning Reachable on TCP port 22. Your evidence shows that the earlier DNSResolution failure existed only while the A record was absent.

Secret mission

Triage a Wrong-Answer DNS Incident

A service name resolves to an unused private IP, but the faulty layer is still yours to prove. Trace the incident through connectivity, security, routing, and DNS evidence before changing anything else. Restore service with one evidence-backed correction.

Clean Up Your Resources

Clean Up Your Resources

Your Microsoft Azure lab can remain running, pause for later, or be deleted entirely. Running virtual machines, managed disks, DNS resources, and diagnostic checks can affect your Azure bill.

Cost warning

Both nw-source-vm and nw-target-vm are currently Running. Azure bills their instance usage in this state.

Managed disks can continue to incur charges after deallocation. Azure DNS bills hosted zones and queries.

Network Diagnostic Tools include 1,000 checks per month. Later checks use per-check billing.

Resources you used:

• Resource group NextWork-Troubleshooting containing the isolated lab.
• Two Standard_B1s Linux virtual machines named nw-source-vm and nw-target-vm.
• Two managed OS disks named nw-source-osdisk and nw-target-osdisk.
• Two network interfaces named nw-source-nic and nw-target-nic.
• An Azure Virtual Network named nw-troubleshooting-vnet with source-subnet and target-subnet.
• Two network security groups named nw-source-nsg and nw-target-nsg.
• Route table nw-source-rt associated with source-subnet.
• An Azure Private DNS zone named nextwork.internal with virtual network link nw-vnet-link.
• Private DNS A record target.nextwork.internal resolving to 10.90.2.4.

Keep everything running

No action needed. Choose this if you want to repeat the incidents or collect more diagnostic evidence.

• Both virtual machines remain Running. Azure continues billing their instance usage.
• The managed OS disks remain available for both virtual machines. Disk charges can continue.
• The Private DNS zone remains hosted. Azure DNS charges for hosted zones and received queries continue.
• Your known-good direct-IP and FQDN paths remain ready for another troubleshooting run.
• Further Network Diagnostic Tools checks count toward the monthly allowance of 1,000 checks.

Pause - I'll come back to this later

Pausing deallocates both virtual machines while preserving the rest of the lab. This stops VM instance billing while keeping your troubleshooting environment available.

• Search for Resource groups in the Azure portal search bar.
• Open NextWork-Troubleshooting.
• Select nw-source-vm.
• Select Stop.
• Confirm the stop request.
• Return to NextWork-Troubleshooting.
• Select nw-target-vm.
• Select Stop.
• Confirm the stop request.
• Wait until both VM statuses read Stopped (Deallocated).

Both virtual machines are now deallocated, so VM instance billing has stopped. Managed disks and Private DNS resources can still incur charges.

Delete - I don't want to use this again

Deleting the resource group is irreversible. This option removes only the isolated lab inside NextWork-Troubleshooting.

• Search for Resource groups in the Azure portal search bar.
• Open NextWork-Troubleshooting.
• Select Delete resource group.
• Type NextWork-Troubleshooting when Azure asks for the resource-group name.
• Confirm the deletion.

Azure may spend several minutes removing the virtual machines, disks, networking resources, and Private DNS resources.

• Wait until Azure reports that the deletion has finished.
• Return to Resource groups.
• Search for NextWork-Troubleshooting.

You should no longer see NextWork-Troubleshooting in the results. That absence confirms the complete isolated lab has been deleted.

Nice Work!

Nice Work!

You turned a private two-VM Azure lab into a repeatable incident workflow with Network Watcher. Every failure left diagnostic evidence that pointed to one layer.

You learned how to:

• Built an isolated troubleshooting lab with two private Azure Virtual Machines. Established a known-good TCP port 22 baseline by private IP. Confirmed the same path by private FQDN.
• Used Connection troubleshoot to expose an NSG denial. Confirmed the matching rule with IP flow verify. Located the same deny in effective security rules. Proved a blackhole user-defined route with Next hop. Located the winning route in effective routes.
• Separated a Private DNS failure from healthy packet delivery by retesting the private IP. Restored the target A record. Verified target.nextwork.internal returned Reachable again.
• Secret Mission: Traced a wrong-answer DNS incident through the full layered workflow. Identified 10.90.2.99 as the incorrect resolution. Corrected only the faulty A record. Confirmed both paths returned Reachable.

Ready to quiz yourself?