Azure Monitoring Platform with Grafana

Deploy a centralized Azure monitoring platform with self-hosted Grafana.

Introduction

30 Second Summary

When something goes wrong in a live system, the first question is always "what happened?" Without a monitoring platform watching your servers, containers, and network traffic in real time, you are flying blind.

In this project, you will build a centralized monitoring and observability platform in Microsoft Azure. You will deploy Linux VMs, a NAT Gateway, container apps, and wire everything into a self-hosted Grafana dashboard with automated email alerting.

What You'll Build

A production-style Azure monitoring platform with 4 Linux VMs, a NAT Gateway, 3 Container Apps, and a self-hosted Grafana instance providing live operational dashboards and threshold-based alerting via Gmail.

By the end of this project, you'll have:

4 Linux VMs with Azure Monitor Agent collecting CPU, Memory, Disk, and Swap metrics into a centralized Log Analytics Workspace.
A NAT Gateway for outbound traffic monitoring and 3 Azure Container Apps with full diagnostic logging, live dashboards, and error tracking.
Self-hosted Grafana OSS behind Apache2 with 6 live dashboards and Gmail SMTP alerting for threshold breaches, container crashes, and NAT Gateway transit alerts.
Secret Mission: Add a Grafana annotation system that marks deployment events on your dashboards.

Are there any prerequisites?

You need an active Azure subscription with Owner or Contributor role. You also need a Gmail account with 2-Factor Authentication enabled so you can generate an App Password for Grafana's SMTP alerting.

Create the Foundation Infrastructure

Every monitoring platform needs a foundation. Before you can collect metrics, stream logs, or fire alerts, you need a central place to store telemetry data and a network that connects your monitored resources.

In this step, you'll create the three building blocks that everything else in this project depends on: a Resource Group to organize your resources, a Virtual Network to connect your VMs, and a Log Analytics Workspace to store all your monitoring data in one place.

In this step, get ready to:

Create a Resource Group to hold all your monitoring resources.
Create a Virtual Network with two subnets for VMs and Grafana.
Create a Log Analytics Workspace as your central telemetry store.

Sign in to the Azure Portal

The Azure Portal is the web-based interface where you'll create and manage all your cloud resources.

Open your browser and navigate to portal.azure.com.
Sign in with the Microsoft account that has your Azure subscription.

You should see the Azure Portal home page with the search bar at the top and a list of Azure services.

Create the Resource Group

A Resource Group is a logical container that holds related Azure resources. When you're done with this project, you can delete the entire group to remove everything at once.

In the search bar at the top of the Azure Portal, type Resource groups and select it from the results.
Click Create.
Under Subscription, select your active Azure subscription.
In the Resource group field, enter rg-monitoring-lab.
For Region, select a region close to you (for example, East US or West Europe). Remember your choice because all resources must go in the same region.
Click Review + create.
Click Create.

You should see a Deployment succeeded notification. Your resource group is now ready.

Why a single Resource Group?

Placing all resources in one group makes cleanup simple. At the end of this project, one delete command removes everything. It also makes it easy to assign permissions at the group level, which you'll use later for Grafana's Managed Identity.

Don't see the Create button?

Make sure your account has Owner or Contributor role on the subscription. You can check this under Subscriptions > your subscription > Access control (IAM).

Help me troubleshoot my Azure permissions

Create the Virtual Network with two subnets

A Virtual Network (VNet) provides private connectivity between your Azure VMs. You'll split it into two subnets: one for the monitored VMs and one for the Grafana server.

In the search bar, type Virtual networks and select it from the results.
Click Create.
Under Resource group, select rg-monitoring-lab.
In the Virtual network name field, enter vnet-monitoring.
For Region, select the same region you chose for your Resource Group.
Click Next to move to the Security tab. Leave the defaults and click Next again to reach the IP Addresses tab.

Now you'll configure the address space and subnets.

Set the IPv4 address space to 10.0.0.0/16. If a default subnet already exists, delete it by clicking the trash icon next to it.
Click Add a subnet.
Enter subnet-vms as the Subnet name.
Set the Starting address to 10.0.1.0 with a size of /24 (256 addresses).
Click Add.
Click Add a subnet again.
Enter subnet-grafana as the Subnet name.
Set the Starting address to 10.0.2.0 with a size of /24.
Click Add.
Click Review + create, then click Create.

Wait for the deployment to complete. You should see Your deployment is complete with both subnets listed.

Why two separate subnets?

Separating the monitored VMs from the Grafana server gives you better network isolation and security control. In later steps, you'll attach a NAT Gateway to only the VM subnet for centralized outbound traffic routing. The Grafana subnet gets its own public IP so you can access the dashboards directly.

Seeing an address overlap error?

Make sure you deleted the default subnet before adding your own. Two subnets cannot overlap in address ranges. If the default subnet uses 10.0.0.0/24, it won't conflict with your subnets, but it's cleaner to remove it.

Help me fix my VNet address space configuration

Create the Log Analytics Workspace

A Log Analytics Workspace is where all your monitoring data lands. Every VM metric, container log, and NAT Gateway measurement in this project will flow into this single workspace.

In the search bar, type Log Analytics workspaces and select it from the results.
Click Create.
Under Resource group, select rg-monitoring-lab.
In the Name field, enter law-monitoring-central.
For Region, select the same region as your Resource Group.
Click Review + Create, then click Create.

Wait for the deployment to complete. This usually takes about 30 seconds.

Once deployed, click Go to resource.
In the left sidebar, click Usage and estimated costs.
Click Data Retention.
Set the retention slider to 30 days.
Click OK to save.

What is data retention?

Retention controls how long Azure keeps your log data before automatically deleting it. 30 days is enough for this project and keeps ingestion costs low. In a later step, you'll set up Blob Storage archival for longer-term storage.

Name already taken?

Log Analytics Workspace names must be globally unique. If law-monitoring-central is taken, add your initials or a number to the end (for example, law-monitoring-central-jd01). Use this name wherever the project references law-monitoring-central.

Help me resolve a naming conflict for my Log Analytics Workspace

Your foundation is in place. You have a resource group, a network with two isolated subnets, and a central workspace ready to receive telemetry. Next up, you'll deploy a NAT Gateway to control and monitor outbound traffic from your VM subnet.

Deploy NAT Gateway

Your foundation is in place. You have a resource group, a virtual network with two subnets, and a Log Analytics Workspace ready to receive telemetry.

In a production environment, you want outbound internet traffic from your VMs to flow through a single, controlled exit point. This is exactly what an Azure NAT Gateway provides. By routing all outbound traffic through one public IP, you gain visibility into how much data leaves your network and can detect anomalies like unexpected data exfiltration.

In this step, get ready to:

Create a Static Public IP Address for your NAT Gateway.
Create the NAT Gateway and associate it with your Public IP.
Attach the NAT Gateway to your VM subnet for centralized outbound routing.

Create a Public IP Address

A NAT Gateway needs a public IP address to represent all outbound traffic from your private subnet. You will create a static IP so the address never changes.

In the Azure Portal search bar at the top, type Public IP addresses and select it from the results.
Click Create.
Set Subscription to your active Azure subscription.
Set Resource group to rg-monitoring-lab.
Set Region to the same region you used for your resource group.
Enter pip-nat-gateway in the Name field.
Set SKU to Standard.
Set Assignment to Static.
Click Review + create, then click Create.

Wait for the deployment to complete. You should see a Your deployment is complete message.

Why Standard SKU and Static?

NAT Gateways require a Standard SKU public IP. The Static allocation ensures the IP address remains the same even if you deallocate and restart the gateway.

This is important for monitoring because your dashboards will track traffic flowing through a consistent, known IP.

Create the NAT Gateway

Now you will create the NAT Gateway itself and link it to the public IP you just made.

In the Azure Portal search bar, type NAT gateways and select it from the results.
Click Create.
Set Resource group to rg-monitoring-lab.
Enter natgw-monitoring in the NAT gateway name field.
Set Region to the same region as your resource group.
Leave Availability zone at the default.
Leave TCP idle timeout at the default of 4 minutes.
Click Next: Outbound IP.

On the Outbound IP tab, you will associate the public IP you created a moment ago.

Under Public IP addresses, click the dropdown and select pip-nat-gateway.
Click Next: Subnet.

Associate the NAT Gateway with subnet-vms

The Subnet tab is where you tell Azure which subnet should route its outbound traffic through this NAT Gateway. You will attach it to subnet-vms so that all three monitored VMs you deploy later will automatically use this gateway for internet-bound traffic.

Set Virtual network to vnet-monitoring.
Under Subnet name, check the box next to subnet-vms.
Click Review + create.
Review your settings, then click Create.

Wait for the deployment to complete. You should see Your deployment is complete with a green checkmark.

What does the NAT Gateway do for monitoring?

Without a NAT Gateway, each VM would need its own public IP for outbound internet access. Traffic would scatter across multiple IPs, making it hard to track total data leaving your network.

With the NAT Gateway, all outbound traffic funnels through a single IP (pip-nat-gateway). Later, you will build a dashboard tracking bytes in/out, SNAT connections, and dropped packets. All of that data comes from this one resource.

Click Go to resource to open your new NAT Gateway.
In the Overview pane, confirm you see pip-nat-gateway listed under Public IP addresses.
Click Subnets in the left menu and confirm subnet-vms is listed as an associated subnet.

Don't see subnet-vms associated?

Make sure you selected vnet-monitoring as the virtual network during creation. If you accidentally picked a different VNet, the subnet won't appear.

You can fix this without recreating the gateway. Click Subnets in the left menu of the NAT Gateway, then click Associate to add subnet-vms from vnet-monitoring.

Help me troubleshoot my NAT Gateway subnet association in Azure

Your network plumbing is ready. All VMs you deploy into subnet-vms will now route their outbound internet traffic through natgw-monitoring. Next up, you will deploy the three monitored VMs into that subnet.

Deploy the Monitored VMs

Your network backbone is in place. The NAT Gateway is ready to route outbound traffic from subnet-vms, so any VM you place in that subnet gets internet access without needing its own public IP.

Now you will deploy the three workload VMs that represent a typical production environment: a web server, an application server, and a database server. These are the machines your monitoring platform will watch.

In this step, get ready to:

Create 3 Ubuntu 22.04 LTS VMs in subnet-vms with no public IPs.
Verify all 3 VMs are running in the Azure Portal.
Enable System-Assigned Managed Identity on each VM.

Create web-server-01

You will create the first of three VMs. All three share the same configuration, so you will walk through the full setup once, then repeat for the other two.

In the Azure Portal search bar, type Virtual machines and select Virtual machines from the results.
Click + Create, then select Azure virtual machine.

You are now on the Create a virtual machine form. Configure the Basics tab:

Set Resource group to rg-monitoring-lab.
Set Virtual machine name to web-server-01.
Set Region to the same region you used for rg-monitoring-lab.
Set Image to Ubuntu Server 22.04 LTS.
Set Size to Standard_B1s. Click See all sizes if it is not listed in the dropdown.
Under Administrator account, set Authentication type to SSH public key.
Enter a Username (the default azureuser works fine).
Set SSH public key source to Generate new key pair.
Set Key pair name to web-server-01_key.
Under Inbound port rules, select Allow selected ports.
Set Select inbound ports to SSH (22).

Now switch to the Networking tab:

Set Virtual network to vnet-monitoring.
Set Subnet to subnet-vms (10.0.1.0/24).
Set Public IP to None.

Why no public IP?

These VMs represent internal workloads that should not be directly reachable from the internet. They get outbound internet access through the NAT Gateway you associated with subnet-vms in the previous step.

This mirrors production environments where backend servers stay private and route outbound traffic through a centralized gateway.

Click Review + create.
Review the summary, then click Create.
A dialog will prompt you to download the SSH private key. Click Download private key and create resource.
Save the downloaded .pem file somewhere safe. You will need it later for SSH access.

Wait for the deployment to complete. You should see Your deployment is complete on the confirmation page.

Create app-server-01 and db-server-01

The remaining two VMs use the exact same configuration as web-server-01. Repeat the creation process for each one with these name changes.

For the second VM:

Navigate back to Virtual machines and click + Create > Azure virtual machine.
Set Virtual machine name to app-server-01.
Set Key pair name to app-server-01_key.
Use the same settings for everything else: Ubuntu Server 22.04 LTS, Standard_B1s, SSH authentication, subnet-vms, no public IP, inbound SSH allowed.
Click Review + create, then Create. Download and save the private key.

For the third VM:

Navigate back to Virtual machines and click + Create > Azure virtual machine.
Set Virtual machine name to db-server-01.
Set Key pair name to db-server-01_key.
Use the same settings for everything else: Ubuntu Server 22.04 LTS, Standard_B1s, SSH authentication, subnet-vms, no public IP, inbound SSH allowed.
Click Review + create, then Create. Download and save the private key.

Keep all three .pem files together

Create a folder on your local machine (e.g. monitoring-lab-keys) and store all three .pem files there. You will need them when you SSH into the VMs through the Grafana server later.

Verify all 3 VMs are running

In the Azure Portal search bar, type Virtual machines and select Virtual machines.
Confirm that web-server-01, app-server-01, and db-server-01 all show a Status of Running.
Click on web-server-01 to open its overview page. Verify that Virtual network/subnet shows vnet-monitoring/subnet-vms and Public IP address shows a dash (no IP assigned).

VM stuck in Creating or Failed state?

If a VM stays in Creating for more than 5 minutes, check your subscription quota for the Standard_B1s size in your chosen region. Some regions have limited B-series availability for free-tier or trial subscriptions.

If you see a quota error, try a different region or request a quota increase from the Subscriptions > Usage + quotas page in the portal.

help me troubleshoot my Azure VM deployment failure

Enable System-Assigned Managed Identity on each VM

A Managed Identity gives your VM an Azure AD identity without storing any credentials on the machine. The Azure Monitor Agent will use this identity to authenticate when sending telemetry to your Log Analytics Workspace.

From the Virtual machines list, click on web-server-01.
In the left menu, scroll down to the Security section and click Identity.
On the System assigned tab, set the Status toggle to On.
Click Save.
A confirmation dialog appears asking if you want to enable the system assigned managed identity. Click Yes.

After a few seconds, an Object ID appears on the page. This confirms the identity is active.

Repeat the same process for app-server-01: navigate to Security > Identity > set Status to On > Save > Yes.
Repeat the same process for db-server-01: navigate to Security > Identity > set Status to On > Save > Yes.

Why enable Managed Identity now?

The Azure Monitor Agent (which you will install in a later step) requires a Managed Identity to authenticate with Azure. Enabling it now means the agent can start sending data the moment it is installed.

Your three monitored workloads are deployed and identity-ready. Next, you will create the Grafana server VM that will become your central observability dashboard.

Deploy the Grafana Server VM

Your three monitored VMs are running in subnet-vms, sending all outbound traffic through the NAT Gateway. Now you need a place to actually see all that telemetry data.

In this step, you will deploy a dedicated Grafana server VM. This VM lives in its own subnet, has a public IP so you can access its dashboards from your browser, and uses Managed Identity to securely read monitoring data without storing any secrets.

In this step, get ready to:

Create the grafana-server VM in subnet-grafana with a public IP and open ports for SSH and HTTP.
Enable System-Assigned Managed Identity on the VM.
Assign Reader and Log Analytics Reader roles so Grafana can query your monitoring data.

Create the grafana-server VM

Unlike the three monitored VMs (which have no public IPs), the Grafana server needs to be directly accessible from your browser. You will place it in subnet-grafana and give it a public IP with inbound rules for both SSH and HTTP traffic.

In the Azure Portal, type Virtual machines in the top search bar and select Virtual machines from the results.
Click + Create and select Azure virtual machine.

On the Basics tab, configure the following settings:

Set Resource group to rg-monitoring-lab.
Set Virtual machine name to grafana-server.
Set Region to the same region you used for your other resources.
Set Image to Ubuntu Server 22.04 LTS.
Set Size to Standard_B2s.
Set Authentication type to SSH public key.
Enter a Username (e.g. azureuser).
Under Inbound port rules, select Allow selected ports.
Set Select inbound ports to SSH (22) and HTTP (80).

Why Standard_B2s instead of B1s?

Grafana needs more memory than the lightweight monitored VMs. Standard_B2s provides 2 vCPUs and 4 GB RAM, which is enough to run Grafana plus the Apache2 reverse proxy you will install later.

Now configure the networking to place this VM in the correct subnet with a public IP:

Click the Networking tab at the top of the page.
Set Virtual network to vnet-monitoring.
Set Subnet to subnet-grafana (10.0.2.0/24).
Confirm that Public IP is set to create a new public IP (this is the default).

Click Review + create.
Review the summary and click Create.

Azure will prompt you to download the SSH private key file. Save it somewhere safe on your computer. You will need it to connect to the VM later.

Wait for the deployment to complete, then click Go to resource.

You should see the grafana-server overview page showing its status as Running, with a public IP address listed.

Don't see Standard_B2s in the size dropdown?

Not all VM sizes are available in every region. If B2s is unavailable, try selecting See all sizes and filtering for B2s. If it is still missing, select a different region that supports it, or use Standard_B2ms as an alternative.

Help me find an available VM size for my Grafana server.

Enable System-Assigned Managed Identity

A Managed Identity gives your VM an Azure AD identity without needing to store any credentials. Grafana will use this identity to authenticate with Azure Monitor and read your telemetry data.

On the grafana-server overview page, click Identity in the left sidebar under Settings.
On the System assigned tab, toggle Status to On.
Click Save.
A confirmation dialog appears. Click Yes to enable the system-assigned managed identity.

After a few seconds, an Object (principal) ID appears on the page. This confirms the identity is active.

Why Managed Identity instead of storing credentials?

The alternative is to create an App Registration with a client secret and paste that secret into Grafana's configuration. Managed Identity eliminates that entirely. Azure handles authentication behind the scenes, so there is no secret to rotate, leak, or expire.

Assign Reader and Log Analytics Reader roles

The identity exists, but it has no permissions yet. You need to give it two roles so Grafana can read resource metadata and query log data from your Log Analytics workspace.

In the top search bar, type Resource groups and select Resource groups.
Click rg-monitoring-lab.
Click Access control (IAM) in the left sidebar.
Click + Add and select Add role assignment.

First, assign the Reader role:

On the Role tab, search for Reader and select the Reader role (not any variation like "Reader and Data Access").
Click Next.
On the Members tab, select Managed identity under Assign access to.
Click + Select members.
In the Managed identity dropdown, select Virtual machine.
Select grafana-server from the list and click Select.
Click Review + assign twice to complete the assignment.

Now repeat the process to assign the Log Analytics Reader role:

Click + Add and select Add role assignment again.
Search for Log Analytics Reader and select it.
Click Next.
Select Managed identity, click + Select members, choose Virtual machine, and select grafana-server.
Click Review + assign twice to complete.

What do these two roles do?

Reader lets Grafana see all the resources in your resource group (VMs, NAT Gateway, etc.) and their metrics. Log Analytics Reader lets Grafana run KQL queries against the data stored in your Log Analytics Workspace.

Together, they give Grafana full read-only access to your monitoring data. No write permissions are granted.

To verify the role assignments were applied, stay on the Access control (IAM) page and click the Role assignments tab. You should see grafana-server listed twice, once under Reader and once under Log Analytics Reader.

Don't see grafana-server in the members list?

Make sure you enabled the System-Assigned Managed Identity in the previous substep. Go back to the grafana-server VM, check Identity under Settings, and confirm the status shows On with an Object ID displayed.

Help me troubleshoot why my grafana-server Managed Identity isn't appearing in the role assignment member picker.

Your Grafana server VM is ready. It has a public IP for browser access, a managed identity for secure authentication, and the correct roles to read all your monitoring data. Next up, you will install the Azure Monitor Agent on your monitored VMs so they start sending telemetry to the Log Analytics Workspace.

Install Azure Monitor Agent and Create Data Collection Rules

Your VMs are running, but right now you have zero visibility into what's happening inside them. CPU could be spiking, memory filling up, or disks approaching capacity, and you'd never know until something breaks.

To fix this, you'll create a Data Collection Rule (DCR) that tells Azure exactly which performance metrics and logs to gather from each VM. When you add VMs to the DCR, Azure automatically installs the Azure Monitor Agent on each one. No manual agent installation required.

In this step, get ready to:

Create a Data Collection Rule named dcr-vm-performance targeting Linux VMs.
Configure performance counters and Syslog collection with law-monitoring-central as the destination.
Add all 3 monitored VMs as resources, triggering automatic Azure Monitor Agent installation.

Create a new Data Collection Rule

A Data Collection Rule defines what telemetry to collect, how often to sample it, and where to send it. You'll create one rule that covers all three monitored VMs.

In the Azure Portal, search for Monitor in the top search bar and select **Monitor**.

In the left sidebar under Settings, click Data Collection Rules.
Click + Create at the top of the page.

The Create Data Collection Rule wizard opens on the Basics tab.

Enter dcr-vm-performance in the Rule Name field.
Select your subscription from the Subscription dropdown.
Select rg-monitoring-lab for the Resource Group.
Confirm the Region matches the region you used for your resource group.
Set Platform Type to Linux.
Click Next: Resources.

What is a Data Collection Rule?

A DCR is a configuration object that tells Azure Monitor what data to collect from a machine's guest OS, how frequently to sample it, and which Log Analytics Workspace to send it to. Think of it as a telemetry recipe that you attach to one or more VMs.

Setting the Platform Type to Linux ensures only Linux-compatible performance counters appear in the configuration options. This avoids conflicts with Windows-style counter names.

Add your monitored VMs as resources

The Resources tab is where you select which VMs this DCR applies to. Adding a VM here automatically installs the Azure Monitor Agent on it. No SSH required.

Click + Add resources.
Expand your subscription, then expand rg-monitoring-lab.
Check the boxes next to web-server-01, app-server-01, and db-server-01.
Do NOT select grafana-server. That VM runs Grafana and does not need to be monitored by this rule.
Click Apply.
Click Next: Collect and deliver.

Why does adding VMs install the agent automatically?

When you add a VM as a resource in a DCR through the Azure Portal, Azure checks whether the Azure Monitor Agent extension is already installed. If it is not, Azure installs it as a VM extension automatically. No reboot is required.

The agent uses the VM's System-Assigned Managed Identity (which you enabled in previous steps) to authenticate with Azure Monitor.

Configure performance counters and Syslog data sources

The Collect and deliver tab is where you define what data to pull from each VM. You'll add two data sources: Performance Counters for metrics and Linux Syslog for log events.

Click + Add data source.
From the Data source type dropdown, select Performance Counters.

A predefined set of Linux performance counters appears. These include the metrics you need: CPU %, Available Memory, Disk %, and Network bytes.

Leave the default counters selected. Verify you can see counters for Processor (% Processor Time), Memory (Available MBytes), Logical Disk (% Used Space), and Network (Bytes Total/sec).
If you do not see Swap counters in the predefined set, select Custom and add a counter for Swap % Used Space.
Click the Destination tab within this data source panel.
Confirm that Destination type is set to Azure Monitor Logs.
Select law-monitoring-central as the Account or namespace.
Click Add data source to save this data source.

Where does performance data end up?

Performance counters are stored in the Perf table in your Log Analytics Workspace. Later, when you build Grafana dashboards, you'll query this table using KQL to visualize CPU, memory, disk, and network metrics per VM.

Now add the second data source for Syslog collection.

Click + Add data source again.
From the Data source type dropdown, select Linux Syslog.
Check the box next to Facility at the top to select all facilities.
In the Set minimum log level for selected facilities dropdown, select LOG_WARNING.
Click the Destination tab within this data source panel.
Confirm Azure Monitor Logs is selected and law-monitoring-central is set as the workspace.
Click Add data source to save.

Why LOG_WARNING and above?

Syslog severity levels range from Debug (noisiest) to Emergency (most critical). Collecting everything from Debug level up generates massive amounts of data and drives up ingestion costs.

Setting the minimum to LOG_WARNING captures Warning, Error, Critical, Alert, and Emergency events. These are the severity levels that matter for operational monitoring. Syslog data lands in the Syslog table in Log Analytics.

Click Review + create.
Review the summary. You should see 2 data sources (Performance Counters and Linux Syslog) and 3 resources (your three monitored VMs).
Click Create.

Azure creates the DCR and begins installing the Azure Monitor Agent extension on all three VMs. This takes a few minutes.

Wait for the deployment notification to show success.
Navigate back to Monitor > Data Collection Rules.
Click on dcr-vm-performance to open it.
Click the Resources tab. You should see all 3 VMs listed with the Azure Monitor Agent extension status showing as Provisioned or Succeeded.

Agent status not showing as Provisioned?

The Azure Monitor Agent can take 5-10 minutes to install. If the status is still showing as pending, wait a few minutes and refresh the page.

If the installation fails, verify that your VMs have System-Assigned Managed Identity enabled (you did this in Step 3). Also confirm the VMs are in the Running state.

help me troubleshoot why the Azure Monitor Agent extension is not installing on my VMs

How do I verify data is flowing?

Performance data takes about 5-10 minutes to start appearing. To check, navigate to your law-monitoring-central workspace, click Logs, and run the query Perf | take 10. If you see rows with your VM hostnames, data collection is working.

Your monitoring foundation is now active. All three VMs are reporting CPU, memory, disk, swap, and network metrics into a single workspace, alongside system log events. Next up, you'll deploy Azure Container Apps to add containerized workloads to your monitoring platform.

Deploy Azure Container Apps

Your VMs are now feeding performance metrics and syslog data into your central workspace. But modern production environments run more than just VMs.

In this step, you will deploy three Azure Container Apps to simulate a microservices architecture. These containerized services will generate application logs, system events, and metrics that all flow into the same law-monitoring-central workspace your VMs already use.

In this step, get ready to:

Create a Container Apps Environment with centralized logging.
Deploy three container apps simulating auth, API, and frontend services.
Configure diagnostic settings to send all container telemetry to your workspace.

Create the Container Apps Environment

A Container Apps Environment is the shared hosting boundary for your container apps. All apps in the same environment share networking and logging configuration.

In the Azure portal search bar, type Container Apps and select Container Apps from the results.
Click Create.
Select your subscription and resource group rg-monitoring-lab.
Enter auth-service in the Container app name field. You will rename this later, but Azure requires an app name to create the environment.
Select the same region you used for your other resources.
In the Container Apps Environment field, click Create new.

The environment creation panel opens. Configure it with these settings:

Enter cae-monitoring-lab in the Environment name field.
Select the Monitoring tab at the top of the environment creation panel.
Set Logs Destination to Azure Log Analytics.
In the Log Analytics workspace dropdown, select law-monitoring-central.
Click Create to confirm the environment settings.

What is a Container Apps Environment?

Think of it as a shared boundary for a group of container apps. Apps in the same environment share a virtual network and can write logs to the same destination.

By pointing the logging destination at law-monitoring-central, every container in this environment automatically sends its console output and system events to your central workspace.

Now finish creating the first container app:

Select the Container tab at the top.
Uncheck Use quickstart image.
Enter auth-service in the Name field.
Set Image source to Docker Hub or other registries.
Enter mcr.microsoft.com/azuredocs/containerapps-helloworld in the Image and tag field.
Select the Ingress tab at the top.
Set Ingress to Enabled.
Select Accepting traffic from anywhere for external ingress.
Set Target port to 80.
Click Review + create, then click Create.

Wait for the deployment to complete. This may take 2-3 minutes as Azure provisions the environment and deploys the first container.

Why these container images?

The containerapps-helloworld image is a lightweight sample app from Microsoft that responds on port 80. It works well for simulating services that generate standard console logs.

The httpbin image (used next for the API service) exposes endpoints like /status/500 that return any HTTP status code on demand. This makes it perfect for generating error logs in later dashboard steps.

Deployment stuck or failed?

If provisioning takes more than 5 minutes, check that your subscription has the Microsoft.App and Microsoft.OperationalInsights resource providers registered. Go to your subscription, select Resource providers, search for these providers, and click Register if they show as NotRegistered.

help me troubleshoot my Container Apps Environment deployment failure

Deploy the remaining container apps

You now have the environment and your first app running. Next, deploy the two remaining services into the same environment.

In the portal search bar, type Container Apps and select Container Apps.
Click Create.
Select resource group rg-monitoring-lab.
Enter api-service in the Container app name field.
Select the same region as your other resources.
In the Container Apps Environment dropdown, select cae-monitoring-lab (the environment you just created).
Select the Container tab.
Uncheck Use quickstart image.
Enter api-service in the Name field.
Set Image source to Docker Hub or other registries.
Enter docker.io/kennethreitz/httpbin in the Image and tag field.
Select the Ingress tab.
Set Ingress to Enabled.
Select Accepting traffic from anywhere.
Set Target port to 80.
Click Review + create, then click Create.

Wait for the deployment to complete, then repeat the same process for the third app:

Click Create again from the Container Apps page.
Select resource group rg-monitoring-lab.
Enter frontend-service in the Container app name field.
Select the same region.
Select cae-monitoring-lab as the environment.
Select the Container tab.
Uncheck Use quickstart image.
Enter frontend-service in the Name field.
Set Image source to Docker Hub or other registries.
Enter mcr.microsoft.com/azuredocs/containerapps-helloworld in the Image and tag field.
Select the Ingress tab.
Set Ingress to Enabled.
Select Accepting traffic from anywhere.
Set Target port to 80.
Click Review + create, then click Create.

Once the deployment completes, verify all three apps are running.

Navigate to Container Apps in the portal.
Confirm you see auth-service, api-service, and frontend-service listed with status Running.
Click on api-service and copy its Application Url from the Overview page.
Open that URL in your browser. You should see the httpbin interface with a list of available endpoints.

App shows Provisioning or Degraded status?

If api-service is stuck in provisioning, the docker.io/kennethreitz/httpbin image may be slow to pull. Wait up to 5 minutes for the initial pull to complete.

If a revision shows as degraded, click on the app, go to Monitoring then Log stream, and check the System logs for image pull errors.

help me troubleshoot why my Container App revision is degraded or stuck provisioning

Configure diagnostic settings on the environment

The environment already sends basic console logs to your workspace via the Log Analytics destination you configured during creation. Diagnostic settings give you additional telemetry categories including system logs and platform metrics.

In the portal search bar, type Container Apps Environments and select Container Apps Environments.
Click on cae-monitoring-lab.
In the left menu, expand Monitoring and select Diagnostic settings.
Click Add diagnostic setting.
Enter diag-cae-to-law in the Diagnostic setting name field.
Under Logs, check ContainerAppConsoleLogs.
Check ContainerAppSystemLogs.
Under Metrics, check AllMetrics.
Under Destination details, check Send to Log Analytics workspace.
Select law-monitoring-central from the workspace dropdown.
Click Save.

What do these diagnostic categories capture?

ContainerAppConsoleLogs captures everything your containers write to stdout and stderr. This is the primary source for application-level logging.

ContainerAppSystemLogs captures platform events like revision provisioning, container crashes, image pull failures, and restart events. These are critical for infrastructure observability.

AllMetrics sends CPU, memory, network, and replica count metrics to your workspace for querying in Grafana later.

Refresh if Diagnostic settings is missing

If you do not see the Diagnostic settings option under Monitoring, refresh the portal page. It can take a moment to appear after the environment is created.

Verify the diagnostic setting saved successfully by confirming it appears in the list with status Enabled.

Diagnostic setting not saving?

Make sure you selected at least one log category AND a destination. The Save button stays greyed out until both are configured.

Also confirm your account has Contributor or Owner role on the resource group. Reader-level access cannot create diagnostic settings.

help me fix my Container Apps diagnostic settings configuration

Your container apps platform is now fully wired into your central monitoring workspace. In the next step, you will install Grafana on your dedicated server and connect it to this telemetry data.

Install and Configure Grafana with Apache2

Your monitoring data is flowing into law-monitoring-central from VMs and Container Apps. Now you need a way to visualize it all.

In this step, you will install Grafana OSS on your dedicated grafana-server VM and put Apache2 in front of it as a reverse proxy. This lets users access Grafana on port 80 (standard HTTP) without exposing the raw Grafana port 3000 to the internet.

In this step, get ready to:

Install Grafana OSS from the official APT repository on grafana-server.
Configure Apache2 as a reverse proxy to serve Grafana on port 80.
Access Grafana in your browser and set a secure admin password.

SSH into grafana-server and install Grafana OSS

Grafana runs as a lightweight service on your VM, listening on port 3000 by default. You will install it from Grafana Labs' official APT repository so it stays up to date with system package management.

Open a terminal on your local machine.
SSH into your grafana-server by running this command:

ssh [[SSH_USER="your-admin-username"]]@[[GRAFANA_PUBLIC_IP="your grafana-server public IP address"]]

Find your grafana-server Public IP

In the Azure Portal, navigate to rg-monitoring-lab and click on grafana-server. The Public IP address is shown on the Overview page.

Once connected, install the prerequisite packages by running:

sudo apt-get install -y apt-transport-https wget gnupg

Import the Grafana GPG key by running these commands:

sudo mkdir -p /etc/apt/keyrings
sudo wget -O /etc/apt/keyrings/grafana.asc https://apt.grafana.com/gpg-full.key
sudo chmod 644 /etc/apt/keyrings/grafana.asc

Add the Grafana stable repository by running:

echo "deb [signed-by=/etc/apt/keyrings/grafana.asc] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

Update packages and install Grafana OSS by running:

sudo apt-get update
sudo apt-get install -y grafana

Start the Grafana service and enable it to run on boot by running:

sudo systemctl start grafana-server
sudo systemctl enable grafana-server

Verify the service is running by running:

sudo systemctl status grafana-server

You should see active (running) in green text. This confirms Grafana is listening on localhost:3000.

What did you just install?

Grafana OSS is an open-source visualization platform. It connects to data sources like Azure Monitor and Log Analytics, then renders live dashboards and handles alerting.

Installing from the APT repository means you can update Grafana in the future with a simple apt-get update && apt-get upgrade.

Grafana service not starting?

If the status shows failed or inactive, check the journal logs by running sudo journalctl -u grafana-server --no-pager -n 50. Common causes include port 3000 already in use or a corrupted install. Try reinstalling with sudo apt-get install --reinstall grafana.

Help me troubleshoot why my Grafana service isn't starting on Ubuntu

Install Apache2 and configure the reverse proxy

Grafana is running on port 3000, but your NSG only allows inbound traffic on port 80. A reverse proxy bridges this gap by accepting HTTP requests on port 80 and forwarding them internally to Grafana.

Why a reverse proxy instead of changing Grafana's port?

Running services directly on port 80 requires root privileges, which is a security risk. A reverse proxy like Apache2 handles the public-facing connection while Grafana runs safely as a non-root service on its default port.

Install Apache2 by running:

sudo apt-get install -y apache2

Enable the proxy modules that Apache2 needs for forwarding by running:

sudo a2enmod proxy
sudo a2enmod proxy_http

What do these modules do?

The proxy module gives Apache the ability to forward requests to other servers. The proxy_http module adds support for proxying HTTP specifically (as opposed to FTP or WebSockets).

Create a new VirtualHost configuration file for Grafana by running:

sudo tee /etc/apache2/sites-available/grafana.conf > /dev/null <<EOF
<VirtualHost *:80>
    ProxyPass / http://localhost:3000/
    ProxyPassReverse / http://localhost:3000/
</VirtualHost>
EOF

What does this VirtualHost do?

ProxyPass tells Apache to forward all incoming requests on port 80 to Grafana at localhost:3000. ProxyPassReverse rewrites response headers so that URLs in Grafana's responses point back to port 80 instead of 3000.

Disable the default Apache site and enable your new Grafana site by running:

sudo a2dissite 000-default
sudo a2ensite grafana

Restart Apache2 to apply all changes by running:

sudo systemctl restart apache2

Verify Apache2 is running by running:

sudo systemctl status apache2

You should see active (running) in the output. Apache2 is now accepting traffic on port 80 and forwarding it to Grafana.

Apache2 not starting?

If you see an error about port 80 being in use, another process may be occupying it. Run sudo lsof -i :80 to find what is using the port. If the config test fails, run sudo apache2ctl configtest to see which line has the syntax error.

Help me fix my Apache2 configuration

Access Grafana and change the admin password

With Apache2 forwarding traffic, Grafana should now be reachable from your browser on port 80. The first login uses default credentials that you must change immediately.

Open your browser and navigate to your grafana-server Public IP address:

http://[[GRAFANA_PUBLIC_IP="your grafana-server public IP address"]]

You should see the Grafana login page with the Grafana logo and two fields for email/username and password.

Enter admin in the Email or username field.
Enter admin in the Password field.
Click Log in.

Grafana immediately prompts you to change the default password.

Enter a new secure password in the New password field.
Confirm the password in the Confirm new password field.
Click Submit.

You should now see the Grafana home dashboard with a welcome banner. This confirms Grafana is fully accessible through Apache2 on port 80.

Can't reach the Grafana login page?

Make sure you are using http:// (not https://). Verify your NSG allows inbound traffic on port 80. You can also test from the VM itself by running curl http://localhost to confirm Apache is responding locally.

Help me troubleshoot why I can't reach Grafana on port 80

Your Grafana instance is live and accessible from the internet. Next, you will connect it to Azure Monitor so it can query your Log Analytics Workspace and start building dashboards.

Configure Grafana Data Source with Managed Identity

Your Grafana server is up and running, but it has no idea where to pull monitoring data from. Right now it is just an empty dashboard tool with no connection to your Azure Monitor telemetry.

In this step, you will connect Grafana to Azure Monitor using Managed Identity. Because your grafana-server VM already has a system-assigned identity with the right roles, you won't need to manage any client secrets or passwords for this connection.

In this step, get ready to:

Add the Azure Monitor data source in Grafana.
Configure authentication using Managed Identity.
Test the connection to confirm Grafana can query your Log Analytics Workspace.

Add the Azure Monitor data source

The Azure Monitor data source is built into Grafana core. You do not need to install any separate plugins.

Open your browser and navigate to your Grafana instance at http:// followed by your grafana-server public IP address.
Log in with the admin credentials you set in the previous step.
In the left sidebar, click the hamburger menu icon (three horizontal lines), then click Connections.
Click Add new connection.
In the search bar, type Azure Monitor.
Click the Azure Monitor result that appears.
Click Add new data source in the top right corner.

You should now see the Azure Monitor data source configuration page with fields for authentication settings.

Why is Azure Monitor built into Grafana?

Azure Monitor has been included in Grafana core since version 6.0. The older grafana-azure-monitor-datasource plugin is deprecated and archived.

This means every Grafana installation already has everything it needs to connect to Azure Monitor, Log Analytics, and Azure Resource Graph without any extra setup.

Set authentication to Managed Identity

Managed Identity lets your Grafana VM authenticate to Azure services without storing any client secrets or certificates. Azure handles the token exchange behind the scenes.

On the Azure Monitor data source configuration page, find the Authentication section.
Click the Authentication dropdown.
Select Managed Identity from the dropdown options.

The form should simplify after selecting Managed Identity. You won't see fields for Directory ID, Application ID, or Client Secret because none are needed.

Why Managed Identity instead of an App Registration?

Since your Grafana VM runs directly on Azure, Managed Identity eliminates the need to manage client secrets entirely. Azure automatically provisions and rotates the credentials for you.

This works because you already assigned the Reader and Log Analytics Reader roles to grafana-server's identity in Step 4. Those roles give Grafana permission to query metrics and logs from law-monitoring-central.

Save and test the connection

Scroll to the bottom of the data source configuration page.
Click Save & Test.

You should see a green success banner confirming that Grafana successfully connected to Azure Monitor. The message indicates that Grafana can query your subscriptions and Log Analytics workspaces.

Seeing an authentication error?

If the test fails with a permissions or authentication error, check these common causes:

Verify that the grafana-server VM has its System-Assigned Managed Identity enabled. Go to the VM in the Azure Portal, click Identity in the left menu, and confirm the status is On.
Confirm the Managed Identity has the Reader and Log Analytics Reader roles assigned at the rg-monitoring-lab resource group scope. Check this under Access control (IAM) on the resource group.
Role assignments can take a few minutes to propagate. If you just assigned the roles, wait 5 minutes and try Save & Test again.

Help me troubleshoot why my Grafana Azure Monitor data source is failing with a Managed Identity authentication error

Your Grafana instance is now connected to Azure Monitor with zero secrets to manage. Next up, you will start building live dashboards that visualize the CPU, memory, disk, and swap metrics flowing in from your monitored VMs.

Build Per-VM Live Dashboards

Your Grafana server is connected to law-monitoring-central and can query every performance counter your VMs are reporting.

Now you will turn that raw telemetry into live dashboards. Each of your three monitored VMs will get its own dashboard showing CPU, Memory, Disk, Swap, and Network traffic in real time. You will write KQL queries against the Perf table and visualize the results as time series panels that auto-refresh every 5 seconds.

In this step, get ready to:

Build a 5-panel Grafana dashboard for web-server-01 with CPU, Memory, Disk, Swap, and Network panels.
Duplicate the dashboard for app-server-01 and db-server-01.
Set all three dashboards to auto-refresh every 5 seconds.

Create the web-server-01 CPU panel

You will start with a single panel that queries CPU usage from the Perf table. The Perf table stores all performance counter data collected by the Azure Monitor Agent on your Linux VMs.

In Grafana, click **Dashboards** in the left sidebar.
Click New and select New Dashboard.
On the empty dashboard, click + Add visualization.
Select Azure Monitor as the data source.
In the query editor, select the Logs service.
Select your law-monitoring-central workspace as the resource.
Paste the following KQL query into the query editor:

Perf
| where $__timeFilter(TimeGenerated)
| where ObjectName == "Processor" and CounterName == "% Processor Time"
| where Computer == "web-server-01"
| summarize avg(CounterValue) by bin(TimeGenerated, 1m)
| order by TimeGenerated asc

What does this query do?

The Perf table holds all performance counters collected by the Azure Monitor Agent. The $__timeFilter(TimeGenerated) macro is a special Grafana function that automatically filters results to match your dashboard's selected time range.

The query filters for the Processor object and % Processor Time counter, then narrows results to only web-server-01. It averages the value into 1-minute time bins and sorts chronologically for the time series visualization.

On Linux VMs, % Processor Time represents overall CPU utilization as a percentage.

Click Refresh to run the query and see data populate.
In the visualization type selector on the right, confirm Time series is selected.
Under Panel options, set the title to CPU %.
Click Apply in the top right to save the panel to the dashboard.

Not seeing any data?

Performance data can take 5-10 minutes to appear after the Azure Monitor Agent starts collecting. If your panels are empty, wait a few minutes and click Refresh again.

Make sure the Computer filter matches the exact hostname. Run Perf | distinct Computer in the query editor to see what hostnames your workspace has recorded.

help me troubleshoot why my Grafana CPU panel shows no data from the Perf table

Add the Memory, Disk, Swap, and Network panels

Each Linux performance counter lives in the Perf table under a different ObjectName and CounterName combination. You will add four more panels to complete the web-server-01 dashboard.

Click Add in the dashboard header, then select Visualization.
Select Azure Monitor as the data source and choose the Logs service with law-monitoring-central as the resource.
Paste the following Memory query:

Perf
| where $__timeFilter(TimeGenerated)
| where ObjectName == "Memory" and CounterName == "% Used Memory"
| where Computer == "web-server-01"
| summarize avg(CounterValue) by bin(TimeGenerated, 1m)
| order by TimeGenerated asc

Set the visualization to Time series and title the panel Memory %.
Click Apply.
Add another visualization panel and paste the Disk query:

Perf
| where $__timeFilter(TimeGenerated)
| where ObjectName == "Logical Disk" and CounterName == "% Used Space"
| where Computer == "web-server-01"
| summarize avg(CounterValue) by bin(TimeGenerated, 1m)
| order by TimeGenerated asc

Title this panel Disk % and click Apply.
Add another visualization panel and paste the Swap query:

Perf
| where $__timeFilter(TimeGenerated)
| where ObjectName == "Memory" and CounterName == "% Used Swap Space"
| where Computer == "web-server-01"
| summarize avg(CounterValue) by bin(TimeGenerated, 1m)
| order by TimeGenerated asc

Title this panel Swap % and click Apply.
Add one final visualization panel and paste the Network Traffic query:

Perf
| where $__timeFilter(TimeGenerated)
| where ObjectName == "Network" and CounterName in ("Total Bytes Received", "Total Bytes Transmitted")
| where Computer == "web-server-01"
| summarize avg(CounterValue) by bin(TimeGenerated, 1m), CounterName
| order by TimeGenerated asc

Why split Network by CounterName?

Adding CounterName to the summarize by clause creates two separate lines on the time series graph. One line shows Total Bytes Received (inbound) and the other shows Total Bytes Transmitted (outbound). This gives you instant visibility into traffic direction.

Title this panel Network Traffic and click Apply.
Set the dashboard auto-refresh interval by clicking the refresh dropdown in the top toolbar and selecting 5s.
Save the dashboard by pressing Ctrl+S (Windows) or Cmd+S (macOS).
Enter web-server-01 Metrics as the dashboard title and click Save.

You should now see 5 panels on your dashboard, each displaying a live time series graph. The panels update every 5 seconds.

Seeing 'No data' on Memory or Swap panels?

Linux counter names are case-sensitive. Verify the exact counter names available in your workspace by running this discovery query in any panel:

Perf
| where Computer == "web-server-01"
| distinct ObjectName, CounterName

Compare the output to the ObjectName and CounterName values in your panel queries. Adjust if your DCR uses slightly different counter names.

help me find the correct Linux performance counter names in my Log Analytics workspace

Duplicate the dashboard for app-server-01 and db-server-01

Instead of building two more dashboards from scratch, you can duplicate the web-server-01 dashboard and update the Computer filter in each panel. Grafana's Save As feature copies all panels, queries, and settings into a new dashboard.

On the web-server-01 Metrics dashboard, click the Save dashboard dropdown arrow (next to the save icon) and select Save As.
Enter app-server-01 Metrics as the new dashboard name and click Save.
You are now on the duplicated dashboard. Click each panel's title, then select Edit.
In the KQL query, change Computer == "web-server-01" to Computer == "app-server-01".
Click Apply after editing each panel.
Repeat for all 5 panels (CPU %, Memory %, Disk %, Swap %, Network Traffic).
Save the dashboard with Ctrl+S (Windows) or Cmd+S (macOS).

Now repeat the same process to create the third dashboard.

Navigate back to Dashboards in the left sidebar and open the web-server-01 Metrics dashboard.
Click the Save dashboard dropdown and select Save As.
Enter db-server-01 Metrics as the new dashboard name and click Save.
Edit each of the 5 panels and change Computer == "web-server-01" to Computer == "db-server-01".
Click Apply on each panel, then save the dashboard.
Confirm the auto-refresh is set to 5s on both new dashboards by clicking the refresh dropdown in the toolbar.

You should now have three dashboards in your Grafana instance. Each shows live CPU, Memory, Disk, Swap, and Network metrics for its respective VM.

Why per-VM dashboards instead of one combined dashboard?

Per-VM dashboards give operations teams a focused view when triaging an incident. If a specific VM is alerting, you can open its dedicated dashboard and immediately see all resource metrics without filtering through other hosts. This mirrors how production SRE teams organize their observability.

Panels still showing web-server-01 data after editing?

Make sure you clicked Apply after editing each panel's query. If you only clicked Back to dashboard without applying, the change is discarded.

Also verify you saved the dashboard after applying changes to all 5 panels. Unsaved dashboards revert on page reload.

help me fix my duplicated Grafana dashboard that still shows data from the wrong VM

Your per-VM dashboards are now live and auto-refreshing. Next up, you will build a dedicated dashboard for NAT Gateway data transit monitoring.

Build the NAT Gateway Data Transit Dashboard

Your per-VM dashboards give you visibility into individual machine health. But they only show what happens inside each VM.

The NAT Gateway sits between your VMs and the internet, routing all outbound traffic through a single public IP. Monitoring it reveals network-level problems that individual VMs cannot see: SNAT port exhaustion, dropped packets, and unexpected data spikes.

In this step, get ready to:

Create a NAT Gateway Transit dashboard with a ByteCount time series panel.
Add a SNAT Connection Count panel to detect port exhaustion.
Add Dropped Packets and Datapath Availability panels with auto-refresh.

Create the NAT Gateway Transit dashboard and ByteCount panel

The ByteCount metric tracks the total number of bytes flowing through the NAT Gateway. Splitting by direction lets you see inbound and outbound traffic independently.

In Grafana, click the + icon in the left sidebar.
Click New dashboard.
Click the gear icon (Dashboard settings) in the top right.
Set the Title to NAT Gateway Transit.
Click Save dashboard, then click Back to dashboard.

Now add the first panel for data transit volume.

Click Add visualization.
Select the Azure Monitor data source.
In the query editor, set Service to Metrics.
For Resource, select your subscription, then resource group rg-monitoring-lab, resource type NAT gateways, and resource natgw-monitoring.
Set Metric to ByteCount.
Set Aggregation to Sum.
Click Add splitting and select Direction as the dimension. This splits the graph into two lines: In and Out.
In the panel title field at the top, enter Data Transit (Bytes In/Out).
Confirm the visualization type is set to Time series in the right sidebar.
Click Apply in the top right to save the panel.

What does this panel show?

The ByteCount metric measures total bytes processed by the NAT Gateway. By splitting on the Direction dimension, you get two lines: one for outbound data (your VMs reaching the internet) and one for inbound response data.

Sudden spikes in outbound traffic could indicate a compromised VM, a misconfigured service making excessive API calls, or a data exfiltration attempt.

You should see a time series graph with two lines representing In and Out directions. If your VMs have been running for a while, you will see some baseline traffic from the Azure Monitor Agent sending telemetry.

Panel shows no data?

Make sure you selected natgw-monitoring as the resource, not one of the VMs. The ByteCount metric lives on the NAT Gateway resource itself.

Try expanding the time range to Last 6 hours using the time picker in the top right. NAT Gateway metrics can take a few minutes to populate initially.

Still stuck? Help me troubleshoot why my Grafana panel shows no data for the NAT Gateway ByteCount metric

Add the SNAT Connection Count panel

SNAT (Source Network Address Translation) connections track how your VMs share the NAT Gateway's public IP. When too many connections compete for available ports, SNAT port exhaustion occurs and outbound connections start failing.

Back on the dashboard, click Add in the top toolbar, then Visualization.
Select the Azure Monitor data source.
Set Service to Metrics.
Select the same NAT Gateway resource: subscription, rg-monitoring-lab, NAT gateways, natgw-monitoring.
Set Metric to SNATConnectionCount.
Set Aggregation to Sum.
Click Add splitting and select ConnectionState as the dimension. This splits into Attempted vs Failed lines.
Set the panel title to SNAT Connections (Attempted vs Failed).
Confirm the visualization type is Time series.
Click Apply.

Why monitor SNAT connections?

Each NAT Gateway public IP provides 64,512 SNAT ports. When all ports are in use, new outbound connections fail. The Failed line on this panel is your early warning system. If you see Failed connections climbing, you need to add more public IPs to the NAT Gateway or reduce concurrent outbound connections.

You should see at least the Attempted line showing connection activity. The Failed line should ideally stay at zero for a healthy environment.

Add Dropped Packets and Datapath Availability panels

These two panels complete your NAT Gateway visibility. Dropped packets indicate network congestion or misconfiguration. Datapath availability tells you whether the NAT Gateway itself is healthy.

Click Add in the top toolbar, then Visualization.
Select the Azure Monitor data source.
Set Service to Metrics.
Select the NAT Gateway resource: rg-monitoring-lab, NAT gateways, natgw-monitoring.
Set Metric to PacketDropCount.
Set Aggregation to Sum.
Set the panel title to Dropped Packets.
Confirm the visualization type is Time series.
Click Apply.

Now add the final panel for datapath availability.

Click Add in the top toolbar, then Visualization.
Select the Azure Monitor data source.
Set Service to Metrics.
Select the NAT Gateway resource: rg-monitoring-lab, NAT gateways, natgw-monitoring.
Set Metric to DatapathAvailability.
Set Aggregation to Average.
In the right sidebar, change the visualization type from Time series to Gauge.
Set the panel title to Datapath Availability (%).
Click Apply.

What do these panels tell you?

Dropped Packets shows traffic the NAT Gateway could not process. A flat line at zero is healthy. Spikes indicate congestion, misrouted traffic, or resource limits being hit.

Datapath Availability is a percentage gauge showing the NAT Gateway's overall health. 100% means fully operational. Any drop below 100% signals a platform-level issue that Azure may be investigating.

Now set the dashboard to auto-refresh so it stays current.

In the top right corner of the dashboard, click the refresh interval dropdown (it may show Off or a time value).
Select 30s from the list.
Click the save icon (floppy disk) in the top toolbar to save the dashboard.

Your NAT Gateway Transit dashboard should now display 4 panels: a time series for bytes in/out, a time series for SNAT connections, a time series for dropped packets, and a gauge for datapath availability.

Gauge showing no data or 0%?

The DatapathAvailability metric only reports data when the NAT Gateway is actively processing traffic. If your VMs have been idle, try SSHing into grafana-server and running a quick outbound request from one of the monitored VMs to generate traffic.

Also confirm the time range is wide enough (at least Last 1 hour). Metrics from NAT Gateway are reported at 1-minute intervals.

Need more help? Help me troubleshoot my Grafana NAT Gateway dashboard panels showing no data

Your NAT Gateway dashboard is live. Next up, you will build dashboards for your Container Apps and set up live log streaming directly in Grafana.

Build Container Apps and Live Logs Dashboards

Your VM dashboards and NAT Gateway dashboard give you infrastructure-level visibility. Now you need the same observability for your containerized workloads.

In this step, you will build two dashboards that focus on your Azure Container Apps. The first summarizes overall health (log volume, CPU, memory, replicas). The second streams logs in real time so you can watch events as they happen.

In this step, get ready to:

Create a Container Apps dashboard with log volume and resource metrics panels.
Create a Live Logs dashboard with real-time console output and system event panels.
Add service-level filters so you can isolate logs per container app.

Create the Container Apps dashboard

This dashboard gives you a single view of how busy your container apps are and how many resources they consume.

In Grafana, click the + icon in the left sidebar and select New dashboard.
Click the gear icon (dashboard settings) at the top.
Set the Name to Container Apps and click Save dashboard.
Click Add visualization to add your first panel.
In the query editor, select Azure Monitor as the data source and choose Logs as the service.
Select your law-monitoring-central workspace from the Resource dropdown.
Paste the following KQL query to show log volume per container app:

ContainerAppConsoleLogs_CL
| summarize LogCount = count() by ContainerAppName_s, bin(TimeGenerated, 5m)
| order by TimeGenerated desc

What does this query do?

This query counts the number of console log entries per container app in 5-minute buckets. A sudden spike in log volume often indicates an error loop or unexpected traffic.

Set the panel visualization to Time series.
Set the panel title to Log Volume per App.
Click Apply in the top-right corner to save the panel.

Now add resource metric panels. These query the AzureMetrics table which stores metrics sent via Diagnostic Settings.

Click Add visualization again.
Select Azure Monitor data source, Logs service, and the law-monitoring-central workspace.
Paste the following query for container CPU usage:

AzureMetrics
| where ResourceProvider == "MICROSOFT.APP"
| where MetricName == "UsageNanoCores"
| summarize AvgCPU = avg(Average) by Resource, bin(TimeGenerated, 5m)
| order by TimeGenerated desc

Set the visualization to Time series and the title to Container CPU (NanoCores).
Click Apply.

Add another panel with this query for container memory:

AzureMetrics
| where ResourceProvider == "MICROSOFT.APP"
| where MetricName == "WorkingSetBytes"
| summarize AvgMemory = avg(Average) by Resource, bin(TimeGenerated, 5m)
| order by TimeGenerated desc

Set the visualization to Time series and the title to Container Memory (WorkingSetBytes).
Click Apply.

Add one more panel with this query for replica count:

AzureMetrics
| where ResourceProvider == "MICROSOFT.APP"
| where MetricName == "Replicas"
| summarize ReplicaCount = avg(Average) by Resource, bin(TimeGenerated, 5m)
| order by TimeGenerated desc

Set the visualization to Time series and the title to Replica Count.
Click Apply.
Click the save icon (disk) at the top to save the dashboard.

What are these metric names?

Azure Container Apps reports UsageNanoCores for CPU consumption, WorkingSetBytes for memory usage, and Replicas for the running instance count. These flow into the AzureMetrics table in your Log Analytics Workspace because you enabled AllMetrics in Diagnostic Settings.

You should now see four panels on your Container Apps dashboard showing log volume, CPU, memory, and replica count across your three container apps.

Panels showing no data?

It can take 5-10 minutes for metrics to flow into the AzureMetrics table after enabling Diagnostic Settings. Widen the time range to Last 1 hour or Last 6 hours to check for older data.

Also confirm that your Diagnostic Settings on cae-monitoring-lab include AllMetrics and point to law-monitoring-central.

help me troubleshoot why my Container Apps metrics panels show no data in Grafana

Create the Live Logs dashboard

Live Logs gives you a streaming view of container console output. This is the dashboard you will watch during deployments and incidents to see what your apps are printing in real time.

Click the + icon in the left sidebar and select New dashboard.
Open dashboard settings and set the Name to Live Logs.
Set Auto refresh to 5s using the refresh interval dropdown at the top-right of the dashboard.
Click Save dashboard.
Click Add visualization.
Select Azure Monitor data source, Logs service, and the law-monitoring-central workspace.
Paste the following query to stream real-time console output:

ContainerAppConsoleLogs_CL
| where ContainerAppName_s in ("auth-service", "api-service", "frontend-service")
| project TimeGenerated, App=ContainerAppName_s, Message=Log_s
| order by TimeGenerated desc
| take 200

What does this query do?

The where clause filters to your three container apps. project selects only the columns you need for readability. take 200 limits the result set so the panel loads quickly on each 5-second refresh.

Set the visualization to Logs (or Table if Logs is unavailable in your Grafana version).
Set the panel title to Console Logs (Real-Time).
Click Apply.

Your Live Logs dashboard now auto-refreshes every 5 seconds, streaming the latest container console output.

Filter by a single service

To isolate logs for one app, change the in clause to a single value like where ContainerAppName_s == "api-service". You can also use Grafana's dashboard variables to create a dropdown filter. For now, the static filter works well.

Add the system logs panel

System logs capture events that the Container Apps platform generates. These include revision provisioning, container crashes, image pull errors, and restart events.

On the same Live Logs dashboard, click Add visualization.
Select Azure Monitor data source, Logs service, and the law-monitoring-central workspace.
Paste the following query for system events:

ContainerAppSystemLogs_CL
| where Log_s contains "revision" or Log_s contains "error" or Log_s contains "restart" or Log_s contains "Provisioning"
| project TimeGenerated, App=ContainerAppName_s, Revision=RevisionName_s, Message=Log_s
| order by TimeGenerated desc
| take 100

What does this query show?

The ContainerAppSystemLogs_CL table stores platform-level events. Filtering for keywords like revision, error, restart, and Provisioning surfaces the most operationally important entries: new revision deployments, failed provisioning, and container crashes.

Set the visualization to Table.
Set the panel title to System Events (Provisioning, Errors, Restarts).
Click Apply.
Save the dashboard by clicking the save icon at the top.

Your Live Logs dashboard should now have two panels. The first streams real-time console output. The second shows system-level events like revision provisioning and container restarts.

System logs panel is empty?

System log entries are only generated when platform events occur (deployments, crashes, scaling). If no changes have happened recently, this panel will be empty.

To generate a test entry, go to one of your container apps in the Azure Portal, click Revision management, and create a new revision. Within a few minutes, a "Creating a new revision" entry should appear.

help me troubleshoot why my ContainerAppSystemLogs_CL query returns no results in Grafana

You now have dedicated container observability dashboards. Next up, you will build an Application Error dashboard to catch HTTP errors and exceptions from your services.

Build the Application Error Dashboard

Your monitoring platform now covers infrastructure metrics, NAT Gateway traffic, container health, and live log streaming. The next critical piece is detecting application-level errors as they happen.

In production, HTTP 500 and 401 errors often go unnoticed until users report them. In this step, you will build a dedicated error dashboard that surfaces these failures in near real-time. You will also generate test errors using the httpbin container to prove the dashboard works.

In this step, get ready to:

Create an Application Errors dashboard with a table panel filtering for error keywords in container logs.
Generate test HTTP errors using the httpbin /status endpoint.
Add a time series panel showing error rate over time grouped by container app.

Create the error log table panel

The ContainerAppConsoleLogs_CL table captures all stdout and stderr output from your container apps. By filtering for keywords like "error", "500", and "crash", you can isolate application failures from the noise of normal log output.

Open your Grafana instance in your browser (the grafana-server public IP on port 80).
Click the + icon in the left sidebar, then select New dashboard.
Click Add visualization.
Select your Azure Monitor data source.
In the query editor, select Logs as the service.
Select your law-monitoring-central workspace as the resource.
Enter the following KQL query:

ContainerAppConsoleLogs_CL
| where Log_s has_any ("error", "500", "401", "crash", "exception")
| project TimeGenerated, ContainerAppName_s, Log_s
| order by TimeGenerated desc

What does this query do?

ContainerAppConsoleLogs_CL contains all console output from your container apps. The has_any operator checks if the Log_s field contains any of the specified error keywords.

The project operator selects only the columns you care about: the timestamp, which app produced the log, and the log message itself. Results are sorted newest-first so you see the latest errors at the top.

In the panel settings on the right, change the visualization type to Table.
Set the panel title to Error Logs.
Click Run query to preview the results.
Click Apply in the top right to save the panel to your dashboard.

The table may be empty at this point. That is expected because no error-producing requests have been generated yet.

Click the dashboard title area at the top and rename it to Application Errors.
Click the Save dashboard icon (floppy disk) in the top right to save.

Query returning an error?

Make sure you selected Logs as the service type in the query editor, not Metrics. KQL queries only work against the Logs service.

Verify that the table name is ContainerAppConsoleLogs_CL with the _CL suffix. This suffix is required because the Container Apps Environment sends logs directly to Log Analytics.

Help me troubleshoot my KQL query in Grafana against ContainerAppConsoleLogs_CL

Generate test errors with httpbin

Your api-service container app runs the httpbin image. This tool has a /status/{code} endpoint that returns whatever HTTP status code you request. This makes it perfect for simulating application errors.

In the Azure Portal, navigate to your api-service Container App.
In the Overview blade, find the Application Url and copy it.
Open a new browser tab and navigate to your api-service URL followed by /status/500.

The URL will look something like:

https://api-service.<your-environment-domain>.azurecontainerapps.io/status/500

Refresh the page 5-10 times to generate multiple 500 error entries in the logs.
Change the URL path to /status/401 and refresh 5-10 times to generate 401 errors.

What should I see in the browser?

The /status/500 endpoint returns a blank page with an HTTP 500 Internal Server Error status. The /status/401 endpoint returns an HTTP 401 Unauthorized status. Both produce log entries in ContainerAppConsoleLogs_CL that your error dashboard can detect.

Wait 2-5 minutes for the logs to flow from the container app into law-monitoring-central.
Switch back to your Grafana browser tab and open the Application Errors dashboard.
Click the refresh icon in the top right of the dashboard.

You should see rows appearing in your Error Logs table with timestamps, the api-service container name, and log entries containing "500" or "401".

Still seeing an empty table after 5 minutes?

Log ingestion can sometimes take up to 10 minutes. Expand the time range in the top right of the Grafana dashboard to Last 30 minutes or Last 1 hour to ensure you are not filtering out recent entries.

You can also verify logs are arriving by running the query directly in the Azure Portal. Navigate to law-monitoring-central, click Logs, and run ContainerAppConsoleLogs_CL | take 10 to confirm data is flowing.

Help me troubleshoot why my Container App error logs are not appearing in Grafana

Add the error rate time series panel

A table shows individual error entries, but a time series graph reveals patterns. Spikes in error rate often indicate deployments gone wrong or upstream dependencies failing.

On the Application Errors dashboard, click Add in the top toolbar, then select Visualization.
Select the Azure Monitor data source.
Set the service to Logs and select law-monitoring-central as the resource.
Enter the following KQL query:

ContainerAppConsoleLogs_CL
| where Log_s has_any ("error", "500", "401", "crash", "exception")
| summarize ErrorCount = count() by bin(TimeGenerated, 5m), ContainerAppName_s
| order by TimeGenerated asc

What does this query do?

This query uses the same error keyword filter as the table panel. The difference is the summarize operator, which groups matching logs into 5-minute time buckets and counts them per container app.

Grouping by ContainerAppName_s means each container app gets its own line on the graph. This lets you immediately identify which service is generating the most errors.

In the panel settings, confirm the visualization type is set to Time series (this should be the default).
Set the panel title to Error Rate by App.
Click Run query to preview the time series graph.
Click Apply to add the panel to the dashboard.

You should see a spike in the graph corresponding to when you hit the /status/500 and /status/401 endpoints. The line will be labeled with api-service since that is the only app generating errors.

Click the Save dashboard icon to save your changes.

Graph shows no data points?

Expand the time range to Last 1 hour. If your test errors were generated more than 15 minutes ago, the default time range may not include them.

If you still see nothing, generate a fresh batch of errors by hitting /status/500 a few more times and waiting 2-5 minutes for ingestion.

Help me debug why my Grafana time series panel shows no error data

Why separate error and rate panels?

The table gives you detail: exact timestamps, which app, and the full log message for each error. The time series gives you trend: is the error rate increasing, stable, or spiking after a deployment?

In production, you would typically spot a spike on the graph first, then drill into the table to see which specific errors are causing it.

Your Application Errors dashboard is now complete with both a detailed error log table and a trend visualization. Next, you will build a Container Infrastructure Error dashboard to catch crashes, image pull failures, and restart loops at the platform level.

Build the Container Infrastructure Error Dashboard

Your Application Errors dashboard catches problems at the HTTP level. But what about failures that happen before your app even starts serving requests?

Container infrastructure errors are a different class of problem. Image pull failures, crash loops, and provisioning timeouts all happen at the platform layer. In this step, you will build a dedicated dashboard for these infrastructure-level failures and then intentionally trigger one to confirm your visibility.

In this step, get ready to:

Create a Container Errors dashboard with a table panel for infrastructure failures.
Add a time series panel showing container restart count over time.
Test the dashboard by deploying a broken container image and verifying error entries appear.

Create the Container Errors dashboard with an infrastructure errors table

The ContainerAppSystemLogs_CL table captures platform-level events that happen outside your application code. Failed image pulls, container crashes, and revision provisioning errors all land here.

In Grafana, click the + icon in the left sidebar.
Click New dashboard.
Click Add visualization.
Select your Azure Monitor data source.
In the query editor, set the Service to Logs.
Select your law-monitoring-central workspace as the resource.
Enter the following KQL query:

ContainerAppSystemLogs_CL
| where Log_s contains "Error provisioning revision"
    or Log_s contains "ErrImagePull"
    or Log_s contains "ContainerCrashing"
    or Log_s contains "Timeout"
| project TimeGenerated, ContainerAppName_s, RevisionName_s, Log_s
| order by TimeGenerated desc

What does this query do?

This query filters the system logs table for the four most critical container infrastructure failure patterns. The project operator selects only the columns you need: when it happened, which app, which revision, and the full error message.

These four patterns cover the most common container platform failures: provisioning errors (revision failed to deploy), image pull errors (image not found or registry auth failure), crash loops (container keeps dying), and timeouts (container took too long to start).

In the panel settings on the right, change the visualization type to Table.
Set the panel title to Infrastructure Errors.
Click Run query to test the query.
Click Apply to save the panel to the dashboard.

The table might be empty right now. That is expected if your container apps have been running without issues.

Query returning an error?

If you see a "table not found" error, check that your Container Apps Environment diagnostic settings are sending logs to law-monitoring-central. The ContainerAppSystemLogs_CL table only appears after the first system event arrives.

help me troubleshoot why ContainerAppSystemLogs_CL is not appearing in my Log Analytics workspace

Add a restart count time series panel

A table shows you individual errors, but a time series reveals patterns. Repeated provisioning events clustered in a short window indicate a restart loop, which is one of the most dangerous container failure modes.

Click Add in the top toolbar, then select Visualization.
Select your Azure Monitor data source.
Set the Service to Logs and select law-monitoring-central as the resource.
Enter the following KQL query:

ContainerAppSystemLogs_CL
| where Log_s contains "provisioning" or Log_s contains "ContainerCrashing" or Log_s contains "ErrImagePull"
| summarize RestartCount = count() by bin(TimeGenerated, 5m), ContainerAppName_s
| order by TimeGenerated asc

What does this query do?

This query counts provisioning and crash events in 5-minute buckets, grouped by container app name. The summarize operator with bin(TimeGenerated, 5m) creates the time buckets that Grafana plots as a time series.

A healthy environment shows zero or very low counts. A spike in this graph means something is repeatedly failing to provision, which often indicates a broken image, misconfigured environment variable, or resource constraint.

In the panel settings, make sure the visualization type is set to Time series.
Set the panel title to Container Restart Count.
Click Run query to verify the query syntax.
Click Apply to add the panel to the dashboard.

Click the Save dashboard icon (floppy disk) in the top toolbar.
Name the dashboard Container Errors.
Click Save.

You should see your Container Errors dashboard with two panels: the Infrastructure Errors table and the Container Restart Count time series. Both panels may show no data yet, which is perfectly fine.

Time series panel showing 'No data'?

This is expected if your container apps have been running without issues. The next substep will generate real error data to populate both panels.

Test the dashboard with a broken container image

A monitoring dashboard that has never fired is a dashboard you cannot trust. You need to intentionally break something to confirm that your queries surface real failures.

In the Azure Portal, navigate to your rg-monitoring-lab resource group.
Click on frontend-service (your Container App).
In the left menu under Application, click Containers.
Click Edit and deploy.
Click on the existing container image name to edit it.
Change the Image name to:

mcr.microsoft.com/azuredocs/containerapps-helloworld:nonexistent

Why a nonexistent tag?

The tag nonexistent does not exist in the container registry. When Azure Container Apps tries to pull this image, it will fail with an ErrImagePull error, followed by an Error provisioning revision entry. These are exactly the events your dashboard queries watch for.

Click Save on the container edit pane.
Click Create to deploy the new revision with the broken image.

Azure will attempt to pull the nonexistent image and fail. Wait 3-5 minutes for the error logs to flow into Log Analytics.

Switch back to your Grafana browser tab.
Open your Container Errors dashboard.
Set the time range to Last 15 minutes using the time picker in the top right.
Click the refresh icon or press Ctrl+R to reload the panels.

You should see ErrImagePull and Error provisioning revision entries in the Infrastructure Errors table panel. The Container Restart Count time series should show a spike for frontend-service.

Still seeing empty panels after 5 minutes?

Log ingestion can occasionally take longer. Try extending the time range to Last 30 minutes. Also confirm that the revision deployment actually failed by checking the Revisions blade of frontend-service in the Azure Portal. You should see a failed revision with a provisioning error.

help me troubleshoot why my Container Errors dashboard is not showing ErrImagePull events

Now that you have confirmed the dashboard works, revert the frontend-service back to its correct image.

In the Azure Portal, navigate back to frontend-service.
Click Containers in the left menu under Application.
Click Edit and deploy.
Click on the container image name to edit it.
Change the Image name back to:

mcr.microsoft.com/azuredocs/containerapps-helloworld

Click Save, then click Create to deploy the corrected revision.
Wait for the new revision to show a Running status on the Revisions blade.

Your frontend-service is now restored to its healthy state with the correct container image.

Error history persists in the dashboard

Even after reverting the image, the error entries remain visible in your dashboard within the time range they occurred. This is intentional. Your dashboard serves as a historical record of all infrastructure failures, helping you correlate past issues with other events.

Your monitoring platform now covers both application-level and infrastructure-level container failures. Next up, you will wire everything together with automated alerting so these errors trigger email notifications instead of sitting silently in a dashboard.

Configure Grafana SMTP and Alerting

Your dashboards provide real-time visibility into VM health, network traffic, container status, and application errors. But dashboards only help when someone is watching them.

In this step, you will configure Grafana to send automated email alerts via Gmail SMTP. When a metric crosses a critical threshold, Grafana fires an alert and delivers a notification to your inbox without anyone needing to stare at a screen.

In this step, get ready to:

Configure Gmail SMTP in Grafana's configuration file.
Create a contact point and verify email delivery.
Create alert rules for CPU, Memory, Disk, and Swap thresholds.

Generate a Gmail App Password

Grafana needs credentials to send email through Gmail's SMTP server. Gmail requires a 16-character App Password for third-party applications that connect over SMTP.

Open a browser and go to myaccount.google.com/apppasswords.
Sign in with the Gmail account you want to use for alerting.
In the App name field, type Grafana SMTP.
Click Create.
Copy the 16-character password that Google displays. You cannot view it again after closing the dialog.

What is a Gmail App Password?

An App Password is a 16-character code that lets a third-party application authenticate with your Google account without exposing your primary password. It is only available when 2-Step Verification is enabled on your account.

Google displays the password with spaces between groups of four characters. The spaces are cosmetic. You can enter it with or without them.

Don't see the App Passwords option?

App Passwords require 2-Step Verification to be enabled. If you have 2FA enabled but still don't see the option, check whether your account uses only security keys for 2FA. Accounts enrolled in Google's Advanced Protection Program also cannot generate App Passwords.

help me generate a Gmail App Password for Grafana SMTP

Configure SMTP in the Grafana configuration file

Grafana reads its SMTP settings from /etc/grafana/grafana.ini. You need to edit this file on the grafana-server VM and restart the service for changes to take effect.

SSH into grafana-server using the public IP by running this command:

ssh [[SSH_USER="your-username"]]@[[GRAFANA_PUBLIC_IP="your grafana-server public IP"]]

Open the Grafana configuration file in a text editor by running:

sudo nano /etc/grafana/grafana.ini

Search for the [smtp] section by pressing Ctrl+W and typing [smtp].
Update the following lines in the [smtp] section. Remove the leading semicolons to uncomment each line:

[smtp]
enabled = true
host = smtp.gmail.com:587
user = your-gmail-address@gmail.com
password = your-16-character-app-password
from_address = your-gmail-address@gmail.com

What does this configuration do?

The enabled = true line activates SMTP support in Grafana. The host points to Gmail's SMTP server on port 587 (TLS).

The user and password fields authenticate with Gmail using your App Password. The from_address sets the sender email that appears in notifications.

Save the file by pressing Ctrl+O, then Enter, then Ctrl+X to exit nano.
Restart the Grafana service to apply the changes by running:

sudo systemctl restart grafana-server

Verify the service is running by running:

sudo systemctl status grafana-server

You should see active (running) in the output. This confirms Grafana has restarted with your SMTP configuration.

Grafana not starting after the edit?

Check that you removed the semicolons from the beginning of each line. Lines starting with ; are comments and Grafana ignores them.

Also confirm there are no extra spaces or typos in the host value. It must be exactly smtp.gmail.com:587 with no trailing whitespace.

help me fix Grafana not starting after editing grafana.ini SMTP settings

Create a contact point and send a test notification

A contact point tells Grafana where to send alert notifications. You will create one that delivers emails to your inbox.

Open Grafana in your browser using the grafana-server public IP on port 80.
In the left sidebar, click Alerting (the bell icon).
Click Contact points.
Click + Add contact point.
In the Name field, enter Gmail Ops Team.
In the Integration dropdown, select Email.
In the Addresses field, enter your email address.
Click Test to send a test notification.
Check your email inbox. You should receive a test notification from Grafana within a few seconds.
Click Save contact point.

Test notification not arriving?

Check your spam folder first. Gmail sometimes routes automated emails there.

If there is no email at all, SSH back into the grafana-server and check the Grafana logs for SMTP errors by running sudo journalctl -u grafana-server --since '5 minutes ago'. Common issues include an incorrect App Password or the from_address not matching your Gmail address.

help me debug why Grafana SMTP test notification is not arriving in my email

Create alert rules for VM performance thresholds

With email delivery confirmed, you can now create alert rules that trigger when VM metrics exceed safe thresholds. Each rule queries the Perf table in your Log Analytics workspace using KQL.

You will create four alert rules: CPU above 70%, Memory above 80%, Disk above 75%, and Swap above 80%.

In the left sidebar, click Alerting, then click Alert rules.
Click + New alert rule.
In the Name field, enter CPU > 70%.
In the Define query and alert condition section, select your Azure Monitor data source.
Set the Service to Logs.
Select your law-monitoring-central workspace as the resource.
Enter the following KQL query:

Perf
| where TimeGenerated > ago(5m)
| where ObjectName == "Processor" and CounterName == "% Processor Time" and InstanceName == "_Total"
| summarize avg(CounterValue) by bin(TimeGenerated, 1m)

What does this query do?

This KQL query filters the Perf table to the last 5 minutes of data. It looks for the % Processor Time counter for the _Total instance (overall CPU across all cores).

The summarize avg(CounterValue) line calculates the average CPU percentage per minute. Grafana evaluates the resulting numeric value against your threshold.

Add a Reduce expression. Set the function to Last.
Add a Threshold expression. Set the condition to Is above and enter 70.
In the Set evaluation behavior section, click + New evaluation group.
Name the group VM Performance Alerts and set the evaluation interval to 1m.
Set the Pending period to 5m.
In the notifications section, select the Gmail Ops Team contact point.
Click Save rule.

Why a 5-minute pending period?

The pending period prevents alerts from firing on brief CPU spikes. The condition must remain breached for 5 continuous minutes before Grafana transitions the alert to the Alerting state and sends a notification.

This avoids noisy false positives while still catching genuine sustained load.

Now create the remaining three alert rules using the same process.

Click + New alert rule again.
Name it Memory > 80%. Use this KQL query:

Perf
| where TimeGenerated > ago(5m)
| where ObjectName == "Memory" and CounterName == "% Used Memory"
| summarize avg(CounterValue) by bin(TimeGenerated, 1m)

Add a Reduce expression with function Last.
Add a Threshold expression set to Is above 80.
Select the VM Performance Alerts evaluation group (already set to 1m interval).
Set the Pending period to 5m.
Assign the Gmail Ops Team contact point and click Save rule.

Click + New alert rule.
Name it Disk > 75%. Use this KQL query:

Perf
| where TimeGenerated > ago(5m)
| where ObjectName == "LogicalDisk" and CounterName == "% Used Space" and InstanceName != "_Total"
| summarize avg(CounterValue) by bin(TimeGenerated, 1m)

Add a Reduce expression with function Last.
Add a Threshold expression set to Is above 75.
Select the VM Performance Alerts evaluation group.
Set the Pending period to 5m.
Assign the Gmail Ops Team contact point and click Save rule.

Click + New alert rule.
Name it Swap > 80%. Use this KQL query:

Perf
| where TimeGenerated > ago(5m)
| where ObjectName == "Memory" and CounterName == "% Used Swap Space"
| summarize avg(CounterValue) by bin(TimeGenerated, 1m)

Add a Reduce expression with function Last.
Add a Threshold expression set to Is above 80.
Select the VM Performance Alerts evaluation group.
Set the Pending period to 5m.
Assign the Gmail Ops Team contact point and click Save rule.

You should now see all four alert rules listed under the VM Performance Alerts evaluation group on the Alert rules page.

How does Grafana evaluate these alerts?

Grafana checks each rule every 1 minute (the evaluation interval). If the threshold condition is breached, the alert enters a Pending state. After 5 continuous minutes of the condition being true, it transitions to Alerting and sends a notification to your contact point.

When the condition drops back below the threshold, Grafana sends a Resolved notification and returns the alert to Normal state.

Alert rules showing No Data state?

If your alert rules show a No Data state, the KQL query is returning no results. This usually means the Perf table does not yet contain enough data from your VMs. Wait 5-10 minutes for the Azure Monitor Agent to collect and ingest performance counters.

You can test the query in Grafana Explore first. Select your Azure Monitor data source, choose Logs, and paste the KQL query to confirm it returns results.

help me troubleshoot Grafana alert rules showing No Data state for Azure Monitor Logs queries

Your monitoring platform now actively watches for problems. Next up, you will create additional alerts for NAT Gateway data transit and container crashes to complete your alerting coverage.

Create NAT Gateway Transit Alert and Container Crash Alerts

Your VM threshold alerts are now protecting against CPU, memory, disk, and swap issues. But infrastructure failures extend beyond individual servers.

In this step, you will create alerts for two critical scenarios: excessive outbound traffic through the NAT Gateway (which could indicate a compromised workload or runaway process) and container crashes in your Azure Container Apps environment. You will also test the container crash alert by deliberately breaking a container image to confirm the end-to-end notification pipeline works.

In this step, get ready to:

Create a NAT Gateway transit alert that fires when outbound data exceeds 5GB per hour.
Create container crash and restart loop alerts for your Container Apps.
Test the alerting pipeline by deploying a broken container image and verifying the email arrives.

Create the High NAT Gateway Transit alert rule

A sudden spike in outbound bytes through your NAT Gateway could mean a compromised VM is exfiltrating data or a misconfigured service is flooding an external API. This alert watches for outbound traffic exceeding 5GB in any rolling 1-hour window.

In Grafana, click the hamburger menu (three horizontal lines) in the top-left corner.
Navigate to Alerting, then click Alert rules.
Click + New alert rule.
Enter High NAT Gateway Transit as the alert rule name.

Now configure the query that fetches NAT Gateway metrics:

In the Define query and alert condition section, select your Azure Monitor data source.
Set the Service dropdown to Metrics.
Under Resource, select your subscription and navigate to rg-monitoring-lab to find the natgw-monitoring resource.
Set Metric to Bytes (the REST API name is ByteCount).
Set Aggregation to Total (Sum).
Under Dimension filters, add a filter for Direction and select only Out.
Set the time range to 1 hour so the query evaluates outbound bytes over a rolling 1-hour window.

Now define the threshold condition:

Add a Reduce expression if Grafana shows one. Set the function to Last and the input to the query (A).
Add a Threshold expression. Set the condition to IS ABOVE and enter 5368709120 as the value.

Why 5,368,709,120 bytes?

This is exactly 5GB calculated as 5 x 1024 x 1024 x 1024 = 5,368,709,120 bytes. If your NAT Gateway pushes more than 5GB outbound in a single hour, the alert fires.

Configure the evaluation behavior and contact point:

Under Set evaluation behavior, select or create a folder (e.g. Infrastructure Alerts) and an evaluation group.
Set the evaluation interval to 1 minute.
Set the pending period to 5 minutes.
Under Configure notifications, select Gmail Ops Team as the contact point.
Click Save rule.

What does this alert do?

This alert queries the Azure Monitor Metrics API for the NAT Gateway's ByteCount metric, filtered to outbound traffic only. If the sum of outbound bytes over 1 hour exceeds 5GB, the alert enters the Pending state. If it stays above that threshold for 5 consecutive minutes of evaluation, it fires and sends an email to the Gmail Ops Team contact point.

Don't see the natgw-monitoring resource in the dropdown?

Make sure you selected the correct subscription first. Then browse into the rg-monitoring-lab resource group and look under the Microsoft.Network/natGateways resource type.

Also confirm your Grafana server's Managed Identity has the Reader role on rg-monitoring-lab. Without this, Grafana cannot discover metrics for resources in the group.

help me troubleshoot why my NAT Gateway resource is not appearing in the Grafana alert rule resource picker

Create the Container Crash Alert and Container Restart Loop rules

Container crashes and restart loops are urgent operational events. A single crash might be transient, but repeated restarts indicate a broken deployment that needs immediate rollback. You will create two alert rules that query the ContainerAppSystemLogs_CL table in your Log Analytics Workspace.

Navigate to Alerting > Alert rules and click + New alert rule.
Enter Container Crash Alert as the rule name.
Select your Azure Monitor data source.
Set the Service dropdown to Logs.
Under Resource, select your law-monitoring-central Log Analytics Workspace.

Enter the following KQL query:

ContainerAppSystemLogs_CL
| where TimeGenerated > ago(5m)
| where Log_s contains "Error provisioning revision" or Log_s contains "ContainerCrashing"
| summarize CrashCount = count()

What does this query do?

This query looks at the last 5 minutes of container system logs. It filters for entries containing "Error provisioning revision" (image pull failures, config errors) or "ContainerCrashing" (runtime crashes). It then counts matching entries into a single numeric value that Grafana can evaluate against a threshold.

Grafana alerting requires numeric query results. The summarize operator aggregates the logs into a count that can trigger a threshold condition.

Add a Threshold expression. Set the condition to IS ABOVE and enter 0 as the value.
Under Set evaluation behavior, set the evaluation interval to 1 minute and the pending period to 0 seconds (fire immediately on any crash).
Assign the Gmail Ops Team contact point.
Click Save rule.

Now create the second rule for detecting restart loops:

Click + New alert rule again.
Enter Container Restart Loop as the rule name.
Select the Azure Monitor data source with Service set to Logs.
Select law-monitoring-central as the resource.

Enter the following KQL query:

ContainerAppSystemLogs_CL
| where TimeGenerated > ago(10m)
| where Log_s contains "Error provisioning revision" or Log_s contains "ContainerCrashing"
| summarize RestartCount = count() by ContainerAppName_s
| where RestartCount > 3

What does this query do?

This query looks at a 10-minute window and groups crash or provisioning error events by container app name. If any single container app has more than 3 such events in 10 minutes, the query returns a result. This indicates a restart loop where the container keeps crashing and restarting repeatedly.

Add a Reduce expression with function Last and input set to the query.
Add a Threshold expression. Set the condition to IS ABOVE and enter 0 as the value.
Set the evaluation interval to 1 minute and the pending period to 0 seconds.
Assign the Gmail Ops Team contact point.
Click Save rule.

You should now see three new alert rules listed alongside your existing VM threshold alerts.

Query returns no data or shows an error?

Make sure you selected Logs as the service type (not Metrics). The ContainerAppSystemLogs_CL table only appears in the Logs query editor.

If the table is not found, confirm your Container Apps Environment diagnostic settings are sending to law-monitoring-central. It can take a few minutes for system log data to appear after initial configuration.

help me troubleshoot why my KQL query against ContainerAppSystemLogs_CL is returning no results in my Grafana alert rule

Test the container crash alert

An untested alert is as bad as no alert at all. You will deliberately break one of your container apps by deploying a nonexistent image tag. This forces a provisioning failure that your Container Crash Alert should detect and notify you about.

In the Azure Portal, search for Container Apps and open frontend-service.
In the left menu, click Revisions and replicas.
Click Create new revision.
Click on the container name to edit it.
Change the Image field to a broken tag:

mcr.microsoft.com/azuredocs/containerapps-helloworld:nonexistent

Click Save on the container edit pane.
Click Create to deploy the new revision.

What happens next?

Azure Container Apps will attempt to pull the nonexistent tag, which does not exist. The revision will fail with an ErrImagePull or "Error provisioning revision" event. This event gets logged to ContainerAppSystemLogs_CL in your Log Analytics Workspace.

Your Container Crash Alert query runs every minute and checks for exactly these log entries. When it finds them, it fires immediately (0 second pending period) and sends an email notification to your Gmail.

Wait 3-5 minutes for the failed revision to generate system logs and for the Grafana alert evaluation cycle to pick them up.
Check your Gmail inbox for an alert notification from Grafana titled "Container Crash Alert".

Check the alert state in Grafana

While waiting for the email, navigate to Alerting > Alert rules in Grafana. You can see your Container Crash Alert transition from Normal to Alerting state in real time.

Once you have confirmed the alert fired and the email arrived, revert the broken image:

Go back to frontend-service > Revisions and replicas.
Click Create new revision.
Edit the container and restore the image to the correct value:

mcr.microsoft.com/azuredocs/containerapps-helloworld

Click Save, then Create to deploy the healthy revision.
Wait a minute, then confirm the frontend-service revision status shows Running again.

No email arrived after 5 minutes?

First, check the alert rule state in Grafana. If it still shows Normal, the ContainerAppSystemLogs_CL table may not have received the failure events yet. System logs can take 2-5 minutes to appear in Log Analytics.

If the alert shows Alerting but no email arrived, check your Gmail Ops Team contact point. Send a test notification from Alerting > Contact points to verify SMTP still works. Also check your Spam folder.

help me debug why my Grafana Container Crash Alert did not fire or did not send an email notification after I deployed a broken container image

Your alerting pipeline is now complete. You have threshold alerts for VM resources, a transit alert for the NAT Gateway, and crash detection for your container apps. Next, you will set up long-term log archival so your monitoring data survives beyond the 30-day retention window.

Configure Log Archival with Blob Storage

Your monitoring platform now captures everything in real time. But Log Analytics has a 30-day retention window on law-monitoring-central. After that window, your telemetry data disappears.

For compliance and long-term troubleshooting, you need an archival strategy that keeps data accessible for months or years without paying hot-storage prices. In this step, you will create an Azure Blob Storage account and configure automatic data export with lifecycle tiering.

In this step, get ready to:

Create a Storage Account for log archival.
Set up continuous data export from Log Analytics to the Storage Account.
Configure lifecycle management to tier and delete old data automatically.

Create the archival Storage Account

The data export feature streams selected tables from your workspace to a Storage Account in real time. The Storage Account must be in the same Azure region as your Log Analytics Workspace.

In the Azure Portal, search for Storage accounts in the top search bar and select the service.
Click + Create.
Select your subscription and resource group rg-monitoring-lab.
Enter a storage account name. Use stmonitoringarchive or a similar globally unique name (lowercase letters and numbers only, 3-24 characters).
Set Region to the same region as your rg-monitoring-lab resource group.
Set Performance to Standard.
Set Redundancy to Locally-redundant storage (LRS).
Click Next until you reach the Advanced tab.
Under Blob Storage, confirm the Default access tier is set to Hot.
Click Review + create, then click Create.

Wait for the deployment to complete. You should see a green checkmark and the message "Your deployment is complete."

Why LRS redundancy?

LRS keeps three copies of your data within a single data center. For archival logs that are primarily a backup of data already in Log Analytics, this provides sufficient durability at the lowest cost.

Storage Account name already taken?

Storage Account names are globally unique across all of Azure. If stmonitoringarchive is taken, append your initials or a short number (e.g., stmonitoringarchive01).

Make sure your name is lowercase only, with no hyphens or special characters.

Help me troubleshoot my Storage Account creation.

Create a data export rule from Log Analytics

Data export continuously streams new records from selected tables to your Storage Account as they arrive. Azure creates a separate blob container for each exported table automatically.

In the Azure Portal, search for Log Analytics workspaces and select law-monitoring-central.
In the left menu, scroll to the Settings section and click Data Export.
Click New export rule at the top of the pane.
Enter a rule name such as export-to-archive.
On the Source tab, select the following tables: Perf, Syslog, and ContainerAppConsoleLogs.
On the Destination tab, set the destination type to Storage Account.
Select your subscription and choose the stmonitoringarchive storage account (or whatever name you used).
Click Create.

The rule status shows as Enabled in the Data Export list. You should see your rule name, the destination type (Storage Account), and the enabled status.

How long until data starts flowing?

The export rule takes approximately 30 minutes to provision. After that, new records arriving in the selected tables are continuously exported to the Storage Account.

Azure creates a separate blob container in your Storage Account for each table (e.g., one for Perf, one for Syslog, one for ContainerAppConsoleLogs). You do not need to create these containers yourself.

Don't see a table in the Source list?

Only tables that currently contain data appear in the Source tab. If a table is missing, confirm that the corresponding resource is sending data to law-monitoring-central. For example, ContainerAppConsoleLogs only appears if your Container Apps have generated at least one log entry.

Also verify your Storage Account is in the same region as the workspace. Cross-region export is not supported.

Help me troubleshoot my data export rule.

Configure lifecycle management for automatic tiering

Lifecycle management automatically moves blobs between storage tiers based on their age. This keeps recent data quickly accessible while gradually shifting older data to cheaper tiers.

Navigate to your stmonitoringarchive Storage Account in the Azure Portal.
In the left menu, scroll to the Data management section and click Lifecycle management.
Select the List View tab, then click Add a rule.
On the Details form, enter a rule name such as archive-old-logs.
Set Rule scope to Apply rule to all blobs in your storage account.
Set Blob type to Block blobs.
Click Next to proceed to the Base blobs tab.

Now configure the three tiering actions on the Base blobs tab.

Check the box for Move to cool storage. Set Days after last modification to 30.
Check the box for Move to archive storage. Set Days after last modification to 90.
Check the box for Delete the blob. Set Days after last modification to 365.
Click Add to save the lifecycle rule.

Your new rule appears in the Lifecycle Management list with status Enabled.

What does this lifecycle rule do?

This creates a three-stage archival pipeline for your exported log data.

Days 0-30: Blobs stay in the Hot tier for fast access.
Days 30-90: Blobs move to the Cool tier. Lower storage cost, slightly higher access cost.
Days 90-365: Blobs move to the Archive tier. Lowest cost, but retrieval takes hours.
After 365 days: Blobs are permanently deleted.

Archive tier minimum retention

Data in the Archive tier must stay for at least 180 days. If deleted earlier, you incur an early deletion fee. Your rule moves data to Archive at 90 days and deletes at 365 days, so you are well within this requirement.

Don't see the lifecycle rule taking effect?

Lifecycle management policies can take up to 24 hours after creation before the first run begins. The rule will not move blobs immediately.

Also, your data export rule needs approximately 30 minutes to provision. Until data actually arrives in the Storage Account, there are no blobs for the lifecycle policy to act on.

Help me verify my lifecycle management rule is configured correctly.

Your monitoring platform now has a complete data lifecycle. Telemetry flows from your VMs and Container Apps into Log Analytics for real-time queries, exports continuously to Blob Storage for long-term retention, and automatically tiers down to minimize cost over time.

Secret mission

Add Grafana Annotations for Deployment Events

Your dashboards show real-time metrics, but they lack context. When a spike appears on a graph, how do you know if it was caused by a deployment, a restart, or something else entirely? In this secret mission, you will add Grafana annotations that overlay deployment events and VM restarts as vertical markers directly on your time series panels.

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project uses multiple Azure resources that incur approximately $4-6/day while active.

Cost warning

Your 4 VMs, NAT Gateway, Container Apps, and Log Analytics Workspace are all incurring charges right now. The NAT Gateway alone charges per hour plus per GB of data processed. If you are not actively using these resources, consider pausing or deleting them to avoid unexpected bills.

Resources you used:

4 Azure VMs (web-server-01, app-server-01, db-server-01, grafana-server).
NAT Gateway with Public IP (charged per hour + per GB of data processed).
3 Azure Container Apps (auth-service, api-service, frontend-service).
Log Analytics Workspace (data ingestion charges).
Storage Account (minimal cost for exported log data).
Virtual Network, subnets, and Network Security Groups.

Keep everything running

No action needed. Choose this if you are still actively building or experimenting with your monitoring platform.

Expect approximately $4-6/day in charges while all resources remain active.
Monitor your Azure Cost Management dashboard to track spending.
Your Grafana dashboards and alerts will continue running and sending email notifications for threshold breaches.

Pause - I'll come back to this later

Deallocate all VMs to stop compute charges. Your disks and configurations are preserved so you can restart anytime.

In the Azure Portal, navigate to each VM (web-server-01, app-server-01, db-server-01, grafana-server).
Click Stop on each VM and confirm deallocation.
The NAT Gateway still incurs hourly charges while provisioned, even with no traffic flowing through it.
Container Apps on the consumption plan cost nearly nothing at idle with no incoming requests.
To resume, start all 4 VMs from the portal. Grafana and Apache2 will come back up automatically since they are enabled as system services.

Delete - I don't want to use this again

Remove all project resources permanently by deleting the resource group. This is a single action that removes everything at once.

In the Azure Portal, search for Resource groups in the top search bar.
Click on rg-monitoring-lab.
Click Delete resource group at the top of the page.
Type the resource group name rg-monitoring-lab to confirm deletion.
Click Delete. This removes all 4 VMs, the NAT Gateway, Container Apps, Log Analytics Workspace, Storage Account, Virtual Network, and all associated resources.
Deletion may take several minutes to complete.

That's a Wrap!

Nice work! You just built a production-grade centralized monitoring and observability platform entirely from scratch in Azure.

You've learned how to:

Deployed a full Azure Monitor pipeline with Data Collection Rules, NAT Gateway traffic routing, and Azure Container Apps all feeding into a single Log Analytics Workspace.
Installed self-hosted Grafana OSS behind an Apache2 reverse proxy, connected it via Managed Identity, and built 6 live dashboards covering VM health, NAT Gateway transit, container logs, application errors, and infrastructure errors.
Configured Grafana alerting with Gmail SMTP notifications for CPU, memory, disk, and swap thresholds, container crash and restart loop detection, NAT Gateway transit alerts, and set up long-term log archival with Blob Storage lifecycle management.
Secret Mission: Added Grafana annotation overlays for deployment events and VM restart detection on your dashboards.

Ready to quiz yourself?