Azure Native Monitoring with Grafana

Deploy a self-hosted Grafana platform to monitor Azure VMs and containers.

Introduction

30 Second Summary

When a server runs out of memory at 2 AM or a container keeps crashing in a loop, the teams that recover fastest are the ones who see the problem before users report it. That visibility comes from a monitoring platform.

In this project, you will build a centralized monitoring and observability platform on Azure. You will deploy a self-hosted Grafana instance, connect it to Azure Monitor using Managed Identity, and create operational dashboards with real-time alerts.

What You'll Build

A production-grade monitoring platform that collects VM metrics, container logs, and network traffic data into a Log Analytics Workspace and visualizes everything through custom Grafana dashboards.

By the end of this project, you'll have:

A self-hosted Grafana instance secured behind an Apache2 reverse proxy, authenticated to Azure via Managed Identity.
A monitored workload VM running Azure Monitor Agent with Data Collection Rules for CPU, Memory, Disk, Swap, and Syslog.
6 operational dashboards covering Infrastructure metrics, Container Metrics, Container Live Logs, HTTP Errors, VM Syslog, and NAT Gateway Traffic.
An alerting system with email notifications for VM threshold breaches, container crash detection, and NAT Gateway data transit exceeding 2 GB/hour.
Secret Mission: Add Azure Blob Storage archival with lifecycle management to automatically move logs from Hot to Cool to Archive storage.

Are there any prerequisites?

You need an Azure account (free tier or pay-as-you-go). Every step is done through the Azure Portal UI. The only time you use a terminal is to SSH into your VMs to install Grafana and Apache2.

Create Core Azure Resources

Every monitoring platform needs a home for its data. Before you can collect metrics, logs, or alerts, you need two foundational pieces: a Resource Group to organize everything, and a Log Analytics Workspace to receive all the telemetry.

In this step, you will sign into the Azure Portal and create both of these resources. Everything in the project builds on top of them.

In this step, get ready to:

Create a resource group to organize all your monitoring resources.
Create a Log Analytics workspace to collect telemetry data.
Note the Workspace ID for later configuration steps.

Sign in and create the resource group

A resource group is a container that holds all related Azure resources for a project. Think of it like a folder that keeps your VMs, databases, and networking resources grouped together so you can manage them as a single unit.

Open your browser and navigate to portal.azure.com.
Sign in with your Azure account credentials.
In the search bar at the top of the portal, type Resource groups and select Resource groups from the results.
Click Create.
Under Resource group, enter monitoring-rg.
Under Region, select a region close to you (for example, East US or West Europe).
Click Review + Create, then click Create.

Remember your region

Write down the region you selected. All other resources in this project must be created in the same region to avoid cross-region latency and to meet Data Collection Rule requirements.

You should see a notification that says Resource group created. The deployment takes just a few seconds.

Don't see the Create button?

Make sure you have an active Azure subscription. If your free trial has expired, you may need to upgrade to pay-as-you-go. Check the Subscriptions section in the portal to confirm your subscription is active.

Help me troubleshoot my Azure subscription

Create the Log Analytics workspace

A Log Analytics workspace is a centralized store where Azure Monitor sends all collected metrics, logs, and performance data. You query this data using KQL (Kusto Query Language), and it powers both your Grafana dashboards and alert rules.

In the portal search bar at the top, type Log Analytics workspaces and select Log Analytics workspaces from the results.
Click Create.
Under Resource group, select monitoring-rg from the dropdown.
Under Name, enter monitoring-law.
Under Region, select the same region you used for monitoring-rg.
Click Review + Create, then click Create.

The deployment takes about 30-60 seconds. Wait for the notification banner to confirm the workspace is ready.

Why a Log Analytics workspace?

Log Analytics is the central nervous system of Azure Monitor. Every piece of telemetry in this project (VM performance counters, container logs, syslog messages) flows into this workspace.

The first 5 GB of data ingestion per month is free for all customers. For this project, you will stay well within that limit.

Deployment failed?

Check that you selected the same region as your resource group. If the name monitoring-law is taken (unlikely), try appending a number like monitoring-law-01.

Help me troubleshoot my Log Analytics workspace deployment

Note the Workspace ID

You will need the Workspace ID later when connecting services to this workspace. Let's grab it now while you are on the right page.

Click Go to resource in the deployment notification, or search for monitoring-law in the portal search bar.
On the Overview page, find the Workspace ID field.
Copy this value and save it somewhere you can easily access (a note, a text file, or your clipboard).

What is the Workspace ID?

The Workspace ID is a unique identifier (a long string of letters and numbers) that other Azure services use to route data to your specific workspace. It is different from the workspace name.

Your foundational infrastructure is in place. Next up, you will deploy the Grafana VM and configure it with an Apache2 reverse proxy so you can access your dashboards securely through a browser.

Deploy the Grafana VM with Apache2 Reverse Proxy

Your core Azure resources are ready. Now you need a place to run Grafana, the open-source visualization tool that will display all your monitoring data in custom dashboards.

In this step, you will create a dedicated Azure virtual machine, install Grafana on it, and place it behind an Apache2 reverse proxy so the dashboards are accessible on port 80 without exposing Grafana's default port 3000 to the internet.

In this step, get ready to:

Create the Grafana VM with managed identity and a network security group.
Install Grafana OSS and enable Azure managed identity authentication.
Configure Apache2 as a reverse proxy to serve Grafana on port 80.

Create the Grafana VM in the Azure Portal

This VM will host your Grafana installation. Enabling a system-assigned managed identity during creation lets the VM authenticate to Azure Monitor later without storing any credentials.

In the Azure Portal search bar at the top, type Virtual machines and select it from the results.
Click Create, then select Azure virtual machine.
On the Basics tab, set Resource group to monitoring-rg.
Enter grafana-vm for the Virtual machine name.
For Image, select Ubuntu Server 22.04 LTS - Gen2.
For Size, choose B1s or B2ats v2 (both covered by the Azure free tier for 750 hours/month).
Under Administrator account, set Authentication type to SSH public key.
Enter azureuser for Username.
Leave SSH public key source as Generate new key pair and enter a name like grafana-key.
Under Inbound port rules, select Allow selected ports.
Choose SSH (22) and HTTP (80) from the dropdown.

Why SSH and HTTP only?

Port 22 lets you connect to install software. Port 80 is where Apache2 will serve Grafana to your browser.

Grafana's default port 3000 stays internal, never exposed to the internet.

Click the Management tab at the top of the page.
Under Identity, check the box for Enable system assigned managed identity.
Click Review + create at the bottom of the page.
Review the summary, then click Create.

What happens with the SSH key?

A popup will appear asking you to download the private key file (a .pem file). Click Download private key and create resource.

Save this file somewhere you can find it easily. You will need it to SSH into the VM.

Wait for the deployment to complete, then click Go to resource.
On the VM Overview page, find and copy the Public IP address. You will use this to SSH in and to access Grafana later.

What is a managed identity?

A system-assigned managed identity gives your VM an automatically managed identity in Azure Active Directory. Grafana can use this identity to authenticate with Azure Monitor without needing you to create API keys or store secrets.

Don't see B1s or B2ats v2 in your region?

B1s is being retired in some Azure regions. If you don't see it, select B2ats v2 instead. Both offer 750 free hours per month for 12 months.

Help me find a free tier VM size in my Azure region

Install Grafana OSS and enable managed identity

With the VM running, you will SSH in and install Grafana from its official APT repository. Then you will edit the Grafana config to enable managed identity authentication for Azure services.

Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
Type Terminal (macOS) or PowerShell (Windows) and press Enter to open it.
Set the correct permissions on your downloaded key file by running this command:

chmod 400 [[PEMPATH="path/to/your/grafana-key.pem"]]

What does chmod 400 do?

SSH requires your private key file to be readable only by you. The chmod 400 command sets the file to read-only for your user and no access for anyone else. Without this, SSH will refuse to use the key.

Connect to your Grafana VM by running:

ssh -i [[PEMPATH="path/to/your/grafana-key.pem"]] azureuser@[[GRAFANA_VM_IP="your grafana-vm public IP address"]]

When prompted "Are you sure you want to continue connecting?", type yes and press Enter.

You should now see a terminal prompt showing azureuser@grafana-vm.

Permission denied or connection refused?

Make sure you ran chmod 400 on the correct .pem file. Double-check the IP address matches the Public IP address shown on the VM's Overview page in the portal.

Help me troubleshoot SSH connectivity to my Azure VM

Install Grafana OSS from the official APT repository by running these commands:

# Import the Grafana GPG key
sudo mkdir -p /etc/apt/keyrings
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.asc > /dev/null

# Add the Grafana stable repository
echo "deb [signed-by=/etc/apt/keyrings/grafana.asc] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list

# Update package list and install Grafana
sudo apt-get update
sudo apt-get install grafana -y

What do these commands do?

The first two commands add Grafana's GPG signing key so your system trusts packages from their repository.
The echo command adds the Grafana APT repository to your sources list.
The final two commands refresh the package index and install Grafana OSS 13.x (the latest stable version).

Now edit the Grafana configuration file to enable managed identity by running:

sudo nano /etc/grafana/grafana.ini

Press Ctrl+W to open the search bar inside nano.
Type [azure] and press Enter to jump to the Azure section.
Find the line that says ;managed_identity_enabled = false.
Remove the semicolon at the start of the line (the semicolon means the line is commented out).
Change false to true, so the line reads:

managed_identity_enabled = true

Press Ctrl+O then Enter to save the file.
Press Ctrl+X to exit nano.

Why enable managed identity in Grafana?

This setting tells Grafana to use the VM's managed identity when connecting to Azure data sources. Instead of storing API keys or secrets, Grafana authenticates transparently through the VM's identity.

This is the most secure approach recommended by both Microsoft and Grafana.

Can't find the [azure] section?

The grafana.ini file is long. Make sure you use Ctrl+W to search rather than scrolling manually. If the [azure] section does not exist, scroll to the bottom of the file and add two new lines: [azure] on one line, and managed_identity_enabled = true on the next.

Help me add the azure section to grafana.ini manually

Configure Apache2 as a reverse proxy for Grafana

Grafana listens on localhost:3000 by default. Instead of opening port 3000 to the internet, you will put Apache2 in front of it as a reverse proxy.

Apache2 will listen on port 80 and forward all traffic to Grafana internally.

Install Apache2 by running:

sudo apt-get install apache2 -y

Enable the proxy modules that Apache2 needs by running:

# Enable the core proxy module and the HTTP proxy module
sudo a2enmod proxy
sudo a2enmod proxy_http

What are proxy modules?

Apache2 ships with proxy support disabled by default. The proxy module enables core proxy functionality. The proxy_http module adds HTTP-specific forwarding so Apache2 can relay requests to Grafana's HTTP interface.

Disable the default Apache2 site (you will replace it with a Grafana proxy config) by running:

sudo a2dissite 000-default.conf

Create the Grafana reverse proxy VirtualHost configuration by running:

sudo tee /etc/apache2/sites-available/grafana.conf > /dev/null <<EOF
<VirtualHost *:80>
    ProxyPreserveHost On
    ProxyPass / http://localhost:3000/
    ProxyPassReverse / http://localhost:3000/
</VirtualHost>
EOF

What does this VirtualHost config do?

ProxyPreserveHost On passes the original host header through to Grafana so it knows the address used to access it.
ProxyPass / http://localhost:3000/ forwards all incoming requests on port 80 to Grafana on port 3000.
ProxyPassReverse / http://localhost:3000/ rewrites response headers so redirects from Grafana point back to port 80 instead of port 3000.

Enable the new Grafana site and start both services by running:

# Enable the Grafana VirtualHost
sudo a2ensite grafana.conf

# Start and enable Grafana to run on boot
sudo systemctl start grafana-server
sudo systemctl enable grafana-server

# Restart Apache2 to load the new proxy config
sudo systemctl restart apache2
sudo systemctl enable apache2

What does enable mean here?

The systemctl enable command ensures a service starts automatically when the VM reboots. Without it, you would need to manually start Grafana and Apache2 every time the VM restarts.

Open a web browser on your local machine.
Navigate to http:// followed by your Grafana VM's public IP address (the one you copied from the Overview page).

You should see the Grafana login page. This confirms Apache2 is correctly proxying traffic from port 80 to Grafana on port 3000.

Don't log in yet

You will log into Grafana and configure data sources in Step 5. For now, just confirming the login page appears is enough.

Not seeing the Grafana login page?

Check that both services are running by executing sudo systemctl status grafana-server and sudo systemctl status apache2. Both should show "active (running)".

If Apache2 shows an error, check the proxy modules are loaded by running apache2ctl -M | grep proxy. You should see proxy_module and proxy_http_module in the output.

Also verify the NSG allows inbound traffic on port 80. Go to the VM's Networking tab in the portal and confirm both port 22 and port 80 have Allow rules.

Help me troubleshoot my Apache2 reverse proxy and Grafana connectivity

Your Grafana VM is live and accessible through the browser. Next up, you will deploy the workload VM that Grafana will actually monitor, complete with a NAT Gateway and the Azure Monitor Agent.

Deploy the Workload VM with NAT Gateway and Azure Monitor Agent

Your Grafana instance is live and ready to visualize data. Now you need something to actually monitor.

In this step, you'll deploy a workload VM that sits behind a NAT Gateway for outbound internet access. You'll then install the Azure Monitor Agent on it and configure a Data Collection Rule to stream performance metrics and syslogs into your Log Analytics workspace.

In this step, get ready to:

Create a virtual network, NAT Gateway, and workload VM.
Install the Azure Monitor Linux Agent on the workload VM.
Configure a Data Collection Rule for performance counters and syslogs.

Create the virtual network, NAT Gateway, and workload VM

The workload VM needs its own network and a NAT Gateway for outbound internet access. Since you won't assign a public IP to this VM, the NAT Gateway handles all outbound connections on its behalf.

What is a NAT Gateway?

A NAT Gateway provides outbound internet connectivity for resources in a subnet that don't have their own public IP addresses. It translates private IP addresses to a single public IP for outbound traffic.

This is a common production pattern. Your workload VM stays hidden from inbound internet traffic while still being able to reach external services.

First, create the virtual network:

In the Azure Portal search bar at the top, type Virtual networks and select the service from the results.
Click Create.
Set the resource group to monitoring-rg.
Enter workload-vnet as the name.
Select the same region you used for your other resources.
Click Next until you reach the IP Addresses tab.
Set the address space to 10.0.0.0/16.
Add a subnet named workload-subnet with the range 10.0.1.0/24.
Click Review + Create, then Create.

Wait for the deployment to complete. Next, create the NAT Gateway:

In the portal search bar, type NAT gateways and select the service.
Click Create.
Set the resource group to monitoring-rg.
Enter workload-natgw as the name.
Select the same region as your virtual network.
Click Next: Outbound IP.
Under Public IP addresses, click Create a new public IP address.
Give it a name (e.g. natgw-pip) and click OK.
Click Next: Subnet.
Select virtual network workload-vnet and check the box next to workload-subnet.
Click Review + Create, then Create.

Why attach the NAT Gateway to a subnet?

Any VM placed in workload-subnet without its own public IP will automatically route outbound traffic through the NAT Gateway. This means your workload VM can download packages and send telemetry without being directly reachable from the internet.

Now create the workload VM inside this network:

In the portal search bar, type Virtual machines and select the service.
Click Create > Azure virtual machine.
Set the resource group to monitoring-rg.
Enter workload-vm as the name.
Select Ubuntu Server 22.04 LTS as the image.
Choose size B1s or B2ats v2 (both covered by Azure free tier).
Set authentication type to SSH public key.
Click the Networking tab.
Set Virtual network to workload-vnet.
Set Subnet to workload-subnet.
Set Public IP to None.
Under the NSG settings, create a new NSG that allows inbound Port 22 (SSH) only.
Click Review + Create, then Create.

Why no public IP on the workload VM?

In production environments, backend workloads rarely need a public IP. By routing outbound traffic through the NAT Gateway instead, the VM can still reach the internet (for updates, telemetry, etc.) without exposing any inbound ports to the public.

Wait for the VM deployment to finish. You should see a Your deployment is complete message.

SSH access to the workload VM

Since the workload VM has no public IP, you cannot SSH into it directly from your local machine. If you need SSH access later (for example, to stress-test the CPU), you can use Azure Bastion or connect from the Grafana VM if both VNets are peered. For now, all configuration happens through the portal.

Don't see B1s or B2ats v2 in the size list?

B1s is being retired in some Azure regions. If you don't see it, select B2ats v2 instead. Both are covered under the Azure free tier (750 hours/month for 12 months). You can also click See all sizes to filter by the B-series.

Still having trouble finding a free-tier size?

Install Azure Monitor Linux Agent on the workload VM

The Azure Monitor Agent (AMA) collects telemetry from the VM and sends it to your Log Analytics workspace. You can install it directly from the portal without needing SSH access.

Navigate to your workload-vm in the portal.
In the left menu under Settings, click Extensions + applications.
Click Add.
Search for Azure Monitor Linux Agent and select it from the results.
Click Next, then Review + Create, then Create.
Wait for the extension status to show Provisioning succeeded. This may take 1-2 minutes.

What is the Azure Monitor Agent?

The Azure Monitor Agent (AMA) is a lightweight service that runs on your VM and collects performance data, event logs, and other telemetry. It replaces older agents like the Log Analytics agent.

The agent itself does not decide what to collect. That job belongs to a Data Collection Rule, which you'll create next.

Extension stuck on Transitioning?

It can take a few minutes for the agent to fully install. Refresh the Extensions page by clicking Refresh in the toolbar. If it stays stuck for more than 5 minutes, try removing the extension and adding it again.

My Azure Monitor Agent extension won't finish installing. What should I check?

Create a Data Collection Rule for performance counters and syslogs

A Data Collection Rule (DCR) tells the Azure Monitor Agent exactly what telemetry to collect and where to send it. Without a DCR, the agent sits idle.

In the portal search bar, type Monitor and select the Monitor service.
In the left menu, click Data Collection Rules (under Settings).
Click Create.
Enter vm-perf-syslog-dcr as the rule name.
Set the resource group to monitoring-rg.
Select Linux as the platform type.
Click Next: Resources.
Click Add resources and select workload-vm from the list.
Click Next: Collect and deliver.

Now configure the data sources. You'll add two: performance counters and syslogs.

Click Add data source.
Select Performance Counters as the data source type.
Check all available counters: CPU, Memory, Disk, Swap, and Network.
Under Destination, select your monitoring-law Log Analytics workspace.
Click Add data source to save.

Now add the Syslog data source:

Click Add data source again.
Select Linux Syslog as the data source type.
Set the minimum log level to LOG_WARNING (this collects warnings and above, filtering out noisy informational messages).
Set the destination to monitoring-law.
Click Add data source to save.
Click Review + Create, then Create.

What do these data sources collect?

Performance Counters collect numerical metrics like CPU usage percentage, available memory, disk space, swap usage, and network bytes in/out. These are what you'll graph in your Grafana dashboards.

Syslog collects system-level event messages. By filtering to LOG_WARNING and above, you capture meaningful events (errors, service failures, security warnings) without drowning in routine messages.

After creating the DCR, verify it appears in the Data Collection Rules list with workload-vm as a target resource.

Data takes a few minutes to flow

Performance data won't appear immediately in your Log Analytics workspace. Allow 5-10 minutes for the agent to start collecting and shipping telemetry. You'll verify data is flowing when you build your Grafana dashboards later.

DCR creation fails or workload-vm not showing in resource list?

Make sure the DCR is being created in the same region as your Log Analytics workspace. The DCR and its destination must share the same region. Also confirm that the Azure Monitor Agent extension shows Provisioning succeeded on the workload VM before creating the DCR.

My Data Collection Rule isn't associating with my workload VM. What am I missing?

Your monitoring infrastructure is now in place. The workload VM is streaming performance metrics and syslogs to your Log Analytics workspace through the Azure Monitor Agent. Next up, you'll deploy container workloads to give yourself even more telemetry to visualize.

Deploy Azure Container Apps

Your workload VM is now fully instrumented with Azure Monitor Agent collecting performance metrics and syslogs.

Azure Container Apps let you run microservices without managing servers. In this step, you'll deploy two sample apps and connect their logs directly to your Log Analytics workspace so all your telemetry flows into one place.

In this step, get ready to:

Deploy two Container Apps running sample microservices.
Connect container logs to your Log Analytics workspace.
Verify log data is flowing with a KQL query.

Create the api-service Container App

You'll create your first Container App along with a new Container Apps environment. The environment is a shared boundary that groups your apps and connects them to your logging infrastructure.

In the Azure Portal search bar at the top, type Container Apps and select it from the results.
Click Create.
Under Basics, set the Resource group to monitoring-rg.
Enter api-service as the Container app name.

Now you need to create the Container Apps environment that will host both of your apps.

Under the Container Apps Environment section, click Create new.
Enter monitoring-env as the environment name.
Click the Monitoring tab within the environment creation panel.
Set the Logs destination to Azure Log Analytics.
Select your monitoring-law Log Analytics workspace from the dropdown.
Click Create to finish creating the environment.

What is a Container Apps environment?

A Container Apps environment is a secure boundary around a group of container apps. Apps in the same environment share the same virtual network and logging configuration.

By pointing the environment's logs destination to your monitoring-law workspace, every container app you deploy into it automatically sends logs there.

Now configure the container image and networking for this app.

Back on the main Create Container App page, click the Container tab.
Uncheck Use quickstart image if it is checked.
Set Image source to Docker Hub or other registries.
Enter mcr.microsoft.com/k8se/quickstart:latest in the Image and tag field.
Click the Ingress tab.
Enable Ingress by checking the box.
Set Ingress traffic to Accepting traffic from anywhere (external).
Set the Target port to 80.
Click Review + Create, then click Create.

Wait for the deployment to complete. You should see a Your deployment is complete message.

Deployment failed?

Make sure the image name is exactly mcr.microsoft.com/k8se/quickstart:latest with no extra spaces. Also confirm your resource group is set to monitoring-rg and that you selected the same region as your other resources.

Help me troubleshoot my Container App deployment failure.

Create the frontend-service Container App

Your second container app simulates a frontend microservice. It uses the same environment you just created, so its logs automatically flow to the same workspace.

Go back to the Container Apps service page in the Azure Portal.
Click Create.
Set the Resource group to monitoring-rg.
Enter frontend-service as the Container app name.
Under Container Apps Environment, select the existing monitoring-env environment from the dropdown.
Click the Container tab.
Uncheck Use quickstart image if it is checked.
Set Image source to Docker Hub or other registries.
Enter mcr.microsoft.com/k8se/quickstart:latest in the Image and tag field.
Click the Ingress tab.
Enable Ingress by checking the box.
Set Ingress traffic to Accepting traffic from anywhere (external).
Set the Target port to 80.
Click Review + Create, then click Create.

Wait for the deployment to complete. You should see the same Your deployment is complete confirmation.

Why two container apps?

Running two apps simulates a real microservice architecture where a frontend communicates with a backend API. This gives you realistic log data to monitor, including separate log streams for each service.

Verify container logs are flowing

Your container apps should now be sending console logs to the Log Analytics workspace. You'll confirm this by running a KQL query.

In the Azure Portal search bar, type Log Analytics workspaces and select it.
Click on monitoring-law to open your workspace.
In the left sidebar, click Logs.
Close any welcome dialog or sample queries popup that appears.
Clear any existing text in the query editor and paste the following KQL query:

ContainerAppConsoleLogs_CL | take 10

Click Run.

What does this query do?

KQL (Kusto Query Language) is the query language used in Azure Log Analytics. This query pulls the first 10 rows from the ContainerAppConsoleLogs_CL table, which stores all stdout and stderr output from your container apps.

If you see results in the table below the query editor, your container logs are flowing correctly.

No results returned?

It can take 5-10 minutes for the first container app logs to appear in Log Analytics. Wait a few minutes and click Run again.

If logs still don't appear after 10 minutes, go back to your monitoring-env Container Apps environment and confirm the Logs destination shows your monitoring-law workspace.

Help me troubleshoot why my Container App logs aren't appearing in Log Analytics.

Both container apps are now running and sending logs to your centralized workspace. Next up, you'll connect Grafana to Azure Monitor so you can visualize all this telemetry in custom dashboards.

Configure Grafana and Connect Azure Monitor

Your monitoring infrastructure is fully deployed. You have a Grafana VM, a workload VM collecting metrics via Azure Monitor Agent, Container Apps sending logs, and a NAT Gateway handling outbound traffic.

Now you need to connect Grafana to all that telemetry data. Grafana will authenticate to Azure Monitor using the VM's Managed Identity. This means no passwords or API keys to manage. Azure trusts the VM's identity automatically.

In this step, get ready to:

Grant the Grafana VM's Managed Identity the Monitoring Reader role on your subscription.
Log into Grafana and add the Azure Monitor data source using Managed Identity authentication.
Verify the connection with a successful Save & Test.

Assign the Monitoring Reader role to your Grafana VM

For Grafana to read metrics and logs from Azure Monitor, the VM's identity needs permission. The Monitoring Reader role gives read-only access to all monitoring data across your subscription.

In the Azure Portal, search for Subscriptions in the top search bar.
Click on your subscription name to open it.
In the left menu, click Access control (IAM).
Click Add at the top, then select Add role assignment.

In the Role tab, search for Monitoring Reader and select it.
Click Next.
In the Members tab, select Managed identity under "Assign access to".
Click + Select members.
In the dropdown for Managed identity, choose Virtual machine.
Select grafana-vm from the list, then click Select.
Click Review + assign twice to save the role assignment.

Why Monitoring Reader instead of a broader role?

Monitoring Reader grants read-only access to monitoring data (metrics, logs, alert definitions). It follows the principle of least privilege. Grafana only needs to query data, never modify it.

Assigning it at the subscription level means Grafana can see metrics from every resource in your subscription. This saves you from adding permissions one resource at a time.

You should see a notification confirming the role assignment was added successfully.

Don't see grafana-vm in the managed identity list?

Make sure you selected Virtual machine in the Managed identity dropdown (not App Registration or another type). If grafana-vm still doesn't appear, go back to the VM's Identity blade under Settings and confirm that System assigned is set to On.

Help me troubleshoot my Grafana VM managed identity not appearing in the IAM role assignment screen.

Log into Grafana and add the Azure Monitor data source

With permissions in place, you can now configure Grafana to use the Azure Monitor data source. This data source is built into Grafana OSS. No plugin installation is needed.

Open your web browser and navigate to http:// followed by your Grafana VM's public IP address (the one shown on the VM's Overview page in the portal).
You should see the Grafana login page.

Enter admin for both the username and password.
Grafana will prompt you to change the default password. Enter a new password and click Submit.

Why does Grafana use admin/admin by default?

Grafana ships with default credentials so you can access the UI immediately after installation. It forces a password change on first login to prevent the default from remaining in use.

You should now see the Grafana home page. Next, add the Azure Monitor data source.

In the left sidebar, click Connections.
Click Add new connection.
Search for Azure Monitor in the search field.
Click on Azure Monitor in the results.
Click Add new data source in the top right.

Under the Authentication section, select Managed Identity from the dropdown.
Leave all other fields at their default values.

Why Managed Identity instead of App Registration?

Since Grafana runs on an Azure VM, the VM's system-assigned managed identity can authenticate directly to Azure Monitor. There are no client secrets or certificates to create, rotate, or store.

App Registration requires you to generate a client secret and paste it into Grafana. That secret expires and must be rotated manually. Managed Identity eliminates that operational burden entirely.

Test the connection

Click Save & Test at the bottom of the data source configuration page.

You should see a green success message confirming that Grafana can connect to Azure Monitor.

Seeing a red error on Save & Test?

If you get an authentication error, the IAM role assignment may not have propagated yet. Azure role assignments can take up to 5 minutes to take effect. Wait a few minutes and click Save & Test again.

If the error persists, go back to your subscription's Access control (IAM) page, click Role assignments, and confirm you see grafana-vm listed with the Monitoring Reader role. Also verify that Managed Identity (not App Registration) is selected in the Grafana data source authentication dropdown.

Help me troubleshoot my Grafana Azure Monitor data source connection failing with an authentication error.

Grafana is now connected to Azure Monitor. It can pull metrics from your VMs, NAT Gateway, and Container Apps, plus query logs from your Log Analytics workspace. Next up, you'll build your first dashboard to visualize the workload VM's infrastructure metrics.

Build the VM Infrastructure Dashboard

Your Grafana instance is connected to Azure Monitor and ready to visualize real data from your infrastructure.

Right now, the Data Collection Rule on your workload VM is streaming CPU, memory, disk, swap, and network metrics into your Log Analytics workspace. In this step, you will build a dashboard that turns those raw numbers into panels you can glance at and instantly know the health of your VM.

In this step, get ready to:

Create CPU, memory, disk, and swap metric panels using the Virtual Machine Guest namespace.
Add a network traffic panel showing inbound and outbound bytes.
Add a heartbeat status panel using a KQL log query to show whether the VM is healthy or down.

Create the CPU metric panel

Each Grafana dashboard is made up of individual panels. Each panel runs a query against a data source and displays the result as a chart, gauge, or table. You will start with a CPU usage panel.

In the Grafana sidebar, click Dashboards.
Click New Dashboard.
Click Add visualization.
Select the Azure Monitor data source.
In the query editor at the bottom of the panel, choose Metrics as the query type.
Select your subscription from the Subscription dropdown.
Set Resource Group to monitoring-rg.
Set Resource Name to workload-vm.
Set Metric Namespace to Virtual Machine Guest.
Set Metric to Processor\% Processor Time.
Give the panel a title by clicking the default title text at the top and typing CPU Usage.
Click Apply in the top right corner to add the panel to your dashboard.

You should see a time-series chart showing CPU usage percentage for your workload VM.

What is the Virtual Machine Guest namespace?

Azure Monitor provides two metric namespaces for VMs. The default platform namespace shows basic host-level data. The Virtual Machine Guest namespace shows metrics collected from inside the operating system by the Azure Monitor Agent.

Your Data Collection Rule configured in Step 3 is what makes guest metrics available here. Without the DCR, this namespace would be empty.

Don't see any data in the panel?

Guest metrics can take 5-10 minutes to start flowing after the Data Collection Rule is associated. If you see "No data", wait a few minutes and click the refresh icon in the top right of the dashboard.

Also confirm you selected Virtual Machine Guest as the namespace, not the default platform namespace.

help me troubleshoot why my Grafana CPU panel shows no data for the workload VM

Add memory, disk, and swap panels

You will repeat the same process three more times to cover the remaining guest metrics. Each panel uses the same subscription, resource group, resource, and namespace. Only the metric name changes.

Click the Add button at the top of the dashboard, then select Visualization.
Select the Azure Monitor data source and set the query type to Metrics.
Set the subscription, resource group (monitoring-rg), resource (workload-vm), and namespace (Virtual Machine Guest) the same as before.
Set Metric to % Used Memory.
Title the panel Memory Usage and click Apply.
Add another visualization. Use the same settings, but set the metric to % Used Space.
Title it Disk Usage and click Apply.
Add one more visualization. Set the metric to Used Swap Space Percentage.
Title it Swap Usage and click Apply.

Your dashboard should now have four panels: CPU Usage, Memory Usage, Disk Usage, and Swap Usage. Each shows a time-series line chart with percentage values.

Rearrange panels by dragging

You can drag panels by their title bar to reposition them on the dashboard. Resize them by dragging the bottom-right corner. A 2x2 grid layout works well for the four metric panels.

Add the network traffic panel

Network metrics come from the VM's platform metrics namespace rather than the guest namespace. This means you will select a different namespace for this panel.

Click Add > Visualization.
Select the Azure Monitor data source and set the query type to Metrics.
Set the subscription, resource group (monitoring-rg), and resource (workload-vm).
For Metric Namespace, select the default platform namespace (microsoft.compute/virtualmachines) instead of Virtual Machine Guest.
Set Metric to Network In Total.
To add a second metric on the same panel, click Add query below the first query.
In the second query, set the same subscription, resource group, resource, and platform namespace.
Set the second metric to Network Out Total.
Title the panel Network Traffic and click Apply.

You should see two lines on this panel. One represents bytes received and the other represents bytes sent.

Why a different namespace for network metrics?

Network In Total and Network Out Total are platform-level metrics that Azure collects automatically from the hypervisor. They do not require the Azure Monitor Agent or a Data Collection Rule. The guest namespace only contains metrics the agent explicitly collects from inside the OS.

Add the VM heartbeat status panel

The final panel uses a KQL log query instead of a metric query. It checks the last heartbeat from the Azure Monitor Agent and labels the VM as "Healthy" or "Down".

Click Add > Visualization.
Select the Azure Monitor data source.
Change the query type to Logs.
Select your Log Analytics workspace (monitoring-law) as the resource.
Enter the following KQL query in the query editor:

Heartbeat
| where Computer == "workload-vm"
| summarize LastHeartbeat = max(TimeGenerated) by Computer
| project Computer, LastHeartbeat, Status = iff(LastHeartbeat > ago(5m), "Healthy", "Down")

What does this query do?

The Heartbeat table records a signal every time the Azure Monitor Agent checks in. This query finds the most recent heartbeat from workload-vm and labels it "Healthy" if the check-in was within the last 5 minutes, or "Down" if it was not.

Click Run query to confirm results appear.
In the visualization selector on the right side panel, change the visualization type from Table to Stat.
Title the panel VM Heartbeat Status and click Apply.

You should see a large stat panel displaying "Healthy" (assuming the workload VM is running).

Seeing "No data" on the heartbeat panel?

Make sure you selected Logs as the query type, not Metrics. Also verify that the resource is set to your Log Analytics workspace (monitoring-law), not the VM directly.

If the workload VM was recently started, the Heartbeat table may need a few minutes to populate.

help me troubleshoot why my Grafana heartbeat panel shows no data for workload-vm

Save the dashboard by clicking the floppy disk icon (or pressing Ctrl+S) in the top right.
Name the dashboard Infrastructure Dashboard and click Save.

Your Infrastructure Dashboard now has 6 panels: CPU Usage, Memory Usage, Disk Usage, Swap Usage, Network Traffic, and VM Heartbeat Status.

You now have a single pane of glass for your workload VM's health. Next up, you will build dashboards for your Container Apps to monitor application-level logs and errors.

Build the Container and Logs Dashboards

Your Infrastructure Dashboard gives you visibility into the workload VM's health. Now you need the same level of insight into your Azure Container Apps running in the monitoring-env environment.

In this step, you will build three dashboards focused on containers. One tracks resource consumption metrics, another streams live logs in near real-time, and the third isolates HTTP errors so you can spot problems before users report them.

In this step, get ready to:

Create a Container Metrics dashboard with CPU, Memory, Replica Count, and Restart Count panels.
Create a Container Live Logs dashboard with real-time log streaming.
Create an HTTP Errors dashboard to track 401, 403, 500, and 503 responses.

Create the Container Metrics dashboard

Platform metrics for Container Apps live under the microsoft.app/containerapps namespace in Azure Monitor. You will pull CPU, memory, replica count, and restart count into a single dashboard so you can see all container health indicators at a glance.

In Grafana, click Dashboards in the left sidebar.
Click New, then New Dashboard.
Click Add visualization.
Select your Azure Monitor data source.
Choose Metrics as the query type.
Select your subscription, then resource group monitoring-rg.
Select resource api-service, metric namespace microsoft.app/containerapps.
Select CPU Usage as the metric.
Click Apply in the top right to save this panel.

Target both container apps

You can add both api-service and frontend-service as separate queries in the same panel. When editing a panel, click + Query to add the second container app with the same metric. This shows both apps on one graph for easy comparison.

Now add three more panels to the same dashboard following the same process.

Click the + icon (top of the dashboard) to add a new panel.
Create a panel for Memory Usage using the same namespace and resource selection, choosing the Memory Usage metric.
Create a panel for Replica Count using the Replica Count metric.
Create a panel for Restart Count using the Restart Count metric.
Click the save icon (floppy disk) in the top right.
Name the dashboard Container Metrics and click Save.

You should now see four panels on your dashboard, each showing metrics for your container apps.

Panels showing no data?

Container Apps metrics can take a few minutes to populate after deployment. Make sure you selected the correct namespace (microsoft.app/containerapps) and resource (api-service or frontend-service). Also confirm the time range in the top-right corner covers at least the last 30 minutes.

Still stuck? Help me troubleshoot why my Container Metrics panels are empty in Grafana.

Create the Container Live Logs dashboard

Metrics show you trends over time, but sometimes you need to see exactly what your containers are printing to stdout right now. A live logs panel streams console output from both container apps so you can watch application behavior in real time.

Click Dashboards in the left sidebar.
Click New, then New Dashboard.
Click Add visualization and select the Azure Monitor data source.
Switch the query type to Logs.
Select your monitoring-law workspace as the resource.
Enter the following KQL query in the query editor:

ContainerAppConsoleLogs_CL
| where TimeGenerated > ago(5m)
| project TimeGenerated, ContainerAppName_s, Log_s
| order by TimeGenerated desc

What does this query do?

ContainerAppConsoleLogs_CL is the table where Azure Container Apps send their stdout and stderr output. The query filters to the last 5 minutes, projects only the timestamp, app name, and log message, and sorts newest first.

Click Run query to verify results appear.
Click Apply in the top right to save the panel.

Now set the dashboard to auto-refresh so it streams new logs continuously.

Look at the top-right corner of the dashboard where you see the time range picker.
Click the refresh interval dropdown (next to the refresh icon).
Select 5s from the list.
Click the save icon (floppy disk) in the top right.
Name the dashboard Container Live Logs and click Save.

You should see the panel refresh every 5 seconds, pulling in the latest container log entries.

No logs appearing in the panel?

Container app logs can take 5-10 minutes to flow into Log Analytics after initial deployment. Visit your container app's URL in a browser to generate some traffic, wait a few minutes, then run the query again.

Also confirm you selected the correct workspace (monitoring-law) as the resource, not a VM or container app resource directly.

Need help? Help me troubleshoot why my Container Live Logs dashboard shows no results.

Create the HTTP Errors dashboard

The live logs dashboard shows everything. But in a production environment, you often want a focused view of just the errors. This dashboard isolates HTTP 401, 403, 500, and 503 responses so error spikes are immediately visible.

Click Dashboards in the left sidebar.
Click New, then New Dashboard.
Click Add visualization and select the Azure Monitor data source.
Switch the query type to Logs.
Select your monitoring-law workspace as the resource.
Enter the following KQL query to count errors in 1-minute bins:

ContainerAppConsoleLogs_CL
| where Log_s has "401" or Log_s has "403" or Log_s has "500" or Log_s has "503"
| summarize ErrorCount = count() by bin(TimeGenerated, 1m)
| order by TimeGenerated asc

What does this query do?

This query filters container logs for common HTTP error codes (401 Unauthorized, 403 Forbidden, 500 Internal Server Error, 503 Service Unavailable). It then groups the matches into 1-minute time bins and counts how many errors occurred in each bin.

The result is a time-series that shows error spikes clearly when visualized as a graph.

In the panel editor, change the visualization type to Time series using the dropdown in the top-right of the panel preview area.
Click Apply to save the panel.

Now add a second panel that shows the raw error log messages as a table.

Click the + icon at the top of the dashboard to add a new panel.
Select Azure Monitor data source, query type Logs, and select monitoring-law as the resource.
Enter the following KQL query for the raw error messages:

ContainerAppConsoleLogs_CL
| where Log_s has "401" or Log_s has "403" or Log_s has "500" or Log_s has "503"
| project TimeGenerated, ContainerAppName_s, Log_s
| order by TimeGenerated desc

Change the visualization type to Table.
Click Apply to save the panel.
Click the save icon (floppy disk) in the top right.
Name the dashboard HTTP Errors and click Save.

Your HTTP Errors dashboard now has two panels. The time-series graph shows error volume over time and the table below shows the full error messages.

What if there are no errors yet?

If your container apps have not received any requests that produce 401, 403, 500, or 503 responses, both panels will show empty results. This is normal. The dashboard will light up when errors occur in production traffic.

Query returns an error?

Check that the table name is exactly ContainerAppConsoleLogs_CL (case-sensitive, with the _CL suffix for custom log tables). Confirm you selected your Log Analytics workspace and not a different resource.

Need help? Help me fix my KQL query error in the HTTP Errors dashboard.

You now have three container-focused dashboards working alongside your Infrastructure Dashboard. Next up, you will build a dedicated dashboard for your NAT Gateway to track outbound data transit and connection health.

Build the NAT Gateway Traffic Dashboard

Your container and VM dashboards are live. Now it's time to monitor the network layer that connects your workload VM to the internet.

Your NAT Gateway handles all outbound traffic from the workload subnet. Without visibility into bytes transmitted, connection counts, and packet drops, you'd have no way to spot bandwidth spikes or connectivity issues before they cause real problems.

In this step, get ready to:

Create a NAT Gateway Traffic dashboard with a Bytes panel split by direction.
Add panels for SNAT connections, dropped packets, and datapath availability.
Add an hourly summary panel to visualize data transit approaching the 2 GB threshold.

Create the NAT Gateway Traffic dashboard with a Bytes panel

The ByteCount metric measures total bytes transmitted through the NAT Gateway. Splitting it by the Direction dimension (In/Out) shows inbound and outbound traffic as separate lines on the same chart.

In Grafana, click Dashboards in the left sidebar.
Click New Dashboard.
Click Add visualization.
Select the Azure Monitor data source.
Set the query type to Metrics.
Select your subscription, then set resource group to monitoring-rg.
Select the resource workload-natgw.
Set the metric to Bytes (REST API name: ByteCount).
Set the aggregation to Sum.
Under Dimension, select Direction to split the chart into In and Out as separate lines.
Give the panel a title like NAT Gateway Bytes (In/Out).
Click Apply in the top right to add the panel to your dashboard.

Why use Sum and not Count?

NAT Gateway metrics require Sum (Total) aggregation. The Count aggregation adds up the number of metric data points collected, not the actual byte values.

Using Count would give you meaningless numbers. Sum gives you the actual bytes transmitted through the gateway.

You should see a time-series chart with two lines representing inbound and outbound byte counts.

Not seeing any data on the chart?

NAT Gateway metrics only appear when there is actual traffic flowing through the gateway. SSH into your workload VM (via the Grafana VM as a jump box, or use Azure Bastion) and run a command like curl ifconfig.me to generate some outbound traffic.

Also confirm you selected the correct resource (workload-natgw) and that aggregation is set to Sum, not Count.

help me troubleshoot why my NAT Gateway ByteCount metric shows no data in Grafana

Add SNAT connections, dropped packets, and datapath availability panels

A healthy NAT Gateway needs more than just byte counts. Connection counts reveal port exhaustion risk, dropped packets signal congestion, and datapath availability tells you whether the gateway itself is up.

Click the Add button (top toolbar) and select Visualization.
Select the Azure Monitor data source and set query type to Metrics.
Select resource workload-natgw, metric SNAT Connection Count, aggregation Sum.
Title the panel SNAT Connection Count and click Apply.

Now add the dropped packets panel.

Add another visualization with the same Azure Monitor data source and Metrics query type.
Select resource workload-natgw, metric Dropped Packets (REST API name: PacketDropCount), aggregation Sum.
Title the panel Dropped Packets and click Apply.

Finally, add the datapath availability panel as a Gauge.

Add another visualization with the Azure Monitor data source and Metrics query type.
Select resource workload-natgw, metric Datapath Availability, aggregation Average.
In the panel settings on the right, change the visualization type from Time series to Gauge.
Set the max value to 100 so that 100% fills the gauge completely.
Title the panel Datapath Availability and click Apply.

What does Datapath Availability tell you?

Datapath Availability measures whether the NAT Gateway can successfully route traffic. A value of 100% means the gateway is fully operational. Any drop below 100% means some outbound connections are failing at the infrastructure level.

Your dashboard now has four panels. The Gauge should display 100% if the NAT Gateway is healthy.

Gauge panel showing no data or 0%?

Double check that you selected Average as the aggregation (not Sum or Count). If the NAT Gateway has had no traffic recently, availability data may not populate until traffic flows through the gateway.

help me fix my NAT Gateway Datapath Availability gauge panel in Grafana

Add an hourly ByteCount summary panel and save the dashboard

The first Bytes panel shows granular data at the default time grain. This hourly panel aggregates bytes per hour so you can visually spot when data transit approaches the 2 GB threshold you will alert on in the next step.

Add another visualization with the Azure Monitor data source and Metrics query type.
Select resource workload-natgw, metric Bytes (ByteCount), aggregation Sum.
Set the Time grain to 1 hour.
Title the panel Hourly Data Transit and click Apply.

Why track hourly data transit?

In the next step, you will create an alert that fires when NAT Gateway data exceeds 2 GB in one hour. This panel gives you the visual counterpart to that alert so you can spot trends before they trigger notifications.

Now save the dashboard.

Click the save icon (floppy disk) in the top toolbar.
Name the dashboard NAT Gateway Traffic.
Click Save.

Your NAT Gateway Traffic dashboard should now display five panels: Bytes (In/Out), SNAT Connection Count, Dropped Packets, Datapath Availability (Gauge), and Hourly Data Transit.

NAT Gateway metrics are stored for 93 days

All Azure platform metrics are retained for 93 days. NAT Gateway metrics cannot be exported to Log Analytics via diagnostic settings, which is why you query them directly through Azure Monitor Metrics in Grafana rather than using KQL log queries.

You now have a complete network visibility layer for your NAT Gateway. Next up, you will wire all of this data into automated alerts that notify you by email when thresholds are breached.

Configure Alerts and Email Notifications

Dashboards are great for visual monitoring, but you can't stare at them 24/7. The real power of a monitoring platform comes from alerting. Alerts proactively notify you when something goes wrong, so you can respond before users are affected.

In this step, you'll configure Azure Monitor alert rules that fire when your VM metrics cross dangerous thresholds or your containers crash. Each alert sends an email notification through an Action Group, which is Azure's way of defining who gets notified and how.

In this step, get ready to:

Create an Action Group with email notifications.
Set up metric alert rules for VM thresholds and NAT Gateway data transit.
Create log-based alert rules for container crashes and restart loops, then test your alerts.

Create the Action Group for email notifications

An Action Group defines what happens when an alert fires. You can send emails, trigger webhooks, run automation, or send SMS messages. For this project, you'll create one that sends email notifications to your inbox.

In the Azure Portal search bar, type Monitor and select Monitor from the results.
In the left sidebar, click Alerts.
Click Action groups in the top menu bar.
Click Create.
Set the resource group to monitoring-rg.
Enter monitoring-email-ag as the Action group name.
Enter monitoring-email-ag as the Display name.
Click Next: Notifications.
Under Notification type, select Email/SMS message/Push/Voice.
In the panel that opens, check the Email checkbox and enter your email address.
Click OK.
Give the notification a name like email-notification.
Click Review + Create, then Create.

What is an Action Group?

An Action Group is a reusable notification target in Azure Monitor. Instead of configuring email addresses on every single alert rule, you point all your rules at one Action Group. If you ever need to change who gets notified, you update it in one place.

You should see monitoring-email-ag listed in the Action groups view. Azure may also send a confirmation email to the address you entered.

Don't see your Action Group?

Make sure you selected monitoring-rg as the resource group. If you used a different resource group, the Action Group won't appear when you filter by monitoring-rg later.

Help me troubleshoot my Action Group creation in Azure Monitor.

Create metric alert rules for VM and NAT Gateway thresholds

Metric alerts watch a numeric value (like CPU percentage) and fire when it crosses a threshold you define. You'll create four alerts for your workload VM and one for the NAT Gateway.

In the Azure Portal, go to Monitor > Alerts.
Click Alert rules in the top menu, then click Create.
Under Scope, click Select a resource.
Filter by resource type Virtual machines and select workload-vm.
Click Apply.

Now configure the first alert condition for CPU usage.

Under Condition, click Add condition.
Search for and select the Percentage CPU metric.
Set the operator to Greater than.
Set the threshold value to 70.
Set the aggregation granularity (period) to 5 minutes.
Under Actions, click Add action group and select monitoring-email-ag.
Under Details, name the alert rule CPU > 70%.
Click Review + Create, then Create.

Repeat the same process to create three more alert rules for workload-vm. Use the following thresholds:

Memory > 80% over 5 minutes. Search for the Available Memory Bytes metric (or the guest metric equivalent for percentage used memory).
Disk > 75% over 5 minutes. Select the disk usage percentage metric.
Swap > 80% over 5 minutes. Select the swap usage percentage metric.

Attach each alert rule to the monitoring-email-ag Action Group.

Speed up alert creation

After creating the first alert rule, you can stay on the Alert rules page and click Create again. The scope often stays pre-filled with the same resource, saving you a few clicks.

Now create the NAT Gateway data transit alert.

Click Create to start a new alert rule.
Under Scope, select your NAT Gateway resource workload-natgw.
Under Condition, search for and select the Bytes metric (REST API name: ByteCount).
Set the aggregation to Sum.
Set the operator to Greater than.
Set the threshold value to 2147483648 (this equals 2 GB in bytes).
Set the aggregation granularity to 1 hour.
Under Actions, attach monitoring-email-ag.
Name the rule NAT Gateway Data > 2 GB/hour.
Click Review + Create, then Create.

Why use Sum aggregation for NAT Gateway metrics?

Azure NAT Gateway metrics require Sum (Total) aggregation. Do NOT use Count aggregation. Count adds up the number of metric data points, not the actual byte values. Sum gives you the real total bytes transmitted.

Go to Monitor > Alerts > Alert rules. You should see 5 alert rules listed (CPU, Memory, Disk, Swap, NAT Gateway).

Missing alert rules?

Make sure you're viewing alert rules for the correct subscription. Use the subscription filter at the top of the Alert rules page to confirm. Also verify each rule shows a status of Enabled.

Help me troubleshoot missing alert rules in Azure Monitor.

Create log-based alert rules and test your alerts

Log-based alerts use KQL queries to search your Log Analytics data. When the query returns results, the alert fires. You'll create two alerts that detect container problems.

In Monitor > Alerts, click Create to start a new alert rule.
Under Scope, select your Log Analytics workspace monitoring-law as the target resource.
Under Condition, select Custom log search.
Paste the following KQL query for detecting container crashes:

ContainerAppSystemLogs_CL | where Log_s has "ContainerCrashing"

Set the threshold to Greater than 0 (fire whenever any result is returned).
Set the evaluation frequency and lookback period as desired (e.g., every 5 minutes, looking back 5 minutes).
Under Actions, attach monitoring-email-ag.
Name the rule Container Crash Alert.
Click Review + Create, then Create.

Now create the second log-based alert for container restart loops.

Create another alert rule with monitoring-law as the scope.
Under Condition, select Custom log search and paste this KQL query:

ContainerAppSystemLogs_CL | where Log_s has "Creating a new revision" | summarize Count = count() by bin(TimeGenerated, 10m) | where Count > 3

What does this query detect?

This query groups revision-creation events into 10-minute windows. If more than 3 new revisions are created within any single 10-minute period, it means containers are restarting in a loop. That pattern usually indicates a crash-restart cycle that needs investigation.

Set the threshold to Greater than 0.
Attach monitoring-email-ag as the Action Group.
Name the rule Container Restart Loop Alert.
Click Review + Create, then Create.

You should now see 7 alert rules total in Monitor > Alerts > Alert rules: 4 VM metric alerts, 1 NAT Gateway metric alert, and 2 log-based alerts.

Now test that your CPU alert actually works by generating a CPU spike on the workload VM.

SSH into your workload-vm. Since this VM has no public IP, you'll need to SSH through the Grafana VM or use Azure Bastion if configured.
Install the stress testing tool by running:

sudo apt install stress-ng -y

Start a CPU stress test that runs for 5 minutes by running:

stress-ng --cpu 2 --timeout 300

What does this command do?

This command spins up 2 CPU-intensive worker processes for 300 seconds (5 minutes). That's enough to push CPU usage above 70% on a B1s or B2ats v2 VM, which should trigger your CPU alert rule.

Wait 5-10 minutes after the stress test completes.
Check your email inbox for an alert notification from Azure Monitor.
In the portal, go to Monitor > Alerts to see the fired alert in the list.

How long until the alert fires?

Azure Monitor evaluates metric alerts every minute by default. With a 5-minute aggregation window, the alert needs 5 minutes of sustained high CPU before firing. Add another 1-2 minutes for notification delivery. Total wait: approximately 7-10 minutes.

No email notification received?

Check your spam or junk folder. Azure Monitor alert emails come from azure-noreply@microsoft.com. Also verify that the Action Group shows your correct email address by going to Monitor > Alerts > Action groups > monitoring-email-ag.

If the alert rule itself hasn't fired, confirm the CPU actually spiked by checking your Infrastructure Dashboard in Grafana.

Help me troubleshoot why my Azure Monitor CPU alert isn't firing or sending email notifications.

Secret mission

Add Azure Blob Storage Log Archival

Your monitoring platform captures real-time metrics and logs. But what happens to that data after 30 days? In production, compliance requirements often mandate storing logs for months or even years. In this secret mission, you'll add long-term log archival with automated lifecycle management that moves data through Hot, Cool, and Archive tiers before deleting it after a year.

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Some resources in this project are covered by the Azure free tier, but the NAT Gateway incurs costs regardless of usage.

Cost warning

The NAT Gateway costs approximately $0.045/hour (around $32/month if running 24/7) plus $0.045/GB of data processed.

If you are not actively using this project, delete the NAT Gateway resource to stop charges immediately.

Resources you used:

Grafana VM (grafana-vm): covered by free tier (750 hours/month for B1s or B2ats v2).
Workload VM (workload-vm): covered by free tier (750 hours/month for B1s or B2ats v2).
NAT Gateway (workload-natgw): approximately $0.045/hour plus data processing fees. No free tier.
Log Analytics Workspace (monitoring-law): first 5 GB/month ingestion is free.
Container Apps (api-service and frontend-service): first 180,000 vCPU-seconds and 360,000 GiB-seconds per month are free.
Storage Account (Secret Mission only): Hot tier storage costs approximately $0.018/GB/month.

Keep everything running

No action needed. Choose this if you are still actively building or experimenting with your monitoring platform.

Monitor your Azure cost alerts to stay within your budget.
Remember that the NAT Gateway charges approximately $0.045/hour even when idle.
Both VMs are covered under the free tier for the first 12 months (750 hours/month each).
Container Apps scale to zero automatically when not receiving traffic.

Pause - I'll come back to this later

Shut down running resources to stop compute charges while keeping your configuration intact.

Go to each VM (grafana-vm and workload-vm) in the portal, click Overview, then click Stop to deallocate and stop compute charges.
Delete the NAT Gateway resource (workload-natgw) to stop its hourly charges. You can recreate it later.
Container Apps scale to zero automatically, so no action is needed for those.
Your Log Analytics Workspace and dashboards remain intact at no additional cost (within the 5 GB/month free tier).

Delete - I don't want to use this again

Remove all project resources permanently. This deletes everything in one action.

In the Azure Portal, search for Resource groups.
Click on monitoring-rg.
Click Delete resource group.
Type the resource group name monitoring-rg to confirm, then click Delete.
This removes all VMs, the NAT Gateway, the Log Analytics Workspace, Container Apps, and any storage accounts created during the project.

Nice Work!

You just built a production-grade monitoring and observability platform entirely through the Azure Portal. Your environment now has centralized telemetry collection, real-time dashboards, and automated alerting across VMs, containers, and network infrastructure.

You've learned how to:

Deploy a self-hosted Grafana instance secured behind an Apache2 reverse proxy and connected to Azure Monitor via Managed Identity authentication.
Build 6 operational dashboards covering VM infrastructure metrics, Container Apps logs, HTTP error tracking, and NAT Gateway traffic monitoring.
Configure metric alert rules and log-based alert rules with email notifications via Action Groups for CPU, memory, disk, swap, container crashes, restart loops, and NAT Gateway data transit thresholds.
Secret Mission: Add Azure Blob Storage archival with automated lifecycle management that moves logs from Hot to Cool to Archive tier and then deletes them.

Ready to quiz yourself?