Azure Monitoring with Self-Hosted Grafana

Build a centralized Azure monitoring platform with 17 Grafana dashboards.

Introduction

30 Second Summary

When a server runs out of disk space at 2am, you want to know before your users do. Monitoring platforms are how DevOps and SRE teams keep production systems healthy.

In this project, you will build a centralized monitoring platform in Microsoft Azure. You will monitor Linux VMs and containerized microservices, visualize data on 17 real-time dashboards, and set up automated email alerts.

What You'll Build

A production-simulation monitoring and observability platform that collects telemetry from Azure VMs and Container Apps, visualizes everything through self-hosted Grafana dashboards, alerts you when things go wrong, and archives old logs to save money.

By the end of this project, you'll have:

A centralized Log Analytics Workspace receiving metrics and logs from a Linux VM, NAT Gateway, and 3 Container Apps.
A self-hosted Grafana instance with 17 operational dashboards covering CPU, memory, disk, swap, container logs, application errors, and NAT Gateway metrics.
An automated alerting system with 5 alert rules that send email notifications when resources exceed thresholds, plus a cost-effective log archival strategy using Blob Storage lifecycle management.
Secret Mission: Add a Network Traffic Dashboard showing VM bandwidth in Mbps, and create a Container Restart Loop alert.

Are there any prerequisites?

You need an Azure subscription (Free Trial with $200 credit works perfectly) and an SSH client, which is already built into Linux and macOS terminals. No prior Azure experience is required.

This project costs approximately $7-18/day while resources are running. Full cleanup instructions are included so you can delete everything when you are done.

Create the Foundation

Every monitoring platform needs infrastructure to stand on before it can collect a single data point.

In this step, you will create the three foundational building blocks that everything else in this project depends on. Think of it as laying the foundation before building a house.

In this step, get ready to:

Create a Resource Group to organize all your Azure resources.
Create a Virtual Network with two subnets for your VMs.
Create a Log Analytics Workspace as the central data store for all monitoring telemetry.

Create a Resource Group

A Resource Group is a container that holds all related Azure resources together. When you are done with the project, you delete this one group and everything inside gets deleted too.

Open your browser and navigate to portal.azure.com.
Sign in with your Azure account.
In the search bar at the top, type Resource groups and click on the result.
Click + Create.
Select your subscription from the Subscription dropdown.
In the Resource group field, type rg-monitoring-lab.
In the Region dropdown, select East US.
Click Review + create at the bottom.
Click Create.

Use the same region for everything

You can choose a different region if one is closer to you, but you must use the SAME region for every resource in this project. This guide uses East US for all examples.

You should see a confirmation that your resource group was created successfully.

Create a Virtual Network

A Virtual Network (VNet) is a private network in the cloud. Your VMs need to live on a network, just like computers at home connect to a Wi-Fi router.

You will create two subnets inside this network. One subnet is for your monitored VMs and the other is for your Grafana dashboard server.

In the search bar at the top, type Virtual networks and click on the result.
Click + Create.

On the Basics tab:

Select your subscription from the Subscription dropdown.
In the Resource group dropdown, select rg-monitoring-lab.
In the Virtual network name field, type vnet-monitoring.
In the Region dropdown, select the same region as your resource group (East US).
Click Next: IP Addresses.

On the IP Addresses tab, you should see a default address space of 10.0.0.0/16. Keep this as-is.

If you see a default subnet listed, delete it by clicking the trash icon next to it.
Click + Add a subnet.
In the Subnet name field, type snet-vms.
In the Subnet address range field, type 10.0.1.0/24.
Click Add.

Now add the second subnet:

Click + Add a subnet again.
In the Subnet name field, type snet-grafana.
In the Subnet address range field, type 10.0.2.0/24.
Click Add.

Finish creating the Virtual Network:

Click Review + create.
Click Create.

Why two subnets?

Separating resources into different subnets is a security best practice. Your monitored workload VMs live on snet-vms, and your Grafana dashboard server lives on snet-grafana.

This separation lets you apply different network security rules to each group later if needed.

You should see a confirmation that your Virtual Network deployment succeeded.

Don't see the subnets you created?

After the VNet deploys, click Go to resource and then click Subnets in the left menu under Settings. You should see both snet-vms and snet-grafana listed.

If one is missing, you can add it from this page by clicking + Subnet. Help me fix my subnets

Create a Log Analytics Workspace

The Log Analytics Workspace is the central database where ALL your monitoring data ends up. VM metrics, container logs, syslog entries. Everything gets sent here.

You will query this workspace later to build dashboards and configure alerts.

In the search bar at the top, type Log Analytics workspaces and click on the result.
Click + Create.
Select your subscription from the Subscription dropdown.
In the Resource group dropdown, select rg-monitoring-lab.
In the Name field, type law-monitoring-lab.
In the Region dropdown, select the same region (East US).
Click Review + create.
Click Create.

After the workspace finishes creating, you need to configure the data retention period:

Click Go to resource.
In the left menu under Settings, click Usage and estimated costs.
Click Data Retention.
Set the slider to 30 days.
Click OK.

Why set data retention to 30 days?

Log Analytics charges for the amount of data stored. Keeping 30 days of data gives you enough history for meaningful dashboards and alerts without running up costs.

Later in the project, you will set up Blob Storage lifecycle management to archive older logs at a fraction of the cost.

You should see the retention setting update successfully on the Usage and estimated costs page.

Workspace creation failed?

Make sure you selected the same region (East US) as your resource group. Also verify that the workspace name law-monitoring-lab is available. Log Analytics workspace names must be globally unique.

If the name is taken, try appending random numbers (e.g. law-monitoring-lab-2847). Help me troubleshoot my Log Analytics Workspace

Your foundation is locked in. Next up, you will deploy the Linux VM and NAT Gateway that this monitoring platform will watch.

Deploy the Monitored Linux VM and NAT Gateway

Your foundation is in place. You have a resource group, a virtual network with two subnets, and a Log Analytics Workspace ready to receive data.

Now you need something to actually monitor. In this step, you will create a Linux Virtual Machine that simulates a production server, and a NAT Gateway that tracks all outbound network traffic leaving your network.

In this step, get ready to:

Deploy a Linux VM running Ubuntu Server 22.04 LTS with SSH access.
Create a NAT Gateway to track all outbound internet traffic from your VM.
Verify both resources are running and connected correctly.

Create the Linux VM

This VM simulates a production server. Later, you will install a monitoring agent on it and watch its CPU, memory, disk, swap, and logs in real time.

In the Azure Portal search bar at the top, type Virtual machines and click on it.
Click + Create, then select Azure virtual machine.

On the Basics tab, fill in the following fields:

Set Subscription to your subscription (you should only have one).
Set Resource group to rg-monitoring-lab.
Set Virtual machine name to vm-workload-01.
Set Region to the same region as your resource group (East US).
Set Availability options to No infrastructure redundancy required.
Set Security type to Standard.
Click the Image dropdown and select Ubuntu Server 22.04 LTS - x64 Gen2.
Click See all sizes, search for B1s, and select Standard_B1s (1 vCPU, 1 GB RAM).
Set Authentication type to SSH public key.
Set Username to azureuser.
Set SSH public key source to Generate new key pair.
Leave the Key pair name as the default or type vm-workload-01-key.

Now switch to the Networking tab:

Set Virtual network to vnet-monitoring.
Set Subnet to snet-vms (10.0.1.0/24).
Leave Public IP as (new).
Under Select inbound ports, check SSH (22).

Leave all other tabs as defaults.
Click Review + create at the bottom.
Click Create.

A popup appears asking you to download the private key. Click Download private key and create resource.

Save your key file securely

The .pem file that downloads is your only way to connect to this VM via SSH. If you lose it, you cannot log into the VM.

Save it somewhere you will remember, like your Downloads folder. You will use it in a later step.

Why SSH public key instead of a password?

SSH key authentication is more secure than passwords because it uses cryptographic key pairs that are virtually impossible to brute-force. In production environments, password authentication on servers is typically disabled entirely.

Deploy a NAT Gateway

A NAT Gateway sits between your VMs and the internet. All outbound traffic flows through it, which lets you track exactly how much data is leaving your network.

In the search bar, type NAT gateways and click on it.
Click + Create.

On the Basics tab:

Set Subscription to your subscription.
Set Resource group to rg-monitoring-lab.
Set NAT gateway name to natgw-monitoring.
Set Region to the same region (East US).
Set Availability zone to No Zone.
Leave TCP idle timeout at 4 minutes.

Click Next: Outbound IP:

Under Public IP addresses, click Create a new public IP address.
Set the name to pip-natgw.
Click OK.

Click Next: Subnet:

Set Virtual network to vnet-monitoring.
Check the box next to snet-vms.

Click Review + create.
Click Create.

What just happened?

By associating the NAT Gateway with the snet-vms subnet, all outbound internet traffic from vm-workload-01 now flows through the NAT Gateway. Azure automatically tracks bytes, packets, and connections at this point.

Verify everything is working

Before moving on, confirm that both resources are running and connected to each other correctly.

In the search bar, type Virtual machines and click on it.
Click vm-workload-01.
Confirm the Status shows Running on the Overview page.
Note the Public IP address shown on the Overview page. You will need this later for SSH access.

Now verify the NAT Gateway is associated with your subnet:

In the search bar, type NAT gateways and click on it.
Click natgw-monitoring.
In the left menu under Settings, click Subnets.
Verify that snet-vms is listed.

Finally, confirm that Azure is tracking metrics on the NAT Gateway:

Still on natgw-monitoring, click Metrics in the left menu under Monitoring.
Select the metric Bytes.
Set Aggregation to Sum.
Verify that the chart loads. It may show 0 for now, and that is normal.

VM status not showing Running?

If your VM shows a status other than Running, wait 2-3 minutes and refresh the page. VM provisioning can take a moment to complete.

If it shows Failed, check that you selected the correct region and that your subscription has enough quota for the Standard_B1s size.

My VM failed to deploy or shows an unexpected status.

Your monitored infrastructure is live. The VM is running and the NAT Gateway is tracking all outbound traffic. Next up, you will tell Azure exactly what data to collect from this VM by configuring a Data Collection Rule.

Configure Data Collection with Azure Monitor Agent

Your VM is running and your NAT Gateway is tracking outbound traffic. But right now, Azure is not actually collecting any detailed telemetry from inside the VM.

In this step, you will create a Data Collection Rule (DCR) that tells Azure exactly what data to collect and where to send it. The best part is that when you associate a VM with a DCR, Azure automatically installs the Azure Monitor Agent on the VM for you. No SSH required for this part.

In this step, get ready to:

Create a Data Collection Rule for performance counters (CPU, memory, disk, swap, network).
Add Linux Syslog collection to the same rule.
Verify that Azure Monitor Agent installed itself automatically on the VM.

Create a Data Collection Rule for performance counters

A Data Collection Rule defines two things: what data to collect, and where to send it. You will configure counters for CPU, memory, swap, disk, and network traffic.

In the Azure Portal search bar at the top, type Monitor and click on it.
In the left menu, scroll down and click Data Collection Rules (under Settings).
Click + Create.

On the Basics tab, fill in the following:

Enter dcr-vm-performance in the Rule name field.
Select your subscription in the Subscription dropdown.
Select rg-monitoring-lab in the Resource group dropdown.
Select East US in the Region dropdown (same region as all your other resources).
Select Linux in the Platform Type dropdown.

Now you need to tell Azure which VM this rule applies to.

Click Next: Resources.
Click + Add resources.
Expand your subscription, then expand rg-monitoring-lab.
Check the box next to vm-workload-01.
Click Apply.

What is a Data Collection Rule?

A DCR is the modern way Azure tells its monitoring agent what to watch. You define the metrics and logs you care about, Azure installs the agent, and data starts flowing into your Log Analytics Workspace automatically.

This replaces the older Log Analytics agent which was deprecated in August 2024.

Next, you will add the performance counter data source. This tells the agent which system metrics to collect.

Click Next: Collect and deliver.
Click + Add data source.
Select Performance Counters in the Data source type dropdown.
Click the Custom tab (not Basic).

Check the following counters and set the sample rate to 60 seconds for each one:

Under the Processor section, check % Processor Time.
Under the Memory section, check % Used Memory and % Used Swap Space.
Under the Logical Disk section, check % Used Space.
Under the Network section, check Total Bytes Transmitted and Total Bytes Received.

Now configure where this data gets sent.

Click Next: Destination.
Select Azure Monitor Logs as the Destination type.
Select law-monitoring-lab in the Account or namespace dropdown.
Click Add data source.

Why these specific counters?

These six counters give you full visibility into the four resource types that cause most server problems: CPU saturation, memory exhaustion, disk filling up, and network bandwidth issues. At 60-second intervals, you get frequent enough data for troubleshooting without flooding your workspace.

Add Syslog collection to the Data Collection Rule

Performance counters tell you how the machine is doing. Syslogs tell you what the machine is doing. You need both for complete monitoring.

Still on the Collect and deliver tab, click + Add data source again.
Select Linux Syslog in the Data source type dropdown.

Configure the minimum log level for each facility. Set the following:

Set LOG_AUTH to LOG_INFO.
Set LOG_AUTHPRIV to LOG_INFO.
Set LOG_DAEMON to LOG_WARNING.
Set LOG_KERN to LOG_WARNING.
Set LOG_SYSLOG to LOG_INFO.
Set all other facilities to None (do not collect).

Why different log levels per facility?

AUTH and AUTHPRIV are set to INFO because you want to see all login attempts, both successful and failed. DAEMON and KERN are set to WARNING because they generate a lot of noise at INFO level, and you only care about actual problems.

Now point this second data source to the same workspace.

Click Next: Destination.
Select Azure Monitor Logs as the Destination type.
Select law-monitoring-lab in the Account or namespace dropdown.
Click Add data source.

Both data sources are now configured. Time to create the rule.

Click Review + create.
Click Create.

Wait 5-10 minutes

After creating the DCR, Azure needs time to install the monitoring agent on your VM and begin collecting data. Wait at least 5 minutes before proceeding to the verification step.

Verify the agent installed automatically

When you associated vm-workload-01 with the DCR, Azure automatically installed the Azure Monitor Linux Agent on the VM. Let's confirm it worked.

In the search bar, type Virtual machines and click on it.
Click vm-workload-01.
In the left menu, click Extensions + applications (under Settings).

You should see AzureMonitorLinuxAgent listed with Status: Provisioning succeeded.

Don't see AzureMonitorLinuxAgent yet?

The agent can take 5-10 minutes to install after the DCR is created. Wait another 5 minutes, then refresh the page. If it still does not appear, go back to Monitor > Data Collection Rules > dcr-vm-performance > Resources and confirm vm-workload-01 is listed.

help me troubleshoot why AzureMonitorLinuxAgent is not appearing on my VM

Now confirm that data is actually flowing into your workspace.

Still on vm-workload-01, click Logs in the left menu (under Monitoring).
Close any welcome popup that appears.
Click the Tables tab in the query editor.
Expand the Virtual machines category.
Click Run next to Heartbeat.

You should see heartbeat records appearing in the results table. This confirms your agent is running and sending data to law-monitoring-lab.

Heartbeat query returns empty results?

Performance counter data and heartbeats can take an additional 5-10 minutes to appear after the agent installs. Wait 10 minutes and run the query again. Also confirm the VM has outbound internet access, which it needs to communicate with Azure Monitor endpoints.

help me figure out why my Heartbeat query shows no results

Your VM is now fully instrumented. Every 60 seconds, the agent collects CPU, memory, disk, swap, and network metrics. Syslog entries stream in as they happen. Next up, you will deploy three containerized applications to give your monitoring platform even more to watch.

Deploy Azure Container Apps

Your monitoring platform can already see a single VM. In production, most companies also run containerized microservices that need monitoring too.

In this step, you will deploy 3 Azure Container Apps that simulate a real microservices environment. Their logs will automatically flow to your Log Analytics Workspace, giving you container-level visibility alongside your VM data.

In this step, get ready to:

Create a Container Apps Environment linked to your Log Analytics Workspace.
Deploy three container apps simulating auth, API, and frontend services.
Verify all three apps are running and generating logs.

Create a Container Apps Environment

The Environment is the shared home for all your container apps. Every app in the same environment shares logging configuration, which means you only configure the log destination once.

In the Azure Portal search bar, type Container Apps Environments and click on it.
Click + Create.
On the Basics tab, select your subscription.
Set Resource group to rg-monitoring-lab.
Enter cae-monitoring-lab in the Environment name field.
Set Region to the same region you have been using (East US).
Set Environment type to Consumption only.
Click the Monitoring tab.
Set Logs destination to Azure Log Analytics.
Set Log Analytics workspace to law-monitoring-lab.
Click Review + create, then click Create.

Why Azure Log Analytics instead of Azure Monitor?

When configuring the logs destination, you will see two options: "Azure Log Analytics" and "Azure Monitor". The "Azure Log Analytics" option directly links your environment to your workspace, making logs immediately queryable via tables like ContainerAppConsoleLogs_CL.

The "Azure Monitor" option requires additional diagnostic settings configuration. For this project, "Azure Log Analytics" is the simpler and more direct path.

This deployment may take 2-3 minutes to complete.

Deploy three container apps

Each app uses a public Docker image from Docker Hub. No coding or container building is needed. You are simulating three microservices: an authentication service, an API service, and a frontend service.

You will create each app one at a time using the same general process, with different names and images.

Deploy auth-service

In the search bar, type Container Apps and click on it.
Click + Create.
On the Basics tab, select your subscription.
Set Resource group to rg-monitoring-lab.
Enter auth-service in the Container app name field.
Set Region to the same region (East US).
Set Container Apps Environment to cae-monitoring-lab.
Click the Container tab.
Uncheck Use quickstart image.
Set Image source to Docker Hub or other registries.
Enter nginx:alpine in the Image and tag field.
Set CPU and Memory to 0.25 CPU cores, 0.5 Gi memory.
Click the Ingress tab.
Check Enabled.
Set Ingress traffic to Accept traffic from anywhere.
Enter 80 in the Target port field.
Click Review + create, then click Create.

What is ingress?

Ingress controls whether your container app can receive incoming HTTP traffic from the internet. By enabling it and accepting traffic from anywhere, your app gets a public URL that anyone can visit. This is how you will generate access logs later for your dashboards.

Deploy api-service

Repeat the same process with these differences:

In the search bar, type Container Apps and click on it.
Click + Create.
On the Basics tab, set Resource group to rg-monitoring-lab.
Enter api-service in the Container app name field.
Set Region to the same region (East US).
Set Container Apps Environment to cae-monitoring-lab.
On the Container tab, uncheck Use quickstart image.
Set Image source to Docker Hub or other registries.
Enter httpd:alpine in the Image and tag field.
Set CPU and Memory to 0.25 CPU cores, 0.5 Gi memory.
On the Ingress tab, check Enabled.
Set Ingress traffic to Accept traffic from anywhere.
Enter 80 in the Target port field.
Click Review + create, then click Create.

Why httpd:alpine for the API service?

Using a different web server image (httpd:alpine is Apache HTTP Server) makes it easier to distinguish between services in your logs. The auth and frontend services use nginx:alpine, while the API service uses Apache. You will see different log formats from each.

Deploy frontend-service

In the search bar, type Container Apps and click on it.
Click + Create.
On the Basics tab, set Resource group to rg-monitoring-lab.
Enter frontend-service in the Container app name field.
Set Region to the same region (East US).
Set Container Apps Environment to cae-monitoring-lab.
On the Container tab, uncheck Use quickstart image.
Set Image source to Docker Hub or other registries.
Enter nginx:alpine in the Image and tag field.
Set CPU and Memory to 0.25 CPU cores, 0.5 Gi memory.
On the Ingress tab, check Enabled.
Set Ingress traffic to Accept traffic from anywhere.
Enter 80 in the Target port field.
Click Review + create, then click Create.

Verify all three apps are running

Now confirm that all three container apps deployed successfully and are generating logs.

In the search bar, type Container Apps and click on it.
Click on auth-service.
On the Overview page, confirm the Status shows Running.
Click the Application Url link on the Overview page.

You should see the nginx welcome page ("Welcome to nginx!") in your browser. This confirms the app is receiving traffic.

Go back to the Container Apps list and repeat for api-service.
Confirm Status is Running and click the Application Url.

You should see the Apache welcome page ("It works!") for the api-service since it uses the httpd:alpine image.

Go back to the Container Apps list and repeat for frontend-service.
Confirm Status is Running and click the Application Url.

You should see the nginx welcome page again for the frontend-service.

Confirm the log stream is active

Click on any of the three container apps (for example, auth-service).
In the left menu under Monitoring, click Log stream.
Select Console at the top.

You should see log messages appearing. If you visited the Application Url in your browser earlier, you will see access log entries from that traffic.

Not seeing any logs in the log stream?

The nginx and httpd images only generate logs when they receive HTTP requests. Open the Application Url in your browser and refresh a few times. Then check the log stream again.

It can take 5-10 minutes for logs to appear in law-monitoring-lab after the first request. The live log stream should show results faster.

Help me troubleshoot my container app logs

Your microservices environment is live. Three container apps are running and sending their logs to the same Log Analytics Workspace as your VM. Next up, you will deploy a self-hosted Grafana instance to visualize all of this data on real-time dashboards.

Deploy the Grafana VM and Install Grafana

Your monitoring data is flowing into law-monitoring-lab from your VM and all three container apps. Now you need a way to actually see that data.

In this step, you will create a dedicated VM that runs Grafana, an open-source dashboard tool. You will also set up Apache2 as a reverse proxy so you can access your dashboards on port 80 from any web browser.

In this step, get ready to:

Create a new VM for hosting Grafana.
Open port 80 so you can access dashboards from your browser.
SSH into the VM and install Grafana with an Apache2 reverse proxy.

Create the Grafana VM

Grafana needs its own VM because it requires more resources than a lightweight monitoring agent. You will give it 2 vCPUs and 4 GB of RAM so it can handle multiple dashboards at once.

In the Azure Portal search bar, type Virtual machines and click on it.
Click + Create, then select Azure virtual machine.

On the Basics tab, fill in the following:

Set Resource group to rg-monitoring-lab.
Set Virtual machine name to vm-grafana.
Set Region to the same region you have been using (East US).
Set Image to Ubuntu Server 22.04 LTS - x64 Gen2.
Click See all sizes, search for B2s, and select Standard_B2s (2 vCPUs, 4 GB RAM).
Set Authentication type to SSH public key.
Set Username to azureuser.
Set SSH public key source to Generate new key pair.

Now switch to the Networking tab:

Set Virtual network to vnet-monitoring.
Set Subnet to snet-grafana (10.0.2.0/24).
Leave Public IP as the default (new).
Under Select inbound ports, check SSH (22).

Click Review + create.
Click Create.
A popup appears asking to download the private key. Click Download private key and create resource.

Save the key file securely

The .pem file that downloads is the only way to connect to this VM via SSH. Save it somewhere you can find it later (such as your Downloads folder). You will need it in a few minutes.

Why a bigger VM this time?

Your first VM (vm-workload-01) used a Standard_B1s (1 vCPU, 1 GB RAM) because it only runs a monitoring agent. Grafana needs to process queries and render dashboards, so it requires the Standard_B2s (2 vCPUs, 4 GB RAM) to handle that workload.

Open port 80 for web access

By default, Azure only opens the ports you selected during VM creation (SSH on port 22). Grafana listens internally on port 3000, but you will access it through Apache2 on port 80. You need to open port 80 in the VM's network security rules.

After the VM finishes creating, go to Virtual machines and click vm-grafana.
In the left menu under Settings, click Networking, then Network settings.
Click Create port rule, then Inbound port rule.

Fill in the inbound rule form:

Set Destination port ranges to 80.
Set Protocol to TCP.
Set Action to Allow.
Set Priority to 300.
Set Name to Allow-HTTP-80.
Click Add.

Do NOT open port 3000

Grafana listens on port 3000 internally, but you do not want to expose it directly to the internet. Apache2 acts as a reverse proxy, forwarding traffic from port 80 to Grafana on port 3000. This keeps your setup cleaner and more secure.

SSH into the Grafana VM and install everything

Now you will connect to the Grafana VM using SSH and install three things: Grafana itself, a configuration change to enable Managed Identity, and Apache2 as a reverse proxy.

Before you connect, note the public IP address of vm-grafana. Find it on the VM's Overview page in the Azure Portal.
Open your terminal. On macOS, press Cmd+Space and type Terminal. On Windows, press the Windows key and type Terminal.
Set the correct permissions on your key file by running this command:

chmod 400 [[GRAFANA_KEY_PATH="path to your downloaded .pem file, e.g. ~/Downloads/vm-grafana_key.pem"]]

Connect to the VM via SSH by running this command:

ssh -i [[GRAFANA_KEY_PATH="path to your downloaded .pem file, e.g. ~/Downloads/vm-grafana_key.pem"]] azureuser@[[GRAFANA_VM_IP="your vm-grafana public IP address"]]

What does this command do?

The ssh command opens a secure connection to your remote VM. The -i flag points to the private key file that proves your identity. Once connected, your terminal is running commands directly on the Azure VM.

You should see a welcome message and a prompt like azureuser@vm-grafana:~$. You are now inside the VM.

Connection refused or permission denied?

If you see "Permission denied (publickey)", double check that you ran the chmod 400 command on the correct file. If you see "Connection timed out", verify the IP address matches the one shown on the VM Overview page in the Azure Portal.

help me troubleshoot my SSH connection to the Grafana VM

Now install Grafana OSS from the official APT repository by running these commands:

sudo apt-get update
sudo apt-get install -y apt-transport-https software-properties-common wget
sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install -y grafana

What do these commands do?

The first group installs prerequisite tools for adding a new package source. The wget line downloads Grafana's signing key so your system trusts their packages. The echo line adds Grafana's repository. Finally, apt-get install -y grafana installs Grafana itself.

Next, you need to tell Grafana to use Managed Identity for authentication. This allows Grafana to access Azure Monitor data without you managing passwords or secrets.

Open the Grafana configuration file by running:

sudo nano /etc/grafana/grafana.ini

Press Ctrl+W to search. Type [azure] and press Enter.
Below the [azure] line, add this:

[azure]
managed_identity_enabled = true

Press Ctrl+X to exit.
Press Y to confirm saving.
Press Enter to accept the filename.

What is Managed Identity?

Managed Identity is Azure's way of letting a resource (like your VM) prove who it is to other Azure services without storing passwords. You will assign permissions to this identity in a later step so Grafana can query your monitoring data.

Start and enable the Grafana service by running these commands:

sudo systemctl daemon-reload
sudo systemctl start grafana-server
sudo systemctl enable grafana-server
sudo systemctl status grafana-server

You should see active (running) displayed in green. This confirms Grafana is running.

Grafana not showing active (running)?

If the status shows "failed" or "inactive", run sudo journalctl -u grafana-server --no-pager | tail -20 to see the error logs. The most common issue is a typo in the grafana.ini file. Re-open it with sudo nano /etc/grafana/grafana.ini and verify the [azure] section looks correct.

help me debug why Grafana won't start on my Azure VM

Now install Apache2 and enable the modules needed for reverse proxying. Run these commands:

sudo apt-get install -y apache2
sudo a2enmod proxy proxy_http proxy_wstunnel rewrite

Why Apache2 as a reverse proxy?

A reverse proxy sits between the internet and Grafana. Visitors connect to Apache2 on port 80, and Apache2 forwards those requests to Grafana on port 3000 internally. This means you never expose port 3000 to the internet directly.

Create the Apache2 configuration file for Grafana by running:

sudo nano /etc/apache2/sites-available/grafana.conf

Paste the following content into the file:

<VirtualHost *:80>
    ProxyPreserveHost On
    ProxyPass / http://localhost:3000/
    ProxyPassReverse / http://localhost:3000/

    RewriteEngine On
    RewriteCond %{HTTP:Upgrade} =websocket [NC]
    RewriteRule /(.*) ws://localhost:3000/$1 [P,L]
</VirtualHost>

What does this configuration do?

ProxyPass and ProxyPassReverse forward all HTTP traffic from port 80 to Grafana on port 3000. The RewriteRule handles WebSocket connections, which Grafana uses for live dashboard updates.

Press Ctrl+X, then Y, then Enter to save and exit.

Enable the new site, disable the default site, and restart Apache2 by running these commands:

sudo a2dissite 000-default.conf
sudo a2ensite grafana.conf
sudo systemctl restart apache2

What do these commands do?

a2dissite disables the default Apache2 welcome page. a2ensite enables your new Grafana reverse proxy config. The restart applies both changes.

Time to test it. Open your web browser and go to:

http://[[GRAFANA_VM_IP="your vm-grafana public IP address"]]

You should see the Grafana login page.

Enter admin as the username.
Enter admin as the password.
Grafana prompts you to set a new password. Choose a password you will remember.

Can't see the Grafana login page?

Make sure you are using http:// (not https://) in the URL. Verify the IP address matches the one shown on the vm-grafana Overview page.

If the page still does not load, confirm the port 80 rule (Allow-HTTP-80) exists in the VM's Network settings. Also check that Apache2 is running by going back to your SSH session and running sudo systemctl status apache2.

help me troubleshoot why I cannot access Grafana in my browser

Your Grafana instance is up and running. Next up, you will give this VM the permissions it needs to read your monitoring data using Managed Identity and RBAC role assignments.

Configure Managed Identity and Permissions

Grafana is installed and running, but it cannot see any of your monitoring data yet. It needs permission to read metrics and logs from your Azure resources.

In this step, you will give the Grafana VM a Managed Identity and assign it two roles that allow it to read your monitoring data. Managed Identity means no passwords or secrets to manage. Azure handles the authentication automatically.

In this step, get ready to:

Enable System-assigned Managed Identity on the Grafana VM.
Assign the Monitoring Reader role to the VM.
Assign the Log Analytics Reader role to the VM.

Turn on Managed Identity

A System-assigned Managed Identity gives a VM its own identity in Azure Active Directory. Other Azure services can recognize it and grant it access without needing API keys or passwords.

In the Azure Portal search bar, type Virtual machines and click on it.
Click vm-grafana to open it.
In the left menu under Security, click Identity.
Under the System assigned tab, flip the Status toggle to On.
Click Save.
Click Yes on the confirmation dialog.

Wait a few seconds for it to save. You should see an Object (principal) ID appear on the page. This confirms the identity was created successfully.

What is Managed Identity?

Managed Identity is Azure's way of letting resources authenticate to other Azure services without storing credentials. Instead of creating a username/password or API key, Azure automatically manages the identity behind the scenes.

In the previous step, you set managed_identity_enabled = true in Grafana's config. Now you are creating the actual identity that Grafana will use. The two pieces work together.

Don't see the Object (principal) ID?

Refresh the page and ensure you are on the System assigned tab (not the User assigned tab). The toggle should show On and a long alphanumeric ID should be displayed below it.

Still having trouble? Why isn't my Managed Identity showing up on vm-grafana?

Assign the Monitoring Reader role

The identity exists now, but it has zero permissions. You need to give it two RBAC roles so it can read metrics and logs from your resource group.

In the search bar, type Resource groups and click on it.
Click rg-monitoring-lab.
In the left menu, click Access control (IAM).
Click + Add, then select Add role assignment.
In the Role tab, search for Monitoring Reader.
Click on Monitoring Reader in the results, then click Next.
In the Members tab, set Assign access to to Managed identity.
Click + Select members.
In the Managed identity dropdown, select Virtual machine.
Click on vm-grafana in the list.
Click Select.
Click Review + assign.
Click Review + assign again to confirm.

You should see a notification that the role assignment was added successfully.

What does Monitoring Reader do?

The Monitoring Reader role grants read access to Azure Monitor metrics data. This is what allows Grafana to pull platform metrics like NAT Gateway bytes, VM CPU percentage, and other resource-level metrics.

Assign the Log Analytics Reader role

The Monitoring Reader role covers metrics, but your Log Analytics workspace stores logs that require a separate permission. You need a second role assignment.

Stay on the same page (Access control (IAM) for rg-monitoring-lab).
Click + Add, then select Add role assignment.
In the Role tab, search for Log Analytics Reader.
Click on Log Analytics Reader in the results, then click Next.
In the Members tab, set Assign access to to Managed identity.
Click + Select members.
In the Managed identity dropdown, select Virtual machine.
Click on vm-grafana in the list.
Click Select.
Click Review + assign.
Click Review + assign again to confirm.

You should see a second notification confirming the role assignment was added successfully.

Why two separate roles?

Azure follows the principle of least privilege. Monitoring Reader covers platform metrics (like NAT Gateway bytes and VM CPU). Log Analytics Reader covers log queries (like the KQL queries you will write for dashboards).

Grafana needs both because your dashboards will pull from both sources.

Wait 5-10 minutes before the next step

Role assignments can take up to 10 minutes to propagate through Azure. If you try to connect Grafana to Azure Monitor immediately, you may get a permissions error even though you did everything correctly.

Take a break and come back in 5-10 minutes before proceeding to the next step.

Not seeing vm-grafana in the member list?

Make sure you selected Virtual machine in the Managed identity dropdown (not App Registration or another type). If vm-grafana still does not appear, go back and confirm the Managed Identity toggle shows On with an Object ID visible.

Need more help? Why can't I find vm-grafana when assigning a role in Azure IAM?

Your Grafana VM now has the identity and permissions it needs to read all your monitoring data. Next up, you will connect Grafana to Azure Monitor and confirm it can query real data from your workspace.

Connect Grafana to Azure Monitor

In the previous step, you gave your Grafana VM permission to read monitoring data using Managed Identity and RBAC roles.

Now you need to tell Grafana where to find that data. This step links Grafana to your Azure Monitor data so your dashboards can display real-time metrics and logs from the Log Analytics Workspace.

In this step, get ready to:

Add the Azure Monitor data source in Grafana using Managed Identity authentication.
Verify the connection by running a KQL query against your Log Analytics Workspace.

Add the Azure Monitor data source

The Azure Monitor data source plugin is built into Grafana. No additional plugin installation is required.

Open Grafana in your browser at http:// followed by your Grafana VM's public IP address.
Sign in with your admin credentials.
In the left sidebar, click the gear icon (or Connections).
Click Data sources.
Click Add data source.
Search for Azure Monitor and click on it.
Under Authentication, select Managed Identity from the dropdown.
Scroll down and click Save & test.

You should see a green banner that says "Successfully queried the Azure Monitor service".

Why Managed Identity?

Managed Identity lets your Grafana VM authenticate to Azure without storing any passwords or client secrets. Azure handles the credentials behind the scenes.

This works because you enabled the system-assigned identity on vm-grafana in the previous step and assigned it the Monitoring Reader and Log Analytics Reader roles.

Seeing an error instead of the green banner?

Role assignments can take up to 10 minutes to propagate in Azure. Wait 5-10 minutes and click Save & test again.
Confirm that managed_identity_enabled = true is set in /etc/grafana/grafana.ini under the [azure] section, and that you restarted the Grafana service afterwards.
Verify the system-assigned Managed Identity is enabled on vm-grafana by checking the Identity page under Security in the VM settings.

Help me troubleshoot why Grafana can't connect to Azure Monitor with Managed Identity.

Verify you can query data

A successful connection means Grafana can reach Azure Monitor. Now confirm you can actually pull data from your Log Analytics Workspace using a KQL query.

In the left sidebar, click the compass icon (Explore).
At the top, select Azure Monitor as the data source.
Under Service, select Logs.
Under Resource, select your workspace law-monitoring-lab.
In the query box, type:

Heartbeat | take 10

Click Run query.

You should see heartbeat records from your VM listed in the results panel.

What is this query doing?

The Heartbeat table contains periodic "I'm alive" signals sent by the Azure Monitor Agent on your VM. The | take 10 portion limits the results to 10 rows so the query runs quickly.

Seeing results here confirms the full pipeline is working: your VM sends data to the Log Analytics Workspace, and Grafana can read it.

Query returns empty results?

The Azure Monitor Agent may need more time to send data. Wait 10 minutes and run the query again.
Go back to the Azure Portal and check that vm-workload-01 is still running. A stopped VM does not send heartbeats.
Confirm the Data Collection Rule dcr-vm-performance still lists vm-workload-01 as an associated resource under Monitor > Data Collection Rules.

Why is my Grafana KQL Heartbeat query returning empty results from my Log Analytics Workspace?

Your monitoring pipeline is fully connected. Grafana can now pull any metric or log from your Log Analytics Workspace. Next up, you will build your first set of dashboards to visualize VM performance data in real time.

Build the VM Metric Dashboards

In the previous step, you connected Grafana to Azure Monitor and confirmed it can query your Log Analytics Workspace. Your dashboard tool now has a live connection to all the telemetry flowing in from your VM.

Raw data sitting in a database is not useful unless you can see patterns at a glance. In this step, you will build four dedicated dashboards that visualize your VM's CPU, memory, disk, and swap usage as time-series graphs.

Each dashboard uses KQL (Kusto Query Language) to pull exactly the metrics you need from the Perf table in Log Analytics.

In this step, get ready to:

Create a CPU Usage dashboard showing processor utilization over time.
Create a Memory Usage dashboard tracking RAM consumption.
Create Disk Space and Swap dashboards showing storage and swap pressure.

Create the CPU dashboard

This first dashboard walks you through the full process of creating a Grafana panel with a KQL query. You will repeat a similar process for each dashboard after this one.

In the Grafana left sidebar, click Dashboards.
Click New, then select New Dashboard.
Click Add visualization.
In the panel editor, set the data source to Azure Monitor.
At the top of the query section, set Service to Logs.
Under Resource, select law-monitoring-lab.

In the KQL query box, paste the following query:

Perf
| where ObjectName == "Processor" and CounterName == "% Processor Time" and InstanceName == "_Total"
| where Computer == "vm-workload-01"
| summarize AvgCPU = avg(CounterValue) by bin(TimeGenerated, 1m)
| order by TimeGenerated asc

What does this query do?

This KQL query reads from the Perf table, which is where Azure Monitor Agent stores performance counter data.

The first where line filters for CPU-specific counters. _Total means the combined average across all CPU cores.
The second where line limits results to your specific VM.
The summarize line calculates the average CPU value in 1-minute time buckets, creating the data points for the graph.

Click Run query to test it.
On the right side panel, set the Title to CPU Usage %.
Confirm the Visualization dropdown is set to Time series (this is the default).
Click Apply in the top right corner.

Click the floppy disk icon (Save) at the top of the dashboard.
Enter CPU Dashboard as the dashboard name.
Click Save.
Click the gear icon on the dashboard, set Auto-refresh to 30s, and save again.

You should see a time-series line graph showing CPU utilization percentage over the selected time range. If you see data points, your first dashboard is working.

Not seeing any data in the chart?

Performance counter data can take 5-10 minutes to appear in Log Analytics after the Azure Monitor Agent starts collecting. Expand the time range in the top-right corner to "Last 6 hours" or "Last 12 hours" and try again.
Confirm your data source is set to Azure Monitor and the Service is Logs (not Metrics). The Perf table is only accessible through the Logs service.
Verify you selected the correct resource (law-monitoring-lab) in the resource picker.

Help me troubleshoot why my Grafana KQL query returns no data from the Perf table.

Create the Memory dashboard

This dashboard tracks how much RAM your VM is consuming. The process is identical to what you just did for CPU.

Go to Dashboards in the left sidebar.
Click New, then New Dashboard.
Click Add visualization.
Set the data source to Azure Monitor, Service to Logs, and Resource to law-monitoring-lab.

Paste this KQL query:

Perf
| where ObjectName == "Memory" and CounterName == "% Used Memory"
| where Computer == "vm-workload-01"
| summarize AvgMemory = avg(CounterValue) by bin(TimeGenerated, 1m)
| order by TimeGenerated asc

What does this query do?

This query filters the Perf table for the Memory object's "% Used Memory" counter. It calculates the average memory utilization in 1-minute buckets, giving you a smooth line showing RAM pressure on your VM over time.

Click Run query to verify data appears.
Set the panel Title to Memory Usage %.
Confirm the Visualization is Time series.
Click Apply.
Save the dashboard as Memory Dashboard.
Set auto-refresh to 30s via the dashboard gear icon and save again.

You should see a time-series graph showing memory utilization as a percentage. On a B1s VM with 1 GB RAM, you will typically see usage between 30-60% even at idle.

Create the Disk Space and Swap dashboards

These two dashboards monitor storage pressure. The Disk Space dashboard tracks how full your root partition is. The Swap dashboard tracks virtual memory usage when RAM runs low.

Create a new dashboard (Dashboards > New > New Dashboard > Add visualization).
Set the data source to Azure Monitor, Service to Logs, and Resource to law-monitoring-lab.

Paste this KQL query for disk usage:

Perf
| where ObjectName == "Logical Disk" and CounterName == "% Used Space" and InstanceName == "/"
| where Computer == "vm-workload-01"
| summarize AvgDisk = avg(CounterValue) by bin(TimeGenerated, 5m)
| order by TimeGenerated asc

What does this query do?

This query filters for the Logical Disk object's "% Used Space" counter on the root partition (/). It uses 5-minute time bins because disk usage changes slowly compared to CPU or memory.

Click Run query to verify data appears.
Set the panel Title to Disk Usage % (/).
Confirm the Visualization is Time series.
Click Apply.
Save as Disk Space Dashboard and set auto-refresh to 30s.

Now create the final VM metric dashboard for swap usage.

Create another new dashboard (Dashboards > New > New Dashboard > Add visualization).
Set the data source to Azure Monitor, Service to Logs, and Resource to law-monitoring-lab.

Paste this KQL query for swap usage:

Perf
| where ObjectName == "Memory" and CounterName == "% Used Swap Space"
| where Computer == "vm-workload-01"
| summarize AvgSwap = avg(CounterValue) by bin(TimeGenerated, 5m)
| order by TimeGenerated asc

What does this query do?

Swap space is virtual memory stored on disk. When a VM runs out of RAM, it moves data to swap. High swap usage is a signal that your VM needs more memory. This query tracks that usage over 5-minute intervals.

Click Run query.
Set the panel Title to Swap Usage %.
Confirm the Visualization is Time series.
Click Apply.
Save as Swap Dashboard and set auto-refresh to 30s.

The Swap dashboard may show 0% if your VM has not been under memory pressure. That is expected for now.

Click Dashboards in the left sidebar to return to the dashboard list.
Verify that all four dashboards appear: CPU Dashboard, Memory Dashboard, Disk Space Dashboard, and Swap Dashboard.

Dashboard shows "No data" for Swap?

This is normal if your VM has not been under memory pressure yet. Swap usage only increases when RAM is full. You will generate memory load in the testing step later, which will produce swap data.

If ALL four dashboards show no data, the issue is likely that the Azure Monitor Agent has not been collecting long enough. Wait 10-15 minutes and refresh each dashboard.

Help me figure out why my Grafana dashboards are not showing any VM metric data.

You now have four operational dashboards tracking every key health signal from your VM. Next up, you will build dashboards that show real-time logs from your three container apps.

Build the Container Log and Syslog Dashboards

Your four VM metric dashboards give you visibility into hardware-level performance. Now you need visibility into what your applications are doing. Container logs capture every HTTP request, error message, and system event your three microservices produce.

In this step, you will build four dashboards that query the ContainerAppConsoleLogs_CL and Syslog tables in your Log Analytics Workspace. These dashboards show real-time log streams, HTTP errors, log volume trends, and system-level messages from your VM.

In this step, get ready to:

Create a Container Console Logs Dashboard showing live stdout/stderr output from all three apps.
Create an HTTP Error Logs Dashboard filtering for 4xx and 5xx status codes.
Create a Container Log Volume Dashboard and a VM Syslogs Dashboard.

Create the Container Console Logs Dashboard

This dashboard displays a live table of all stdout and stderr output from your three container apps. Every HTTP request they handle produces a log line here.

In Grafana, click Dashboards in the left sidebar.
Click New, then New Dashboard.
Click Add visualization.
Select Azure Monitor as the data source.
At the top of the query section, set Service to Logs.
Set Resource to law-monitoring-lab.
In the KQL query box, paste this query:

ContainerAppConsoleLogs_CL
| project TimeGenerated, AppName=ContainerAppName_s, Revision=RevisionName_s, Message=Log_s
| order by TimeGenerated desc
| take 100

What does this query do?

ContainerAppConsoleLogs_CL is the table where Azure stores all stdout/stderr output from your container apps. The project operator selects only the columns you care about and renames them for readability.

The query grabs the 100 most recent log entries, ordered newest first. This gives you a live feed of what your apps are outputting.

Click Run query to test.
On the right side panel, change Visualization to Table.
Set Title to Live Container Console Logs.
Click Apply in the top right.
Click the floppy disk icon (Save) at the top of the dashboard.
Enter Container Console Logs Dashboard as the dashboard name.
Click Save.
Click the gear icon on the dashboard, set Auto-refresh to 5s, and save again.

You should see a table of log entries from your container apps. If the table is empty, your apps may not have received any HTTP traffic yet. Visit each container app's Application Url in your browser to generate some logs, then wait 5 minutes and refresh.

Seeing no data in the table?

Container apps using nginx:alpine and httpd:alpine only produce logs when they receive HTTP requests. Open each app's URL in your browser to generate traffic.

Log ingestion into Log Analytics can take 5-10 minutes. If you just generated traffic, wait and try again.

help me debug why my ContainerAppConsoleLogs_CL table is empty in Grafana

Create the HTTP Error Logs Dashboard

This dashboard filters container logs to show only HTTP error responses. It helps you spot broken endpoints and failed requests across all three apps.

Go to Dashboards in the left sidebar.
Click New, then New Dashboard, then Add visualization.
Select Azure Monitor as the data source, set Service to Logs, and Resource to law-monitoring-lab.
Paste this KQL query:

ContainerAppConsoleLogs_CL
| where Log_s contains "401" or Log_s contains "403" or Log_s contains "500" or Log_s contains "502" or Log_s contains "503"
| project TimeGenerated, AppName=ContainerAppName_s, Message=Log_s
| order by TimeGenerated desc
| take 50

What does this query do?

The where clause filters logs to only show entries containing common HTTP error codes. 401 and 403 are authentication/authorization failures. 500, 502, and 503 are server-side errors.

In a production environment, this dashboard would immediately surface broken services or failed authentication attempts.

Click Run query to test.
Change Visualization to Table.
Set Title to HTTP Error Logs (4xx/5xx).
Click Apply.
Save the dashboard as HTTP Error Logs Dashboard.
Set Auto-refresh to 5s and save again.

Generate some error logs

This dashboard will be empty until your apps return error codes. Try visiting an invalid path like https://<app-url>/nonexistent on any of your container apps. This generates a 404 in the logs. You will generate more errors in the testing step later.

Create the Container Log Volume and VM Syslogs Dashboards

The log volume dashboard tracks how many logs each app produces over time. A sudden spike means something changed. The VM Syslogs dashboard captures system-level messages like authentication events and daemon warnings.

Start with the Container Log Volume Dashboard:

Go to Dashboards, click New, then New Dashboard, then Add visualization.
Select Azure Monitor as the data source, set Service to Logs, and Resource to law-monitoring-lab.
Paste this KQL query:

ContainerAppConsoleLogs_CL
| summarize LogCount = count() by ContainerAppName_s, bin(TimeGenerated, 5m)
| order by TimeGenerated asc

What does this query do?

The summarize operator counts log entries per app, grouped into 5-minute time buckets. This produces a time series line per container app, letting you compare their activity levels side by side.

Click Run query to test.
Leave Visualization as Time series (the default).
Set Title to Container Log Volume by App.
Click Apply.
Save as Container Log Volume Dashboard.
Set Auto-refresh to 30s and save.

Now create the VM Syslogs Dashboard:

Click New, then New Dashboard, then Add visualization.
Select Azure Monitor as the data source, set Service to Logs, and Resource to law-monitoring-lab.
Paste this KQL query:

Syslog
| where Computer == "vm-workload-01"
| project TimeGenerated, Facility, SeverityLevel, SyslogMessage
| order by TimeGenerated desc
| take 100

What does this query do?

The Syslog table receives system-level messages from your VM via the Data Collection Rule you configured earlier. It captures authentication events, daemon warnings, and kernel messages.

The Facility column tells you which subsystem generated the message (auth, daemon, kernel). SeverityLevel tells you how serious it is.

Click Run query to test.
Change Visualization to Table.
Set Title to Live VM Syslogs.
Click Apply.
Save as VM Syslogs Dashboard.
Set Auto-refresh to 5s and save.

You should see syslog entries with facilities like auth, daemon, and syslog appearing in the table. These messages include SSH session events and system service updates.

Syslog table showing no results?

Verify that your Data Collection Rule (dcr-vm-performance) includes a Linux Syslog data source. Go to Azure Portal, search for Data Collection Rules, open your rule, and check the Data Sources tab.

Syslog data may take 5-10 minutes to first appear after the Azure Monitor Agent starts collecting.

help me troubleshoot missing syslog data in my Grafana dashboard

You now have 8 dashboards. Four show how your infrastructure is performing (CPU, memory, disk, swap). Four show what your applications and operating system are doing (console logs, HTTP errors, log volume, syslogs). Next up, you will build dashboards that specifically track application errors and service health.

Build the Application Error Dashboards

Your previous dashboards show what is happening across your VMs and containers. Now you need dashboards that answer a more specific question: is anything breaking?

Application error dashboards help you spot problems before users notice them. In this step, you will build 4 dashboards that track error trends over time, surface container system failures, show which apps are active, and flag authentication failures on your VM.

In this step, get ready to:

Create an Error Trends dashboard that visualizes error spikes over time.
Create a Container System Errors dashboard that surfaces infrastructure-level failures.
Create Container App Status and VM Auth Failures dashboards to track app health and security events.

Create the Error Trends dashboard

This dashboard aggregates errors from your container apps into 5-minute buckets, broken down by app name. When a problem starts, you will see a spike on the graph for the affected service.

In Grafana, click Dashboards in the left sidebar.
Click New, then New Dashboard.
Click Add visualization.
Select Azure Monitor as the data source.
Set Service to Logs.
Set Resource to law-monitoring-lab.
Paste the following KQL query into the query box:

ContainerAppConsoleLogs_CL
| where Log_s contains "error" or Log_s contains "Error" or Log_s contains "500" or Log_s contains "401"
| summarize ErrorCount = count() by bin(TimeGenerated, 5m), ContainerAppName_s
| order by TimeGenerated asc

What does this query do?

This query searches the ContainerAppConsoleLogs_CL table for any log line containing common error indicators like "error", "500", or "401".

It then counts how many errors occurred in each 5-minute window, grouped by which container app produced them. This lets you see exactly when errors started and which service is responsible.

Click Run query to test it.
On the right panel, confirm the Visualization is set to Time series (this should be the default).
Set the Title to Error Trends Over Time.
Click Apply in the top right.
Click the floppy disk icon (Save) at the top of the dashboard.
Enter Error Trends Dashboard as the dashboard name.
Click Save.
Click the gear icon on the dashboard, set Auto-refresh to 30s, and save again.

You should see a time series graph. It may show data if your container apps have already generated error-like log entries from your earlier testing.

Graph is empty?

If no data appears, your container apps may not have generated any errors yet. You will generate test traffic in a later step. The dashboard will populate once error-like logs exist.

If the query itself fails, confirm that law-monitoring-lab is selected as the Resource and that the Service is set to Logs (not Metrics).

Why isn't my Error Trends query returning results in Grafana?

Create the Container System Errors dashboard

Container console logs capture what your apps print to stdout. System logs capture what the container platform itself reports. This dashboard surfaces infrastructure-level failures like crash loops and scheduling errors.

Go to Dashboards in the left sidebar.
Click New, then New Dashboard.
Click Add visualization and select Azure Monitor as the data source.
Set Service to Logs and Resource to law-monitoring-lab.
Paste the following KQL query:

ContainerAppSystemLogs_CL
| where Log_s contains "error" or Log_s contains "Error" or Log_s contains "ERROR" or Log_s contains "BackOff"
| project TimeGenerated, AppName=ContainerAppName_s, Message=Log_s
| order by TimeGenerated desc
| take 50

What does this query do?

This query searches the ContainerAppSystemLogs_CL table, which contains platform-level messages from Azure Container Apps. It filters for error keywords and "BackOff" (which indicates a container is repeatedly crashing and being restarted).

The project operator selects only the columns you need. The most recent 50 entries are returned.

Click Run query to test.
On the right panel, change Visualization to Table.
Set the Title to Container System Errors.
Click Apply.
Save the dashboard as Container System Errors Dashboard.
Set Auto-refresh to 30s and save again.

You should see a table view. It may be empty if your containers have not experienced any system-level errors yet.

Create the Container App Status and VM Auth Failures dashboards

The final two dashboards in this group each serve a different purpose. The status dashboard gives you a quick "are my apps alive?" check. The auth failures dashboard flags suspicious security events on your VM.

Start with the Container App Status dashboard.

Go to Dashboards, click New, then New Dashboard.
Click Add visualization and select Azure Monitor (Logs, law-monitoring-lab).
Paste the following KQL query:

ContainerAppSystemLogs_CL
| summarize LastLog = max(TimeGenerated) by ContainerAppName_s
| extend Status = iff(LastLog > ago(10m), "Active", "Inactive")
| project ContainerAppName_s, Status, LastLog

What does this query do?

This query checks the most recent system log entry for each container app. If the last log is within 10 minutes, the app is marked "Active". Otherwise it is marked "Inactive".

The iff() function works like an if-else in KQL. The ago(10m) function returns the timestamp from 10 minutes ago.

Click Run query to test.
Change Visualization to Table.
Set the Title to Container App Status.
Click Apply.
Save the dashboard as Container App Status Dashboard.
Set Auto-refresh to 30s and save again.

You should see a table listing all 3 container apps with their current status. All should show "Active" since they are running.

Now create the VM Auth Failures dashboard.

Go to Dashboards, click New, then New Dashboard.
Click Add visualization and select Azure Monitor (Logs, law-monitoring-lab).
Paste the following KQL query:

Syslog
| where Facility == "auth" or Facility == "authpriv"
| where SeverityLevel in ("err", "crit", "alert", "emerg")
| project TimeGenerated, Computer, SyslogMessage
| order by TimeGenerated desc
| take 50

What does this query do?

This query filters the Syslog table for authentication-related facilities (auth and authpriv) and only shows entries with high severity levels (err, crit, alert, emerg).

In a production environment, entries here would indicate failed SSH attempts, invalid credentials, or brute-force attacks against your VM.

Click Run query to test.
Change Visualization to Table.
Set the Title to VM Authentication Failures.
Click Apply.
Save the dashboard as VM Auth Failures Dashboard.
Set Auto-refresh to 30s and save again.

This table may be empty for now. You will generate test auth failure entries in the testing step later.

Not seeing any results in the auth failures table?

An empty table is normal at this point. Your VM has not experienced authentication failures yet. The testing step later in this project will generate sample auth entries using the logger command.

If the query returns an error instead of an empty result, double-check that the Service is set to Logs and the Resource is law-monitoring-lab.

Why is my Syslog query for auth failures not working in Grafana?

You now have 12 dashboards covering VM performance, container logs, and application errors. Next up, you will build the 5 NAT Gateway dashboards that track all outbound network traffic from your infrastructure.

Build the NAT Gateway Dashboards

Your 12 dashboards already cover VM metrics, container logs, and application errors. The last piece of your observability puzzle is the network layer.

Your NAT Gateway tracks every byte and packet leaving your network. In this step, you will build 5 dashboards that visualize data transfer, packet counts, connection states, availability, and dropped packets. These dashboards use Azure Monitor platform metrics instead of KQL log queries, which means the Grafana configuration is slightly different from what you have done so far.

In this step, get ready to:

Create a NAT Data Transfer Dashboard with 3 panels showing bytes in/out, outbound-only traffic, and an hourly total stat.
Create a NAT Packet Count Dashboard showing packets by direction.
Create NAT SNAT Connections, Datapath Availability, and Dropped Packets Dashboards.

Why Metrics instead of Logs?

Your previous dashboards queried the Perf, Syslog, and ContainerAppConsoleLogs_CL tables using KQL. Those are log-based data stored in your Log Analytics Workspace.

NAT Gateway exposes platform metrics directly through Azure Monitor. These are pre-aggregated numbers (bytes, packets, connections) that Azure tracks automatically. In Grafana, you select the Metrics service instead of Logs when building these panels.

Create the NAT Data Transfer Dashboard

This dashboard has 3 panels showing how much data flows through your NAT Gateway. The first panel splits traffic by direction (in vs out). The second isolates outbound traffic. The third shows an hourly total with color-coded thresholds.

In Grafana, click Dashboards in the left sidebar.
Click New, then New Dashboard.
Click Add visualization.

Configure the first panel (Data Transfer In/Out):

Select Azure Monitor as the data source.
For Service, select Metrics (not Logs).
Click Select a resource.
Expand your subscription, then expand rg-monitoring-lab, then expand Microsoft.Network/natGateways.
Select natgw-monitoring.
For Metric namespace, select Microsoft.Network/natGateways.
For Metric, select Bytes.
For Aggregation, select Sum.
Click Add next to Dimension.
Select Direction.
For Values, select both In and Out.
On the right panel, set Title to NAT Gateway Data Transfer (In/Out).
Under Standard options, set Unit to bytes (SI) so values display as KB/MB/GB.
Click Apply in the top right.

Now add the second panel (Outbound Traffic Only):

On the dashboard, click Add, then Visualization.
Select Azure Monitor as the data source.
Set Service to Metrics.
Select resource natgw-monitoring (same as before).
Set Metric to Bytes and Aggregation to Sum.
Add a Dimension filter: set Dimension to Direction, Operator to equals, and Value to Out.
Set the Title to NAT Gateway Outbound Traffic Only.
Click Apply.

Now add the third panel (Stat visualization for hourly total):

Click Add, then Visualization.
Set data source to Azure Monitor, Service to Metrics, and resource to natgw-monitoring.
Set Metric to Bytes and Aggregation to Sum.
On the right panel, change Visualization from Time series to Stat.
Set Title to Data Transferred (Last Hour).
Under Standard options, set Unit to bytes (SI).
Under Thresholds, set Green as the base (default).
Add a Yellow threshold at 26843545600 (25 GB in bytes).
Add a Red threshold at 53687091200 (50 GB in bytes).
Click Apply.

Save and configure auto-refresh:

Click the floppy disk icon (Save) at the top of the dashboard.
Set Dashboard name to NAT Data Transfer Dashboard.
Click Save.
Click the gear icon on the dashboard, set Auto-refresh to 30s, then save again.

What do the thresholds mean?

The Stat panel changes color based on how much data your NAT Gateway has transferred. Green means normal. Yellow (25 GB) means traffic is elevated. Red (50 GB) means something unusual is happening and you should investigate potential data exfiltration or unexpected egress.

Panels showing 'No data'?

NAT Gateway only shows metrics when traffic passes through it. If your vm-workload-01 has not made any outbound internet requests recently, the charts may be empty. This is normal. Data will appear after the testing step later in this project.

Also confirm the resource path is correct. You should see natgw-monitoring under Microsoft.Network/natGateways in the resource picker.

Why isn't my NAT Gateway panel showing any metrics data?

Create the NAT Packet Count Dashboard

This dashboard tracks the number of packets flowing through the NAT Gateway in each direction.

Go to Dashboards in the left sidebar.
Click New, then New Dashboard, then Add visualization.
Select Azure Monitor as the data source.
Set Service to Metrics.
Select resource natgw-monitoring.
Set Metric to Packets.
Set Aggregation to Sum.
Add Dimension: select Direction with Values In and Out.
Set Title to NAT Gateway Packet Count.
Leave Visualization as Time series (default).
Click Apply.
Save the dashboard as NAT Packet Count Dashboard.
Set auto-refresh to 30s and save again.

Create the SNAT Connections, Availability, and Dropped Packets Dashboards

These three dashboards round out your network monitoring. SNAT connections show if your gateway is running out of ports. Datapath Availability shows the gateway's health percentage. Dropped Packets reveals lost traffic.

Create the NAT SNAT Connections Dashboard:

Go to Dashboards, click New, then New Dashboard, then Add visualization.
Select Azure Monitor > Metrics > resource natgw-monitoring.
Set Metric to SNAT Connection Count.
Set Aggregation to Sum.
Add Dimension: select ConnectionState (this splits the chart into Attempted vs Failed).
Set Title to NAT SNAT Connections (Attempted vs Failed).
Click Apply.
Save as NAT SNAT Connections Dashboard with auto-refresh set to 30s.

What is SNAT?

SNAT (Source Network Address Translation) maps your VM's private IP to the NAT Gateway's public IP for outbound connections. Each outbound connection uses a SNAT port. If "Failed" connections appear greater than 0 on this dashboard, it means your gateway is running out of available ports and some connections are being dropped.

Create the NAT Datapath Availability Dashboard:

Create a new dashboard with a new visualization (same steps as before).
Select Azure Monitor > Metrics > resource natgw-monitoring.
Set Metric to Datapath Availability.
Set Aggregation to Average.
On the right panel, change Visualization to Gauge.
Set Title to NAT Gateway Health.
Under Thresholds, configure: Green above 90, Yellow between 80 and 90, Red below 80.
Click Apply.
Save as NAT Datapath Availability Dashboard with auto-refresh set to 30s.

Create the NAT Dropped Packets Dashboard:

Create a new dashboard with a new visualization.
Select Azure Monitor > Metrics > resource natgw-monitoring.
Set Metric to Dropped Packets.
Set Aggregation to Sum.
Leave Visualization as Time series.
Set Title to NAT Gateway Dropped Packets.
Click Apply.
Save as NAT Dropped Packets Dashboard with auto-refresh set to 30s.

You should now see all 5 NAT Gateway dashboards listed in your Grafana Dashboards page.

Why track dropped packets?

A healthy NAT Gateway should show 0 or near 0 dropped packets. If this number climbs, it means the gateway is overwhelmed or misconfigured. Catching this early prevents hard-to-diagnose connectivity issues for all VMs on the subnet.

You now have 17 operational dashboards covering every layer of your infrastructure. Next up, you will configure alerts that email you automatically when something goes wrong.

Set Up Alerts with Email Notifications

Your 17 dashboards are live and showing real-time data from VMs, containers, and the NAT Gateway.

Dashboards only help if someone is actively watching them. Alert rules solve this by automatically notifying you when something breaks. In this step, you will create 5 alert rules that send email notifications when CPU spikes, memory fills up, disk runs low, network egress is unusually high, or containers crash.

In this step, get ready to:

Create an Action Group that sends email notifications.
Create 5 alert rules covering CPU, memory, disk, NAT Gateway, and container errors.

Create an Action Group for email delivery

An Action Group defines WHO gets notified and HOW when an alert fires. You will create one Action Group and reuse it across all 5 alert rules.

In the Azure Portal search bar, type Monitor and click on it.
In the left menu, click Alerts.
Click the Action groups tab at the top.
Click + Create.

On the Basics tab:

Select your subscription.
For Resource group, select rg-monitoring-lab.
For Action group name, type ag-monitoring-email.
For Display name, type MonitorEmail.

Click the Notifications tab:

For Notification type, select Email/SMS message/Push/Voice.
For Name, type email-notify.
In the panel that opens, check Email and type your personal email address.
Click OK.

Click Review + create.
Click Create.

What is an Action Group?

An Action Group is a reusable notification target. Instead of configuring email settings on every single alert rule, you create one Action Group and attach it to as many rules as you want.

Action Groups can also trigger webhooks, Azure Functions, or Logic Apps for automated remediation. For this project, email is all you need.

Create the CPU alert rule

This alert fires when your VM's CPU stays above 70% for 5 minutes. It uses a platform metric directly from the VM resource.

Go to Monitor > Alerts.
Click the Alert rules tab, then click + Create.

Configure the Scope:

Click Select a resource.
Find and select vm-workload-01.
Click Apply.

Configure the Condition:

Click Add condition.
Search for Percentage CPU and click on it.
Set Threshold to Static.
Set Operator to Greater than.
Set Threshold value to 70.
Set Aggregation type to Average.
Set Aggregation granularity to 5 minutes.
Set Frequency of evaluation to Every 5 minutes.

Configure the Actions:

Click Add action groups.
Select ag-monitoring-email.

Configure the Details tab:

For Alert rule name, type alert-cpu-high.
For Severity, select 2 - Warning.
Click Review + create, then click Create.

Why 70% as the CPU threshold?

In production, you want to catch problems before they become outages. A CPU that stays above 70% for 5 minutes is under sustained pressure. Alerting at 70% gives you time to investigate before it hits 100% and causes service degradation.

Create the memory, disk, NAT Gateway, and container alert rules

The remaining 4 alerts use a different approach. Instead of platform metrics from the VM, they use custom log searches against your Log Analytics Workspace (or the NAT Gateway resource for the network alert).

Memory Alert (>80%):

Go to Monitor > Alerts > Alert rules > + Create.
For Scope, select law-monitoring-lab (the workspace, not the VM).
For Condition, select Custom log search.

Paste this KQL query:

Perf
| where ObjectName == "Memory" and CounterName == "% Used Memory"
| where Computer == "vm-workload-01"
| summarize AvgMemory = avg(CounterValue) by bin(TimeGenerated, 5m)
| where AvgMemory > 80

Set Measurement to Table rows.
Set Operator to Greater than and Threshold to 0.
Set Frequency to 5 minutes and Lookback period to 5 minutes.
For Actions, select ag-monitoring-email.
For Alert rule name, type alert-memory-high.
Set Severity to 2 - Warning.
Click Review + create, then Create.

Why use a custom log search instead of a platform metric?

The CPU alert used a built-in platform metric (Percentage CPU) because Azure VMs expose it natively. Memory, disk, and swap metrics come from the Azure Monitor Agent and land in the Perf table inside your Log Analytics Workspace.

To alert on those, you write a KQL query that filters the Perf table. If the query returns any rows, the alert fires.

Disk Alert (>75%):

Create a new alert rule with the same settings as the memory alert.
Scope: law-monitoring-lab.
Condition: Custom log search with this query:

Perf
| where ObjectName == "Logical Disk" and CounterName == "% Used Space" and InstanceName == "/"
| where Computer == "vm-workload-01"
| summarize AvgDisk = avg(CounterValue) by bin(TimeGenerated, 5m)
| where AvgDisk > 75

Measurement: Table rows > Greater than 0.
Frequency: 5 minutes.
Actions: select ag-monitoring-email.
Alert rule name: alert-disk-high.
Severity: 2 - Warning.
Click Review + create, then Create.

NAT Gateway Alert (>50GB in 1 hour):

Go to Monitor > Alerts > Alert rules > + Create.
For Scope, click Select a resource and select natgw-monitoring (the NAT Gateway resource).
For Condition, search for Bytes and click on it.
Set Threshold to Static.
Set Operator to Greater than.
Set Threshold value to 53687091200 (this is 50 GB in bytes).
Set Aggregation type to Total.
Set Aggregation granularity to 1 hour.
Set Frequency of evaluation to Every 15 minutes.
For Actions, select ag-monitoring-email.

On the Details tab:

For Alert rule name, type alert-natgw-high-data-transfer.
Set Severity to 1 - Error.
For Description, type: NAT Gateway processed more than 50GB in the last hour. Investigate potential data exfiltration or unexpected egress.
Click Review + create, then Create.

Why is the NAT Gateway alert severity 1 instead of 2?

50 GB of outbound data in a single hour could indicate data exfiltration, a misconfigured backup job, or a compromised VM flooding the internet. This is a security-relevant event that needs immediate attention, so it gets a higher severity (1 = Error) compared to the resource utilization alerts (2 = Warning).

Container Error Alert:

Go to Monitor > Alerts > Alert rules > + Create.
Scope: select law-monitoring-lab.
Condition: Custom log search with this query:

ContainerAppSystemLogs_CL
| where Log_s contains "error" or Log_s contains "BackOff" or Log_s contains "CrashLoopBackOff"
| where TimeGenerated > ago(5m)
| summarize ErrorCount = count() by ContainerAppName_s
| where ErrorCount > 0

Measurement: Table rows > Greater than 0.
Frequency: 5 minutes.
Actions: select ag-monitoring-email.
Alert rule name: alert-container-errors.
Severity: 2 - Warning.
Click Review + create, then Create.

Verify all 5 alert rules exist by going to Monitor > Alerts > Alert rules. You should see all 5 rules listed with their names and severity levels.

Don't see all 5 rules listed?

Make sure the subscription filter at the top of the Alert rules page includes your subscription. Also confirm the resource group filter is set to rg-monitoring-lab or All.

If a rule failed to create, check that you selected the correct scope for each rule. The CPU alert scopes to the VM, the NAT alert scopes to the NAT Gateway, and the other three scope to the Log Analytics Workspace.

Help me troubleshoot my Azure Monitor alert rules.

Your monitoring platform now watches itself. Next up, you will configure log archival so old data moves to cheaper storage automatically.

Configure Log Archival with Blob Storage

Your monitoring platform is now collecting data from VMs, containers, and your NAT Gateway. All of that telemetry sits in your Log Analytics Workspace in "hot" storage, which is fast to query but expensive to keep long-term.

In production, you rarely need instant access to logs older than 30 days. This step sets up Azure Blob Storage with automatic lifecycle rules that move old data to progressively cheaper storage tiers over time, and eventually delete it after a year.

In this step, get ready to:

Create a Storage Account and Blob container for log archives.
Configure three lifecycle management rules (cool, archive, delete).
Build a cost-effective log retention strategy that runs automatically.

Create a Storage Account

A Storage Account is the top-level container for all Azure storage services. You need one before you can create blob containers inside it.

In the Azure Portal search bar, type Storage accounts and click on it.
Click + Create.
On the Basics tab, configure the following settings:

Set Resource group to rg-monitoring-lab.
Set Storage account name to stmonitoringarchive. If the name is taken, add random numbers to the end (e.g., stmonitoringarchive2847).
Set Region to the same region you have been using (East US).
Set Performance to Standard.
Set Redundancy to Locally-redundant storage (LRS).

Click Review + create.
Click Create.

Why LRS redundancy?

LRS keeps 3 copies of your data within a single data center. It is the cheapest redundancy option.

For archived logs that you could regenerate from the original source if needed, LRS provides enough durability without extra cost.

Wait for the deployment to complete, then click Go to resource.

Create a Blob container

A Blob container is like a folder inside the Storage Account. All your archived log files will live here.

In the left menu of your storage account, click Containers (under Data storage).
Click + Container.
Set Name to log-archives.
Set Anonymous access level to Private (no anonymous access).
Click Create.

You should see log-archives appear in the containers list.

Set up lifecycle management rules

Lifecycle management lets you define rules that automatically move or delete blobs based on their age. You will create three rules that form a cost-saving pipeline: hot to cool to archive to deleted.

What are storage tiers?

Azure Blob Storage has three access tiers. Hot is the default and most expensive. Cool is cheaper but charges more to read data (minimum 30-day retention).

Archive is the cheapest but takes hours to retrieve data (minimum 180-day retention). This tiered approach mirrors how production teams handle log storage.

In the left menu of your storage account, click Lifecycle management (under Data management).
Click Add a rule.

Configure the first rule (move to cool after 30 days):

Set Rule name to move-to-cool.
Set Rule scope to Apply rule to all blobs in your storage account.
Set Blob type to Block blobs.
Set Blob subtype to Base blobs.
Click Next.
On the Base blobs tab, select Last modified.
Set the condition to More than 30 days ago.
Set the action to Move to cool storage.
Click Add.

Now create the second rule (move to archive after 90 days):

Click Add a rule again.
Set Rule name to move-to-archive.
Set the same scope, blob type, and blob subtype as before.
Click Next.
Set the condition to Last modified more than 90 days ago.
Set the action to Move to archive storage.
Click Add.

Finally, create the third rule (delete after 365 days):

Click Add a rule again.
Set Rule name to delete-old-logs.
Set the same scope, blob type, and blob subtype as before.
Click Next.
Set the condition to Last modified more than 365 days ago.
Set the action to Delete the blob.
Click Add.

You should now see all three rules listed on the Lifecycle management page.

How does this save money?

Fresh logs (0-30 days) stay in hot storage for fast querying. Older logs (30-90 days) move to cool storage at a lower cost. Logs older than 90 days move to archive, which is the cheapest tier available.

After a full year, logs are deleted entirely. This tiered approach mirrors what production teams use to balance cost against data retention requirements.

Policies take up to 24 hours

Lifecycle management rules can take up to 24 hours to run for the first time after creation. You will not see blobs moving between tiers immediately.

Don't see Lifecycle management in the menu?

Make sure you are inside the storage account resource, not the resource group. Look for the Data management section in the left menu. If you still cannot find it, try refreshing the page.

Help me find the Lifecycle management setting in my Azure Storage Account.

Your log archival strategy is now in place. Logs will automatically flow from expensive hot storage to cheaper tiers as they age. Next up, you will stress-test your entire monitoring platform and confirm everything works end-to-end.

Test Your Monitoring Platform End-to-End

Your monitoring platform is fully built. Dashboards are connected, alerts are active, and archival policies are in place. But none of it means anything until you prove it works with real data flowing through the system.

In this step, you will generate actual load on your VM, create traffic to your container apps, and then walk through every single dashboard to confirm data appears. You will also trigger an alert and see it arrive in your email inbox.

In this step, get ready to:

Generate CPU, memory, and syslog test data on your monitored VM.
Generate HTTP traffic and error logs on all 3 container apps.
Verify all 17 Grafana dashboards and confirm an alert fired.

Generate load on the monitored VM

Dashboards are only useful if they show real data. Right now your VM is idle, so most metrics will be flat. You need to create artificial load to prove your monitoring pipeline captures spikes and events correctly.

Open your terminal.
Connect to your workload VM via SSH by running this command:

ssh -i [[WORKLOAD_KEY_PATH="path to your vm-workload-01 key file"]] azureuser@[[WORKLOAD_VM_IP="vm-workload-01 public IP address"]]

Where do I find the public IP?

Go to Virtual machines in the Azure Portal, click vm-workload-01, and look for Public IP address on the Overview page. Your key file is the .pem file you downloaded when you created this VM in Step 2.

You should see the terminal prompt change to azureuser@vm-workload-01, confirming you are connected.

Connection refused or permission denied?

If you see a permissions error, make sure your key file permissions are restricted. Run chmod 400 followed by your key file path before trying SSH again.

If the connection times out, confirm the VM is running in the Azure Portal and that SSH (port 22) is allowed in its network security group.

Why can't I SSH into my Azure VM?

Install the stress tool and generate a CPU spike by running these commands:

# Update package list and install stress testing tool
sudo apt-get update
sudo apt-get install -y stress

# Max out 1 CPU core for 2 minutes
stress --cpu 1 --timeout 120

What does this do?

The stress tool generates artificial load on your VM. The --cpu 1 flag consumes one full CPU core, and --timeout 120 stops it after 120 seconds.

Your CPU alert threshold is set to 70%. Since the B1s VM only has 1 CPU core, this will push utilization to nearly 100% and trigger the alert.

The command will hold your terminal for 2 minutes while it runs. Wait for it to finish before continuing.

Once the CPU stress finishes, generate a memory spike by running:

# Allocate 256MB of memory for 2 minutes
stress --vm 1 --vm-bytes 256M --timeout 120

Wait another 2 minutes for this to complete. On the 1 GB RAM B1s VM, 256 MB is a significant portion of available memory.

Generate test syslog entries that will appear in your VM Syslogs and Auth Failures dashboards by running:

# Generate test log entries at different facilities and severity levels
logger -p auth.info "Test login event from monitoring lab"
logger -p auth.err "Test FAILED login attempt"
logger -p daemon.err "Test error: service crash simulation"

What do these logger commands do?

The logger command writes entries directly to the system log (syslog). The -p flag sets the facility and severity level.

The auth.err entry simulates a failed login, which your Data Collection Rule is configured to capture. This entry will appear in both the VM Syslogs dashboard and the VM Auth Failures dashboard.

Disconnect from the VM by typing:

exit

Your terminal returns to your local machine. The stress tests are already done, and the syslog entries are written.

Generate container app traffic

Your container apps (nginx and httpd) only produce log entries when they receive HTTP requests. Without traffic, the container log dashboards will be empty. You need to visit each app in your browser to generate access logs and error logs.

In the Azure Portal, go to Container Apps.
Click on auth-service.
On the Overview page, click the Application Url link to open the app in a new browser tab.
Refresh the page 5-10 times to generate access log entries.
In the browser address bar, add /nonexistent to the end of the URL and press Enter. This generates a 404 error log.
Repeat the same process for api-service and frontend-service. Visit each app's URL, refresh several times, and also visit an invalid path.

Why visit invalid paths?

Your HTTP Error Logs Dashboard filters for status codes like 404. By requesting a page that does not exist, you force the web server to return a 404 response. This creates entries that appear in your error-specific dashboards.

You should see either the nginx welcome page or the Apache "It works!" page when visiting valid URLs. Invalid paths show a "Not Found" error page.

Wait for log ingestion

Container app logs can take 5-10 minutes to appear in Log Analytics after being generated. If dashboards look empty in the next substep, wait a few minutes and refresh.

Verify all 17 dashboards

This is the payoff. Every dashboard you built across the last 4 steps should now have real data flowing through it. Open Grafana in your browser and work through each category below.

Open Grafana at http://your Grafana VM public IP in your browser.
In the left sidebar, click Dashboards.

Check each dashboard one by one. Here is what you should see in each category:

VM metric dashboards

Open the CPU Dashboard. You should see a clear spike to nearly 100% from your stress test.
Open the Memory Dashboard. You should see a memory increase during the 256 MB stress window.
Open the Disk Space Dashboard. You should see current disk usage as a percentage.
Open the Swap Dashboard. You may see swap activity from the memory stress. If swap shows 0, that is normal on a fresh VM with low memory pressure.

Container log dashboards

Open the Container Console Logs Dashboard. You should see access log entries from your browser visits.
Open the HTTP Error Logs Dashboard. You should see the 404 errors you generated by visiting invalid paths.
Open the Container Log Volume Dashboard. You should see log counts per app as a time series graph.
Open the VM Syslogs Dashboard. You should see the three logger entries you created (auth.info, auth.err, daemon.err).

Application error dashboards

Open the Error Trends Dashboard. You should see error spikes corresponding to your 404 requests.
Open the Container System Errors Dashboard. This may be empty or show minor system messages. That is expected since your containers are running normally.
Open the Container App Status Dashboard. All 3 apps should show "Active" status.
Open the VM Auth Failures Dashboard. You should see the "Test FAILED login attempt" entry from your logger command.

NAT Gateway dashboards

Open the NAT Data Transfer Dashboard. You should see bytes flowing out from the VM's outbound traffic (apt-get updates, stress tool download).
Open the NAT Packet Count Dashboard. You should see packet activity in both directions.
Open the NAT SNAT Connections Dashboard. You should see attempted connections.
Open the NAT Datapath Availability Dashboard. The gauge should show near 100%.
Open the NAT Dropped Packets Dashboard. This should show 0 or near 0. Anything higher indicates a problem with your NAT Gateway.

Verify your alert fired

The CPU stress test should have triggered your alert-cpu-high alert rule. It may take 5-10 minutes after the stress test for Azure Monitor to evaluate the rule and fire the alert.

In the Azure Portal, search for Monitor and click on it.
In the left menu, click Alerts.
Look for a fired alert with the name alert-cpu-high. It should show severity "Warning" and a fired timestamp during your stress test window.
Check your email inbox for a notification from Azure Monitor. The subject line references the alert rule name.

Alert has not fired yet?

Alert rules evaluate every 5 minutes. If you just finished the stress test, wait at least 10 minutes before checking. The metric data needs time to flow from the VM through Azure Monitor Agent into the workspace, and then the alert rule needs to evaluate.

If the alert still has not fired after 15 minutes, go to Monitor > Alerts > Alert rules, click alert-cpu-high, and verify the scope is set to vm-workload-01 and the threshold is 70.

Why hasn't my Azure Monitor CPU alert fired after a stress test?

Dashboards showing no data?

If any dashboard appears empty, check the time range in the top-right corner of Grafana. Set it to "Last 1 hour" or "Last 30 minutes" to match when you ran your tests. Performance counter data and container logs can take 5-10 minutes to appear after being generated.

Secret mission

Network Traffic Dashboard and Container Restart Alert

Your 17 dashboards and 5 alerts cover CPU, memory, disk, containers, and NAT Gateway. But what about network bandwidth on the VM itself? And what happens when a container enters a restart loop? This mission adds an 18th dashboard and a 6th alert rule to close those gaps.

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project uses paid Azure resources that cost approximately $7-18/day while running.

Cost warning

The NAT Gateway charges hourly even when your VMs are stopped. If you want to minimize costs without deleting everything, stop the VMs and also delete the NAT Gateway separately.

If you are using an Azure Free Trial, your $200 credit will be consumed within 11-28 days if you leave everything running. Delete the resource group when you are done to avoid unexpected charges.

Resources you used:

2 Virtual Machines (vm-workload-01 and vm-grafana).
1 NAT Gateway (natgw-monitoring) with a public IP address.
1 Container Apps Environment (cae-monitoring-lab) with 3 container apps.
1 Log Analytics Workspace (law-monitoring-lab).
1 Storage Account (stmonitoringarchive).
1 Virtual Network (vnet-monitoring).
5 Alert Rules and 1 Action Group in Azure Monitor.
1 Data Collection Rule (dcr-vm-performance).

Keep everything running

No action needed. Choose this if you are still actively building or experimenting with dashboards and alerts.

Your resources will cost approximately $7-18/day while running.
Monitor your Azure spending under Cost Management + Billing in the Azure Portal.
If you are on the Free Trial, keep an eye on your remaining credit.

Pause - I'll come back to this later

Stop the VMs to reduce compute charges. Disks, the NAT Gateway, and Container Apps will still incur costs.

Go to Virtual machines in the Azure Portal.
Click vm-workload-01, then click Stop at the top.
Go back and click vm-grafana, then click Stop.
Wait for both VMs to show a Status of Stopped (deallocated).
This reduces costs to approximately $3-6/day (NAT Gateway, disks, and Container Apps still running).
To resume later, go back to each VM and click Start. Your Grafana dashboards will be exactly as you left them.

Delete - I don't want to use this again

Remove all project resources in one click by deleting the resource group. This is permanent and cannot be undone.

In the Azure Portal search bar, type Resource groups and click on it.
Click rg-monitoring-lab.
Click Delete resource group at the top.
Type rg-monitoring-lab to confirm deletion.
Click Delete.
Deletion may take 5-10 minutes. Once complete, all VMs, networking, containers, storage, alerts, and monitoring resources are permanently removed.
Your costs drop to $0 immediately after deletion completes.

That's a Wrap!

Nice work. You just built a production-grade centralized monitoring platform in Azure from scratch.

Here's what you accomplished:

Deployed a Log Analytics Workspace collecting telemetry from VMs and containers, connected Grafana via Managed Identity, and built 17 operational dashboards covering CPU, memory, disk, container logs, application errors, and NAT Gateway metrics.
Created 5 Azure Monitor alert rules with email notifications that automatically fire when CPU spikes, memory fills up, disk runs low, NAT Gateway transfers excessive data, or containers crash.
Configured Blob Storage lifecycle management to automatically move old logs from hot to cool to archive storage, reducing long-term costs without manual intervention.
Secret Mission: Added a Network Traffic Dashboard showing VM bandwidth in Mbps and created a Container Restart Loop alert that triggers when any container restarts more than 3 times in 5 minutes.

Ready to quiz yourself?