Observability Stack for Canvas LMS

Monitor a 3-VM Canvas LMS with Prometheus, Grafana, and Alertmanager.

Introduction

30 Second Summary

When thousands of students hit submit on an exam at the same time, you need to know your servers can handle it before they crash. Without visibility into what's happening inside your infrastructure, you're flying blind.

In this project, you will deploy a production-grade observability stack that monitors your Canvas LMS infrastructure in real time. You'll collect metrics from three servers and build dashboards that show you exactly where problems are brewing.

What You'll Build

A Prometheus, Grafana, and Alertmanager stack running in WSL on your Windows PC, scraping live metrics from your 3-VM Canvas LMS architecture over the network.

By the end of this project, you'll have:

  • A running Prometheus instance in WSL scraping system, database, and web server metrics from all three VMs in your Canvas LMS architecture.
  • A Grafana dashboard visualizing RED signals (request rate, error rate, response duration) tailored to an LMS workload at 10K concurrent users.
  • An Alertmanager configuration with proactive alerting rules that fire before users experience degradation.
  • Secret Mission: Build a Canvas-specific application dashboard tracking active database sessions, query latency, and Apache worker saturation during peak load.

Are there any prerequisites?

This project assumes you already have a 3-VM Ubuntu architecture running Canvas LMS (an app VM with Apache, a database VM with PostgreSQL, and a file storage VM). You need SSH access and sudo privileges on each VM, plus WSL with Ubuntu installed on your Windows PC.

Install Prometheus and Alertmanager in WSL

Your Canvas LMS infrastructure is running across 3 VMs, but right now you have no visibility into what's happening under the hood. If CPU spikes on the app VM or the database runs out of connections, you won't know until users start complaining.

In this step, you'll install Prometheus and Alertmanager inside your WSL Ubuntu instance. These two services form the central monitoring hub that will pull metrics from your remote VMs over the network.

In this step, get ready to:
  • Download and install Prometheus 3.12.0 and Alertmanager 0.32.2 binaries.
  • Create configuration files and systemd unit files for both services.
  • Verify both web UIs are accessible from your Windows browser.
Download and install the binaries

Prometheus uses precompiled binaries as the recommended installation method. You'll download the tarball, extract it, create a dedicated system user, and place the binaries in the standard Linux location.

  • Download and install Prometheus by running these commands:
# Download Prometheus 3.12.0
wget https://github.com/prometheus/prometheus/releases/download/v3.12.0/prometheus-3.12.0.linux-amd64.tar.gz

# Extract the archive
tar xvfz prometheus-3.12.0.linux-amd64.tar.gz

# Create a dedicated system user
sudo useradd --no-create-home --shell /bin/false prometheus

# Create configuration and data directories
sudo mkdir -p /etc/prometheus
sudo mkdir -p /var/lib/prometheus

# Move binaries to /usr/local/bin/
sudo cp prometheus-3.12.0.linux-amd64/prometheus /usr/local/bin/
sudo cp prometheus-3.12.0.linux-amd64/promtool /usr/local/bin/

# Copy console templates (used by the web UI)
sudo cp -r prometheus-3.12.0.linux-amd64/consoles /etc/prometheus/
sudo cp -r prometheus-3.12.0.linux-amd64/console_libraries /etc/prometheus/

# Set ownership to the prometheus user
sudo chown -R prometheus:prometheus /etc/prometheus
sudo chown -R prometheus:prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool

# Clean up the downloaded archive
rm -rf prometheus-3.12.0.linux-amd64.tar.gz prometheus-3.12.0.linux-amd64

What do these commands do?

The useradd --no-create-home --shell /bin/false command creates a system user that cannot log in. This follows the principle of least privilege. Prometheus runs as this restricted user instead of root.

The /etc/prometheus directory holds configuration files. The /var/lib/prometheus directory stores the time-series database on disk.

  • Verify the Prometheus binary is installed by running:
prometheus --version

You should see output showing prometheus, version 3.12.0 along with build information.

Command not found?

Make sure you copied the binary to /usr/local/bin/ and that the path is in your $PATH. Run ls /usr/local/bin/prometheus to confirm the file exists.

Help me troubleshoot why prometheus --version is not working in WSL.

  • Download and install Alertmanager by running these commands:
# Download Alertmanager 0.32.2
wget https://github.com/prometheus/alertmanager/releases/download/v0.32.2/alertmanager-0.32.2.linux-amd64.tar.gz

# Extract the archive
tar xvfz alertmanager-0.32.2.linux-amd64.tar.gz

# Create a dedicated system user
sudo useradd --no-create-home --shell /bin/false alertmanager

# Create configuration and data directories
sudo mkdir -p /etc/alertmanager
sudo mkdir -p /var/lib/alertmanager

# Move binaries to /usr/local/bin/
sudo cp alertmanager-0.32.2.linux-amd64/alertmanager /usr/local/bin/
sudo cp alertmanager-0.32.2.linux-amd64/amtool /usr/local/bin/

# Set ownership to the alertmanager user
sudo chown -R alertmanager:alertmanager /etc/alertmanager
sudo chown -R alertmanager:alertmanager /var/lib/alertmanager
sudo chown alertmanager:alertmanager /usr/local/bin/alertmanager
sudo chown alertmanager:alertmanager /usr/local/bin/amtool

# Clean up the downloaded archive
rm -rf alertmanager-0.32.2.linux-amd64.tar.gz alertmanager-0.32.2.linux-amd64

What is amtool?

The amtool binary is Alertmanager's CLI companion. It lets you silence alerts, query active alerts, and test routing configurations from the command line.

  • Verify the Alertmanager binary is installed by running:
alertmanager --version

You should see output showing alertmanager, version 0.32.2 along with build information.

Download failed or timed out?

If wget fails, check your internet connection inside WSL. Run ping github.com to test connectivity. If DNS is the issue, try adding nameserver 8.8.8.8 to /etc/resolv.conf.

Help me fix network issues in WSL so I can download Alertmanager.

Create configuration files and systemd units

Both services need a YAML configuration file to define their behavior. They also need systemd unit files so Linux manages them as background services that start automatically.

  • Create the Prometheus configuration file by running:
sudo nano /etc/prometheus/prometheus.yml
  • Paste the following configuration into the file:
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]
  • Save the file by pressing Ctrl+O, then Enter, then Ctrl+X to exit nano.

What does this configuration do?

The scrape_interval: 15s tells Prometheus to pull metrics from all targets every 15 seconds. The evaluation_interval: 15s controls how often alerting rules are evaluated.

The single scrape job points at Prometheus itself (localhost:9090). This is a validation target to confirm the service is working before you add remote VMs.

  • Create the Alertmanager configuration file by running:
sudo nano /etc/alertmanager/alertmanager.yml
  • Paste the following configuration into the file:
global:
  resolve_timeout: 5m

route:
  group_by: ["alertname", "instance"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: "default-webhook"

receivers:
  - name: "default-webhook"
    webhook_configs:
      - url: "http://localhost:5001/"
        send_resolved: true
  • Save and exit nano (Ctrl+O, Enter, Ctrl+X).

What does the Alertmanager config do?

The route section controls how alerts are grouped and when notifications repeat. group_wait: 30s buffers alerts for 30 seconds before sending, so related alerts arrive together.

The default-webhook receiver sends alerts to a local webhook endpoint. You can later swap this for email or Slack. Using a webhook keeps this step dependency-free.

  • Create the Prometheus systemd service file by running:
sudo nano /etc/systemd/system/prometheus.service
  • Paste the following unit file content:
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus/ \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries
Restart=always

[Install]
WantedBy=multi-user.target
  • Save and exit nano (Ctrl+O, Enter, Ctrl+X).

What does a systemd unit file do?

A systemd unit file tells Linux how to manage a service. The [Service] section defines which user runs the process, what command to execute, and what to do if it crashes (Restart=always means systemd restarts it automatically).

The [Install] section with WantedBy=multi-user.target means the service starts automatically when WSL boots into multi-user mode.

  • Create the Alertmanager systemd service file by running:
sudo nano /etc/systemd/system/alertmanager.service
  • Paste the following unit file content:
[Unit]
Description=Alertmanager
Wants=network-online.target
After=network-online.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/usr/local/bin/alertmanager \
  --config.file=/etc/alertmanager/alertmanager.yml \
  --storage.path=/var/lib/alertmanager/
Restart=always

[Install]
WantedBy=multi-user.target
  • Save and exit nano (Ctrl+O, Enter, Ctrl+X).
  • Confirm all four files exist by running:
ls /etc/prometheus/prometheus.yml /etc/alertmanager/alertmanager.yml /etc/systemd/system/prometheus.service /etc/systemd/system/alertmanager.service

You should see all four file paths listed with no errors.

File not found?

If any file is missing, re-run the sudo nano command for that file path and paste the content again. Make sure you saved with Ctrl+O before exiting.

Help me figure out why my config files are missing after I saved them in nano.

✔️ Awesome, I've got everything!

Great. Make sure all four files are saved before moving on.

ⓧ I'd like to double check the full code

Here are the complete contents of all four files for comparison.

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]
global:
  resolve_timeout: 5m

route:
  group_by: ["alertname", "instance"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: "default-webhook"

receivers:
  - name: "default-webhook"
    webhook_configs:
      - url: "http://localhost:5001/"
        send_resolved: true
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus/ \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries
Restart=always

[Install]
WantedBy=multi-user.target
[Unit]
Description=Alertmanager
Wants=network-online.target
After=network-online.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/usr/local/bin/alertmanager \
  --config.file=/etc/alertmanager/alertmanager.yml \
  --storage.path=/var/lib/alertmanager/
Restart=always

[Install]
WantedBy=multi-user.target
Enable and start both services

With the configuration and unit files in place, you can now tell systemd to start both services and enable them to survive restarts.

  • Reload systemd, enable, and start both services by running these commands:
# Reload systemd to pick up the new unit files
sudo systemctl daemon-reload

# Enable both services to start on boot
sudo systemctl enable prometheus
sudo systemctl enable alertmanager

# Start both services now
sudo systemctl start prometheus
sudo systemctl start alertmanager

What is the difference between enable and start?

The enable command registers a service to start automatically at boot. The start command starts it immediately in the current session. You need both.

  • Check that both services are running by running:
sudo systemctl status prometheus --no-pager
sudo systemctl status alertmanager --no-pager

Both services should show active (running) in green text.

  • Open your Windows browser and navigate to http://localhost:9090.
  • You should see the Prometheus web UI with a query input field and navigation bar.
  • Click Status in the top navigation, then click Targets.
  • You should see the prometheus target listed with state UP.
  • Open a second tab and navigate to http://localhost:9093.
  • You should see the Alertmanager web UI showing an empty alerts page.

Can't reach localhost from your Windows browser?

WSL 2 sometimes doesn't forward ports to the Windows host automatically. Run curl http://localhost:9090 inside WSL to confirm Prometheus is listening. If curl works but the browser doesn't, you may need to use the WSL IP address instead.

Find your WSL IP by running hostname -I inside WSL, then try that IP in your browser.

Help me access my WSL Prometheus from the Windows browser.

Your monitoring hub is now running. Next up, you'll deploy exporters onto each VM to expose the metrics that Prometheus will scrape.

Deploy Exporters Across All VMs

With Prometheus running in WSL, it has nothing useful to scrape yet. It only monitors itself.

In this step, you will install exporters on each VM in your Canvas LMS architecture. Exporters are lightweight agents that translate system, database, and web server internals into the Prometheus-formatted metrics that Prometheus knows how to collect.

In this step, get ready to:
  • Install node_exporter and apache_exporter on the App VM.
  • Install node_exporter and postgres_exporter on the DB VM.
  • Install node_exporter on the File Storage VM and verify all exporters are reachable from WSL.
Install node_exporter and apache_exporter on the App VM

The App VM runs Canvas and Apache. You need two exporters here: node_exporter for system-level metrics (CPU, memory, disk, network) and apache_exporter for web server metrics (request rate, response time, worker status).

  • SSH into your App VM from WSL by running:
ssh [[APP_VM_USER="your-username"]]@[[APP_VM_IP="your app VM IP address"]]
  • Download the node_exporter 1.11.1 binary by running:
# Download node_exporter binary
wget https://github.com/prometheus/node_exporter/releases/download/v1.11.1/node_exporter-1.11.1.linux-amd64.tar.gz

# Extract the tarball
tar xvfz node_exporter-1.11.1.linux-amd64.tar.gz
  • Create a dedicated user and install the binary by running:
# Create a dedicated user for node_exporter
sudo useradd --no-create-home --shell /bin/false node_exporter

# Move the binary to /usr/local/bin
sudo cp node_exporter-1.11.1.linux-amd64/node_exporter /usr/local/bin/
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

# Clean up the download
rm -rf node_exporter-1.11.1.linux-amd64 node_exporter-1.11.1.linux-amd64.tar.gz
  • Create the systemd service file for node_exporter by running:
sudo tee /etc/systemd/system/node_exporter.service > /dev/null << 'EOF'
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
Restart=always

[Install]
WantedBy=multi-user.target
EOF
  • Enable and start node_exporter by running:
sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter
  • Verify node_exporter is serving metrics by running:
curl http://localhost:9100/metrics | head -20

You should see lines of Prometheus-formatted text starting with # HELP and # TYPE followed by metric values.

What is node_exporter?

node_exporter is a Prometheus exporter that exposes hardware and OS metrics from Linux systems. It reads from /proc and /sys to report CPU usage, memory consumption, disk I/O, network traffic, and more.

Each VM in your architecture gets its own node_exporter instance so Prometheus can monitor system health across the entire fleet.

Now install the apache_exporter. This exporter scrapes Apache's built-in mod_status page to extract request counts, response durations, and worker states.

  • First, ensure Apache's mod_status is enabled with extended information by running:
# Enable mod_status if not already enabled
sudo a2enmod status

# Verify ExtendedStatus is on and the endpoint is accessible
curl http://localhost/server-status?auto

Why does apache_exporter need mod_status?

Apache exposes internal performance data through its mod_status module at the /server-status endpoint. The ?auto parameter returns machine-readable output that apache_exporter can parse.

If curl returns a 404 or permission error, you need to configure Apache to allow local access to this endpoint. Canvas deployments typically have this enabled already.

  • Download apache_exporter 1.0.12 by running:
# Download apache_exporter binary
wget https://github.com/Lusitaniae/apache_exporter/releases/download/v1.0.12/apache_exporter-1.0.12.linux-amd64.tar.gz

# Extract the tarball
tar xvfz apache_exporter-1.0.12.linux-amd64.tar.gz
  • Create a dedicated user and install the binary by running:
# Create a dedicated user for apache_exporter
sudo useradd --no-create-home --shell /bin/false apache_exporter

# Move the binary to /usr/local/bin
sudo cp apache_exporter-1.0.12.linux-amd64/apache_exporter /usr/local/bin/
sudo chown apache_exporter:apache_exporter /usr/local/bin/apache_exporter

# Clean up the download
rm -rf apache_exporter-1.0.12.linux-amd64 apache_exporter-1.0.12.linux-amd64.tar.gz
  • Create the systemd service file for apache_exporter by running:
sudo tee /etc/systemd/system/apache_exporter.service > /dev/null << 'EOF'
[Unit]
Description=Apache Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=apache_exporter
Group=apache_exporter
Type=simple
ExecStart=/usr/local/bin/apache_exporter --scrape_uri=http://localhost/server-status?auto
Restart=always

[Install]
WantedBy=multi-user.target
EOF
  • Enable and start apache_exporter by running:
sudo systemctl daemon-reload
sudo systemctl enable apache_exporter
sudo systemctl start apache_exporter
  • Verify apache_exporter is serving metrics on port 9117 by running:
curl http://localhost:9117/metrics | head -20

You should see Prometheus-formatted metrics including lines with apache_ prefixes.

Not seeing apache metrics?

If apache_exporter returns an error about connecting to the scrape URI, confirm that curl http://localhost/server-status?auto returns valid output directly on the App VM. If you get a 403 Forbidden, your Apache config may restrict /server-status to specific IPs. Add Require local to the <Location /server-status> block and restart Apache.

help me troubleshoot why apache_exporter cannot scrape mod_status on my Canvas App VM

  • Exit the SSH session to the App VM by running:
exit
Install node_exporter and postgres_exporter on the DB VM

The DB VM runs PostgreSQL for Canvas. You need node_exporter for system metrics and postgres_exporter to surface database-specific metrics like connection pool usage, query latency, and replication status.

  • SSH into your DB VM from WSL by running:
ssh [[DB_VM_USER="your-username"]]@[[DB_VM_IP="your DB VM IP address"]]
  • Download, install, and start node_exporter (same process as on the App VM) by running:
# Download and extract node_exporter
wget https://github.com/prometheus/node_exporter/releases/download/v1.11.1/node_exporter-1.11.1.linux-amd64.tar.gz
tar xvfz node_exporter-1.11.1.linux-amd64.tar.gz

# Create user and install binary
sudo useradd --no-create-home --shell /bin/false node_exporter
sudo cp node_exporter-1.11.1.linux-amd64/node_exporter /usr/local/bin/
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter
rm -rf node_exporter-1.11.1.linux-amd64 node_exporter-1.11.1.linux-amd64.tar.gz

# Create systemd service
sudo tee /etc/systemd/system/node_exporter.service > /dev/null << 'EOF'
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
Restart=always

[Install]
WantedBy=multi-user.target
EOF

# Enable and start
sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter
  • Verify node_exporter is running on port 9100 by running:
curl http://localhost:9100/metrics | head -5

Now set up postgres_exporter. This requires a dedicated PostgreSQL monitoring user with limited read-only privileges, following the principle of least privilege.

  • Create the PostgreSQL monitoring user by running:
# Connect to PostgreSQL as the postgres superuser
sudo -u postgres psql
  • Inside the psql prompt, create the monitoring user and grant the required permissions by running:
-- Create a dedicated monitoring user
CREATE USER prometheus_monitor WITH PASSWORD 'your-password-here';

-- Grant access to read statistics views
GRANT pg_monitor TO prometheus_monitor;

-- Exit psql
\q

Why a dedicated monitoring user?

Using the application database user for monitoring is risky. If the monitoring connection string leaks, an attacker gets full Canvas database access. The prometheus_monitor user with the pg_monitor role can only read statistics. It cannot modify data or schema.

  • Download and install postgres_exporter 0.19.1 by running:
# Download and extract postgres_exporter
wget https://github.com/prometheus-community/postgres_exporter/releases/download/v0.19.1/postgres_exporter-0.19.1.linux-amd64.tar.gz
tar xvfz postgres_exporter-0.19.1.linux-amd64.tar.gz

# Create user and install binary
sudo useradd --no-create-home --shell /bin/false postgres_exporter
sudo cp postgres_exporter-0.19.1.linux-amd64/postgres_exporter /usr/local/bin/
sudo chown postgres_exporter:postgres_exporter /usr/local/bin/postgres_exporter
rm -rf postgres_exporter-0.19.1.linux-amd64 postgres_exporter-0.19.1.linux-amd64.tar.gz
  • Create the systemd service file for postgres_exporter with the connection string configured via the DATA_SOURCE_NAME environment variable by running:
sudo tee /etc/systemd/system/postgres_exporter.service > /dev/null << 'EOF'
[Unit]
Description=PostgreSQL Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=postgres_exporter
Group=postgres_exporter
Type=simple
Environment="DATA_SOURCE_NAME=postgresql://prometheus_monitor:your-password-here@localhost:5432/canvas?sslmode=disable"
ExecStart=/usr/local/bin/postgres_exporter
Restart=always

[Install]
WantedBy=multi-user.target
EOF

What does DATA_SOURCE_NAME do?

The DATA_SOURCE_NAME environment variable tells postgres_exporter how to connect to PostgreSQL. The connection string specifies the prometheus_monitor user, the password you chose, the host (localhost), port (5432), and the Canvas database name (canvas).

Replace your-password-here with the password you set for prometheus_monitor in the previous step.

  • Enable and start postgres_exporter by running:
sudo systemctl daemon-reload
sudo systemctl enable postgres_exporter
sudo systemctl start postgres_exporter
  • Verify postgres_exporter is serving metrics on port 9187 by running:
curl http://localhost:9187/metrics | head -20

You should see Prometheus-formatted metrics including lines with pg_ prefixes such as pg_stat_activity_count and pg_settings_max_connections.

Seeing a connection refused error?

If postgres_exporter fails to start or shows connection errors, check that the password in the DATA_SOURCE_NAME matches what you set in psql. Also verify that PostgreSQL is configured to allow password authentication for local connections in pg_hba.conf.

help me troubleshoot why postgres_exporter cannot connect to my Canvas PostgreSQL database

  • Exit the SSH session to the DB VM by running:
exit
Install node_exporter on the File Storage VM and verify network connectivity

The File Storage VM only needs node_exporter since it serves files without a database or web application layer.

  • SSH into your File Storage VM from WSL by running:
ssh [[STORAGE_VM_USER="your-username"]]@[[STORAGE_VM_IP="your storage VM IP address"]]
  • Download, install, and start node_exporter by running:
# Download and extract node_exporter
wget https://github.com/prometheus/node_exporter/releases/download/v1.11.1/node_exporter-1.11.1.linux-amd64.tar.gz
tar xvfz node_exporter-1.11.1.linux-amd64.tar.gz

# Create user and install binary
sudo useradd --no-create-home --shell /bin/false node_exporter
sudo cp node_exporter-1.11.1.linux-amd64/node_exporter /usr/local/bin/
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter
rm -rf node_exporter-1.11.1.linux-amd64 node_exporter-1.11.1.linux-amd64.tar.gz

# Create systemd service
sudo tee /etc/systemd/system/node_exporter.service > /dev/null << 'EOF'
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
Restart=always

[Install]
WantedBy=multi-user.target
EOF

# Enable and start
sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter
  • Verify node_exporter is running by running:
curl http://localhost:9100/metrics | head -5

You should see Prometheus-formatted metrics output.

  • Exit the SSH session to the File Storage VM by running:
exit

Now verify that all exporters are reachable from WSL over the network. This confirms there are no firewall rules blocking Prometheus from scraping.

  • Back in WSL, test connectivity to all exporters by running:
# Test App VM exporters
curl http://[[APP_VM_IP="your app VM IP address"]]:9100/metrics | head -5
curl http://[[APP_VM_IP="your app VM IP address"]]:9117/metrics | head -5

# Test DB VM exporters
curl http://[[DB_VM_IP="your DB VM IP address"]]:9100/metrics | head -5
curl http://[[DB_VM_IP="your DB VM IP address"]]:9187/metrics | head -5

# Test File Storage VM exporter
curl http://[[STORAGE_VM_IP="your storage VM IP address"]]:9100/metrics | head -5

Each command should return Prometheus-formatted metrics text. If any command hangs or returns a connection error, the corresponding VM's firewall is blocking the exporter port.

Connection timed out from WSL?

If curl works locally on the VM but times out from WSL, the VM's firewall is blocking inbound traffic on the exporter port. Open the required ports (9100, 9117, 9187) in your firewall rules. On Ubuntu with ufw, run sudo ufw allow 9100/tcp on the target VM.

help me troubleshoot network connectivity between WSL and my Ubuntu VMs for Prometheus exporter ports

All five exporters are deployed and verified reachable from WSL. In the next step, you will configure Prometheus to scrape all of these targets automatically and set up alerting rules to catch problems before your users do.

Configure Prometheus Multi-Target Scraping and Alerting Rules

Your exporters are live and serving metrics on every VM. But Prometheus is still scraping only itself.

In this step, you will point Prometheus at all 5 remote exporter endpoints across your 3 VMs. You will also define alerting rules that fire before your users notice problems. By the end of this step, Prometheus becomes a centralized monitoring hub with proactive alerting wired to Alertmanager.

In this step, get ready to:
  • Update the Prometheus scrape configuration to pull metrics from all remote VMs.
  • Create alerting rules for CPU, disk, database connections, and HTTP errors.
  • Restart Prometheus and verify all targets report UP.
Update the Prometheus scrape configuration

Prometheus uses a pull-based model. It reads a list of targets from its configuration file and scrapes each one on a fixed interval. Right now the only target is localhost:9090. You need to add your 3 VMs as targets with their respective exporter ports.

  • Open /etc/prometheus/prometheus.yml in a text editor by running:
sudo nano /etc/prometheus/prometheus.yml
  • Replace the entire contents of the file with the following configuration:
# Prometheus global settings
global:
  scrape_interval: 15s
  evaluation_interval: 15s

# Connect to Alertmanager for routing alerts
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - localhost:9093

# Scrape targets: Prometheus itself plus all remote exporters
scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node"
    static_configs:
      - targets: ["[[APP_VM_IP=\"your app VM's IP address\"]]:9100"]
        labels:
          role: "app"
      - targets: ["[[DB_VM_IP=\"your DB VM's IP address\"]]:9100"]
        labels:
          role: "db"
      - targets: ["[[STORAGE_VM_IP=\"your storage VM's IP address\"]]:9100"]
        labels:
          role: "storage"

  - job_name: "apache"
    static_configs:
      - targets: ["[[APP_VM_IP=\"your app VM's IP address\"]]:9117"]
        labels:
          role: "app"

  - job_name: "postgres"
    static_configs:
      - targets: ["[[DB_VM_IP=\"your DB VM's IP address\"]]:9187"]
        labels:
          role: "db"

What does this configuration do?

The alerting section tells Prometheus where to send fired alerts. It points to your Alertmanager instance on localhost:9093.

Each job_name groups related targets. The node job scrapes system metrics from all 3 VMs. The apache job scrapes web server metrics from the app VM only. The postgres job scrapes database metrics from the DB VM only.

The role labels let you filter and group metrics by VM role later in PromQL queries and Grafana dashboards.

  • Save the file and exit the editor (in nano: Ctrl+O, Enter, then Ctrl+X).

You will not restart Prometheus yet. The next substep creates the alerting rules file that this configuration will reference.

Not sure what your VM IPs are?

Use the same IP addresses you used to SSH into each VM earlier. You can also find them in your cloud provider's console under the instance details page.

help me find my VM IP addresses for the Prometheus scrape config

Create the alerting rules file

Alerting rules define conditions that, when true for a specified duration, fire an alert to Alertmanager. You will create 4 rules covering the most critical failure signals for an LMS serving 10K concurrent users.

  • Create the alerting rules file by running:
sudo nano /etc/prometheus/alerting_rules.yml
  • Paste the following alerting rules into the file:
groups:
  - name: infrastructure
    rules:
      # Alert when CPU usage exceeds 85% for 5 minutes
      - alert: HighCpuUsage
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: "CPU usage is above 85% for more than 5 minutes (current value: {{ $value }}%)."

      # Alert when less than 15% disk space remains on root filesystem
      - alert: LowDiskSpace
        expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 15
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Low disk space on {{ $labels.instance }}"
          description: "Less than 15% disk space remaining on root filesystem (current value: {{ $value }}%)."

      # Alert when PostgreSQL connections exceed 80% of max
      - alert: PostgresConnectionSaturation
        expr: sum(pg_stat_activity_count) / pg_settings_max_connections > 0.8
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "PostgreSQL connection pool near saturation"
          description: "More than 80% of max connections are in use (current value: {{ $value }})."

      # Alert when more than 5% of Apache requests return 5xx errors
      - alert: HighApacheErrorRate
        expr: rate(apache_accesses_total{status=~"5.."}[5m]) / rate(apache_accesses_total[5m]) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High Apache 5xx error rate"
          description: "More than 5% of requests are returning 5xx errors (current value: {{ $value }})."

What do these rules do?

  • HighCpuUsage calculates the percentage of time CPUs spend NOT idle, then fires if it stays above 85% for 5 minutes. This catches runaway processes before they starve Canvas workers.
  • LowDiskSpace checks the ratio of available bytes to total bytes on the root filesystem. Firing at 15% gives you time to act before disk-full crashes.
  • PostgresConnectionSaturation compares active connections to the configured maximum. At 80% saturation, new student requests start queuing.
  • HighApacheErrorRate divides 5xx responses by total responses over a 5-minute window. A 5% threshold catches error spikes during enrollment rushes or exam submissions.

The for: 5m clause on every rule means Prometheus must see the condition continuously for 5 minutes before it fires. This prevents one-off spikes from waking you up at 3 AM.

  • Save the file and exit the editor.

The rules file exists on disk but Prometheus does not know about it yet. The next substep wires it in and restarts the service.

YAML indentation errors?

YAML is whitespace-sensitive. Every level of indentation must use exactly 2 spaces, never tabs. If Prometheus fails to start after the next substep, check that your expr lines align with the alert lines above them.

help me fix YAML indentation in my Prometheus alerting rules file

Wire the rules file and restart Prometheus

Prometheus only loads rule files that are explicitly listed in the rule_files section of its configuration. You need to add this reference, then restart the service so it picks up both the new scrape targets and the alerting rules.

  • Open /etc/prometheus/prometheus.yml again:
sudo nano /etc/prometheus/prometheus.yml
  • Find the blank line between the alerting: section and the scrape_configs: section.
  • Add the following lines in that gap, so they sit between alerting and scrape_configs:
rule_files:
  - "alerting_rules.yml"

What does rule_files do?

The rule_files directive tells Prometheus which files contain recording or alerting rules. Prometheus evaluates these rules at the evaluation_interval (every 15 seconds in your config). Paths are relative to the directory containing prometheus.yml, which is /etc/prometheus/.

  • Save the file and exit the editor.
  • Restart Prometheus to load the updated configuration by running:
sudo systemctl restart prometheus
  • Verify the service started successfully by running:
sudo systemctl status prometheus
  • You should see active (running) in the output.
  • Open your Windows browser and navigate to http://localhost:9090.
  • Click Status in the top navigation bar, then click Targets.
  • You should see all 5 remote targets (3 node exporters, 1 apache exporter, 1 postgres exporter) with state UP in green.
  • Click Status again, then click Rules.
  • You should see the infrastructure group with all 4 alert rules listed and no errors.

Targets showing DOWN or not appearing?

  • Check that the IP addresses in your prometheus.yml match the actual IPs of your VMs. A typo in one digit will cause a connection timeout.
  • Verify the exporter is still running on the target VM by SSHing in and running sudo systemctl status node_exporter (or the relevant exporter service).
  • Confirm no firewall is blocking the exporter ports (9100, 9117, 9187) between WSL and the VMs.

my Prometheus targets are showing DOWN or not appearing at all

✔️ Awesome, I've got everything!

Great. Double check you have saved both files and that your Targets page shows all endpoints as UP.

ⓧ I'd like to double check the full code

Here is the complete /etc/prometheus/prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - localhost:9093

rule_files:
  - "alerting_rules.yml"

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node"
    static_configs:
      - targets: ["[[APP_VM_IP=\"your app VM's IP address\"]]:9100"]
        labels:
          role: "app"
      - targets: ["[[DB_VM_IP=\"your DB VM's IP address\"]]:9100"]
        labels:
          role: "db"
      - targets: ["[[STORAGE_VM_IP=\"your storage VM's IP address\"]]:9100"]
        labels:
          role: "storage"

  - job_name: "apache"
    static_configs:
      - targets: ["[[APP_VM_IP=\"your app VM's IP address\"]]:9117"]
        labels:
          role: "app"

  - job_name: "postgres"
    static_configs:
      - targets: ["[[DB_VM_IP=\"your DB VM's IP address\"]]:9187"]
        labels:
          role: "db"

Here is the complete /etc/prometheus/alerting_rules.yml:

groups:
  - name: infrastructure
    rules:
      - alert: HighCpuUsage
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: "CPU usage is above 85% for more than 5 minutes (current value: {{ $value }}%)."

      - alert: LowDiskSpace
        expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 15
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Low disk space on {{ $labels.instance }}"
          description: "Less than 15% disk space remaining on root filesystem (current value: {{ $value }}%)."

      - alert: PostgresConnectionSaturation
        expr: sum(pg_stat_activity_count) / pg_settings_max_connections > 0.8
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "PostgreSQL connection pool near saturation"
          description: "More than 80% of max connections are in use (current value: {{ $value }})."

      - alert: HighApacheErrorRate
        expr: rate(apache_accesses_total{status=~"5.."}[5m]) / rate(apache_accesses_total[5m]) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High Apache 5xx error rate"
          description: "More than 5% of requests are returning 5xx errors (current value: {{ $value }})."

Your observability hub is now pulling real metrics from every layer of your Canvas LMS architecture. Next up, you will install Grafana and build dashboards that turn these raw numbers into actionable visualizations.

Install Grafana and Build LMS Dashboards

Your monitoring stack is now collecting metrics from all three VMs and has alerting rules ready to fire. The missing piece is visibility. Raw metrics stored in Prometheus are powerful for querying, but you need dashboards that surface problems at a glance.

Grafana is the standard visualization layer for Prometheus. In this step, you will install Grafana inside WSL, connect it to your Prometheus instance, and build a dashboard focused on the RED signals (Rate, Errors, Duration) for your Canvas LMS infrastructure.

In this step, get ready to:
  • Install Grafana OSS and start the service.
  • Connect Grafana to Prometheus as a data source.
  • Build a Canvas LMS Infrastructure dashboard with 6 panels showing live system metrics.
Install Grafana OSS via the APT repository

Grafana OSS is the open-source edition that provides full dashboard and data source capabilities. Installing via the official APT repository ensures you get version 13.0.2 and automatic updates.

  • In your WSL terminal, install the APT prerequisites by running:
sudo apt-get install -y apt-transport-https wget gnupg
  • Import the Grafana GPG key and add the APT repository by running these commands:
sudo mkdir -p /etc/apt/keyrings
sudo wget -O /etc/apt/keyrings/grafana.asc https://apt.grafana.com/gpg-full.key
sudo chmod 644 /etc/apt/keyrings/grafana.asc
echo "deb [signed-by=/etc/apt/keyrings/grafana.asc] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
  • Update the package list and install Grafana OSS by running:
sudo apt-get update
sudo apt-get install grafana

What do these commands do?

The first set of commands imports Grafana's signing key so your system trusts packages from their repository. The echo command adds Grafana's official APT repository to your sources list.

After updating, apt-get install grafana pulls the Grafana OSS package. This also creates a grafana system user and installs a systemd service called grafana-server.

  • Enable and start the Grafana service by running:
sudo systemctl daemon-reload
sudo systemctl enable grafana-server.service
sudo systemctl start grafana-server
  • Verify the service is running by running:
sudo systemctl status grafana-server

You should see active (running) in the output.

  • Open your Windows browser and navigate to http://localhost:3000.
  • Log in with the default username admin and password admin.
  • Grafana prompts you to change the admin password. Enter a new password and click Submit.

Grafana UI not loading?

Make sure the service is running with sudo systemctl status grafana-server. If it shows failed, check the logs with sudo journalctl -u grafana-server -n 50.

If the browser cannot reach localhost:3000, confirm WSL networking is functioning by testing http://localhost:9090 (Prometheus) first.

Help me troubleshoot Grafana not starting in WSL

Add Prometheus as a data source

Grafana needs a data source to query metrics from. Since Prometheus is already running at localhost:9090 inside the same WSL instance, the connection is straightforward.

  • In the Grafana UI, click the hamburger menu icon in the top-left corner.
  • Select Connections from the left sidebar.
  • Click Data sources.
  • Click Add data source.
  • Select Prometheus from the list.
  • In the Prometheus server URL field, enter http://localhost:9090.
  • Scroll to the bottom and click Save & test.

You should see a green banner confirming the data source is working.

Connection test failing?

Verify Prometheus is still running with sudo systemctl status prometheus in your WSL terminal. The URL must be exactly http://localhost:9090 with no trailing slash.

Help me fix the Prometheus data source connection in Grafana

Create the Canvas LMS Infrastructure dashboard

Now you will build a dashboard with 6 panels covering the RED signals for your Canvas LMS infrastructure. Each panel uses a PromQL query targeting the metrics your exporters are already sending.

  • Click the + icon in the top-right area of Grafana.
  • Select New dashboard.
  • Click Add visualization.
  • Select your Prometheus data source.

You are now in the panel editor. Build each of the following 6 panels one at a time.

  • For Panel 1 (CPU Usage Per VM), enter this PromQL query in the query editor:
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
  • Set the panel title to CPU Usage Per VM.
  • Click Apply in the top-right corner to save the panel.

Repeat the same process for each additional panel. Click Add visualization each time to create a new panel, select the Prometheus data source, enter the query, set the title, and click Apply.

  • For Panel 2 (Memory Usage Per VM), enter this query:
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100
  • For Panel 3 (Disk Space Remaining), enter this query:
(node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100
  • For Panel 4 (Apache Requests Per Second), enter this query:
rate(apache_accesses_total[5m])
  • For Panel 5 (Apache Response Duration), enter this query:
rate(apache_duration_ms_total[5m]) / rate(apache_accesses_total[5m])
  • For Panel 6 (PostgreSQL Active Connections), enter this query:
pg_stat_activity_count

What do these queries measure?

  • CPU Usage calculates the percentage of CPU time spent on non-idle work, averaged per instance.
  • Memory Usage shows what percentage of total memory is in use (not available) on each VM.
  • Disk Space Remaining shows the percentage of free space on the root filesystem.
  • Apache Requests Per Second is the rate of incoming HTTP requests to your Canvas app.
  • Apache Response Duration divides total request duration by total requests to get average milliseconds per request.
  • PostgreSQL Active Connections shows how many database sessions are open, helping you spot connection pool exhaustion.
  • After creating all 6 panels, click the Save dashboard icon (floppy disk) in the top-right corner.
  • Enter Canvas LMS Infrastructure as the dashboard name.
  • Click Save.

All 6 panels should now display live data from your VMs. CPU, memory, and disk panels show data for all 3 VMs. Apache and PostgreSQL panels show data for the app and DB VMs respectively.

Panels showing No Data?

Open the Prometheus UI at http://localhost:9090/targets and confirm all targets still show state UP. If a target is down, the exporter service on that VM may have stopped.

Check the time range selector in the top-right of Grafana. Set it to Last 15 minutes to ensure you are looking at recent data.

Help me troubleshoot empty Grafana panels with Prometheus data source

Your observability stack now has full visibility into your Canvas LMS infrastructure. Next up, you will verify the end-to-end alerting pipeline by triggering a test alert and watching it flow from Prometheus through Alertmanager.

Verify the End-to-End Alerting Pipeline

Your Grafana dashboard is live and displaying real-time metrics from all three VMs. You now have visibility into CPU, memory, disk, web traffic, and database connections across your entire Canvas LMS architecture.

Visibility alone is not enough. You need confidence that your alerting pipeline actually fires when something goes wrong. In this step, you will intentionally trigger a test alert and trace it through Prometheus into Alertmanager, proving the entire loop works before a real incident forces you to find out the hard way.

In this step, get ready to:
  • Trigger a test alert by temporarily lowering a threshold.
  • Confirm the alert propagates from Prometheus to Alertmanager.
  • Restore original thresholds and document timing behavior.
Lower the CPU threshold to trigger a test alert

The safest way to test an alerting pipeline is to make a rule fire on purpose. You will temporarily set the CPU threshold to 1%, which is low enough that every VM will exceed it immediately.

  • In your WSL terminal, open the alerting rules file by running:
sudo nano /etc/prometheus/alerting_rules.yml
  • Find the HighCpuUsage rule's expr line:
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
  • Replace > 85 with > 1, so the line now looks like this:
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 1
  • Save the file and exit nano (Ctrl+O, Enter, then Ctrl+X).
  • Restart Prometheus to pick up the new rule by running:
sudo systemctl restart prometheus
  • Verify Prometheus restarted successfully by running:
sudo systemctl status prometheus

You should see active (running) in the output.

Why set the threshold to 1%?

Any running server uses more than 1% CPU at all times. This guarantees the alert expression evaluates to true immediately, without needing to generate artificial load.

The for: 5m clause means the alert must remain true for 5 consecutive minutes of evaluation before it transitions from PENDING to FIRING. With your evaluation_interval: 15s, Prometheus checks this rule every 15 seconds during that 5-minute window.

Prometheus failed to restart?

A YAML syntax error in the rules file is the most common cause. Run journalctl -u prometheus --no-pager -n 20 to see the exact error line. Make sure you only changed the number 85 to 1 and did not accidentally remove any spaces or quotes.

help me fix a Prometheus restart failure after editing alerting_rules.yml

Confirm the alert fires in Prometheus and Alertmanager

Prometheus evaluates rules every 15 seconds. The alert will enter PENDING state almost immediately, then transition to FIRING after the 5-minute for duration elapses.

  • In your Windows browser, navigate to http://localhost:9090.
  • Click Alerts in the top navigation bar.
  • Look for HighCpuUsage in the list. Within a few seconds of the restart, it should appear with a yellow PENDING badge.

What do PENDING and FIRING mean?

PENDING means the alert condition is true, but the for duration has not yet elapsed. Prometheus keeps evaluating, and if the condition stays true for the full 5 minutes, the state transitions to FIRING.

Only FIRING alerts get sent to Alertmanager. This two-stage design prevents transient spikes from triggering false notifications.

  • Wait approximately 5 minutes, then refresh the Alerts page.
  • The HighCpuUsage alert should now show a red FIRING badge with entries for each VM instance.
  • Expand one of the firing alert entries and confirm you see the severity: warning label and the summary annotation containing the instance address.

Now verify the alert reached Alertmanager.

  • Open a new browser tab and navigate to http://localhost:9093.
  • You should see the HighCpuUsage alert listed as an active alert, grouped by alertname and instance.
  • Click on the alert group to expand it and verify the labels (severity, instance) and annotations (summary, description) are correct.

What is group_wait?

Your Alertmanager configuration has group_wait: 30s. This means Alertmanager waits 30 seconds after receiving the first alert in a new group before sending the notification. This batching window lets it collect related alerts into a single notification rather than sending one per instance.

Alert stuck in PENDING after 5 minutes?

Make sure you changed the correct line. The threshold must be on the HighCpuUsage rule's expr field, not on a different rule. Also confirm you restarted Prometheus after saving. Check Status > Rules in the Prometheus UI to verify the rule loaded with the > 1 threshold.

help me debug why my Prometheus alert is stuck in PENDING

Restore the original threshold and confirm resolution

The test proved the pipeline works. Now restore the production threshold so you only get alerted for real problems.

  • Open the alerting rules file again by running:
sudo nano /etc/prometheus/alerting_rules.yml
  • Find the HighCpuUsage rule's expr line that currently reads:
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 1
  • Replace > 1 with > 85, restoring the original threshold:
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
  • Save the file and exit nano (Ctrl+O, Enter, then Ctrl+X).
  • Restart Prometheus by running:
sudo systemctl restart prometheus
  • Return to the Prometheus Alerts page at http://localhost:9090 and refresh.
  • The HighCpuUsage alert should transition back to an inactive (green) state within one evaluation cycle since CPU is no longer above 85%.
  • Switch to the Alertmanager UI at http://localhost:9093.
  • The alert should resolve within the configured resolve_timeout: 5m window. Once resolved, it disappears from the active alerts list.

How does resolve_timeout work?

Your Alertmanager configuration sets resolve_timeout: 5m. This means if Alertmanager stops receiving a FIRING alert from Prometheus, it waits 5 minutes before marking it as resolved. Since your Prometheus config sends alert state updates every evaluation_interval (15s), the resolution is communicated quickly once the condition is no longer true.

The group_interval: 5m setting controls how long Alertmanager waits before sending updates about an existing alert group. Combined with group_wait: 30s for new groups, these timings define how responsive your alerting pipeline is to state changes.

✔️ Awesome, I've got everything!

Great. Double check that you have saved the file with the threshold restored to > 85.

ⓧ I'd like to double check the full code

Here is the complete /etc/prometheus/alerting_rules.yml file as it should look now:

groups:
  - name: infrastructure
    rules:
      - alert: HighCpuUsage
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: "CPU usage is above 85% for more than 5 minutes (current value: {{ $value }}%)."

      - alert: LowDiskSpace
        expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 15
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Low disk space on {{ $labels.instance }}"
          description: "Less than 15% disk space remaining on root filesystem (current value: {{ $value }}%)."

      - alert: PostgresConnectionSaturation
        expr: sum(pg_stat_activity_count) / pg_settings_max_connections > 0.8
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "PostgreSQL connection pool near saturation"
          description: "More than 80% of max connections are in use (current value: {{ $value }})."

      - alert: HighApacheErrorRate
        expr: rate(apache_accesses_total{status=~"5.."}[5m]) / rate(apache_accesses_total[5m]) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High Apache 5xx error rate"
          description: "More than 5% of requests are returning 5xx errors (current value: {{ $value }})."

Alert not resolving in Alertmanager?

Alertmanager's resolve_timeout means it can take up to 5 minutes for a resolved alert to disappear from the UI. Wait the full 5 minutes and refresh. If it persists beyond that, confirm Prometheus actually restarted with the > 85 threshold by checking Status > Rules in the Prometheus UI.

help me debug why my alert is not resolving in Alertmanager

Secret mission

Canvas Application Metrics Dashboard

Your infrastructure dashboard tells you when hardware is struggling. But what about the application layer? Build a second Grafana dashboard that tracks active database sessions, query activity, and Apache worker saturation during peak load periods. This requires intermediate PromQL with label filtering and rate calculations tailored to your LMS workload patterns.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely on open-source tools within your existing infrastructure, so there are no ongoing cloud costs.

Resources you used:

  • Prometheus service and binaries in WSL.
  • Alertmanager service and binaries in WSL.
  • Grafana installation in WSL (APT package, dashboards, data source).
  • node_exporter on all 3 VMs (App, DB, Storage).
  • apache_exporter on the App VM.
  • postgres_exporter on the DB VM.
  • PostgreSQL monitoring user (prometheus_monitor) on the DB VM.
  • Prometheus configuration and data directories in WSL.
  • Alertmanager configuration and data directories in WSL.

Keep everything running

No action needed. Choose this if you are still actively building on your observability stack or plan to extend it with additional dashboards, exporters, or alert routes.

  • All services continue running automatically via systemd on each VM and in WSL.
  • Prometheus retains its time-series data in /var/lib/prometheus/ (default retention is 15 days).
  • Grafana dashboards and data source configuration persist across reboots.

Pause - I'll come back to this later

Stop all monitoring services to free up CPU and memory, while keeping your configuration files and binaries in place for a quick restart later.

In your WSL terminal, stop the local services by running:

sudo systemctl stop prometheus alertmanager grafana-server
sudo systemctl disable prometheus alertmanager grafana-server

On each VM, stop the exporters by running:

# Run on the App VM
sudo systemctl stop node_exporter apache_exporter
sudo systemctl disable node_exporter apache_exporter

# Run on the DB VM
sudo systemctl stop node_exporter postgres_exporter
sudo systemctl disable node_exporter postgres_exporter

# Run on the Storage VM
sudo systemctl stop node_exporter
sudo systemctl disable node_exporter

To resume later, replace stop with start and disable with enable in each command above.

Delete - I don't want to use this again

Remove all project resources completely. This cannot be undone.

Start by removing Grafana in your WSL terminal:

sudo systemctl stop grafana-server
sudo systemctl disable grafana-server
sudo apt-get remove grafana
sudo rm -rf /var/lib/grafana
sudo rm -rf /etc/grafana
sudo rm -i /etc/apt/sources.list.d/grafana.list

Remove Prometheus from WSL:

sudo systemctl stop prometheus
sudo systemctl disable prometheus
sudo rm /etc/systemd/system/prometheus.service
sudo rm /usr/local/bin/prometheus /usr/local/bin/promtool
sudo rm -rf /etc/prometheus
sudo rm -rf /var/lib/prometheus
sudo userdel prometheus

Remove Alertmanager from WSL:

sudo systemctl stop alertmanager
sudo systemctl disable alertmanager
sudo rm /etc/systemd/system/alertmanager.service
sudo rm /usr/local/bin/alertmanager /usr/local/bin/amtool
sudo rm -rf /etc/alertmanager
sudo rm -rf /var/lib/alertmanager
sudo userdel alertmanager

Reload systemd in WSL to clear removed unit files:

sudo systemctl daemon-reload

SSH into the App VM and remove its exporters:

sudo systemctl stop node_exporter apache_exporter
sudo systemctl disable node_exporter apache_exporter
sudo rm /etc/systemd/system/node_exporter.service
sudo rm /etc/systemd/system/apache_exporter.service
sudo rm /usr/local/bin/node_exporter
sudo rm /usr/local/bin/apache_exporter
sudo userdel node_exporter
sudo userdel apache_exporter
sudo systemctl daemon-reload

SSH into the DB VM and remove its exporters and monitoring user:

sudo systemctl stop node_exporter postgres_exporter
sudo systemctl disable node_exporter postgres_exporter
sudo rm /etc/systemd/system/node_exporter.service
sudo rm /etc/systemd/system/postgres_exporter.service
sudo rm /usr/local/bin/node_exporter
sudo rm /usr/local/bin/postgres_exporter
sudo userdel node_exporter
sudo userdel postgres_exporter
sudo systemctl daemon-reload

Drop the PostgreSQL monitoring user on the DB VM:

sudo -u postgres psql -c "DROP USER IF EXISTS prometheus_monitor;"

SSH into the Storage VM and remove node_exporter:

sudo systemctl stop node_exporter
sudo systemctl disable node_exporter
sudo rm /etc/systemd/system/node_exporter.service
sudo rm /usr/local/bin/node_exporter
sudo userdel node_exporter
sudo systemctl daemon-reload

Nice Work!

Nice Work!

You just built a production-grade observability stack from scratch. Your Canvas LMS infrastructure now has real-time visibility powered by Prometheus, Grafana, and Alertmanager, all running in WSL and scraping live metrics from three remote VMs over the network.

You've learned how to:

  • Deploy Prometheus in WSL as a centralized metrics collector with multi-target scraping across a remote 3-VM architecture, pulling system, database, and web server metrics using node_exporter, postgres_exporter, and apache_exporter.
  • Build a Grafana dashboard visualizing RED signals (Rate, Errors, Duration) tailored to a high-scale LMS workload serving 10K concurrent users.
  • Configure Alertmanager with proactive alerting rules for CPU, disk, connection saturation, and error rates, then verify the complete end-to-end alerting pipeline from metric collection through to alert firing.
  • Secret Mission: Created a Canvas-specific application dashboard tracking database session states, query patterns, and Apache worker saturation for peak-load capacity planning.

Ready to quiz yourself?