Monitor a System with Prometheus and Grafana

Build a full observability stack with metrics, dashboards, and alerts.

Introduction

30 Second Summary

Imagine your app goes down at 3 AM and nobody notices until customers start complaining hours later. In production, every team needs a way to watch their systems automatically and get alerted before small problems turn into outages.

In this project, you will build a complete monitoring stack on your local machine using Prometheus, Grafana, and Alertmanager. You will collect real system metrics, visualize them on a live dashboard, and configure alerts that fire automatically when something goes wrong.

What You'll Build

Open Grafana in your browser and see your machine's CPU, memory, and disk usage updating every 15 seconds, while Prometheus and Alertmanager watch for problems and catch them automatically behind the scenes.

By the end of this project, you'll have:

A monitoring pipeline launched with a single Docker Compose command, where four services work together to collect your machine's real metrics every 15 seconds.
A live Grafana dashboard where you see your CPU, memory, and disk usage updating side by side in real time.
Alert rules that detect problems automatically, with an alert you deliberately trigger and watch fire end-to-end from detection to notification.
Secret Mission: Write a predictive alert using predict_linear that forecasts disk space exhaustion before it happens.

Do I need anything before starting?

You need Docker Desktop installed and running on your Windows machine. Everything else in this project runs inside Docker containers, so no additional tools or accounts are required.

Before We Start

Before you write any configuration files, take a moment to lock in what you are about to build and why it matters.

Set Up the Monitoring Stack

Before you can observe anything, you need a running metrics pipeline. Prometheus and Node Exporter form the foundation of this stack.

Node Exporter exposes your machine's system metrics (CPU, memory, disk). Prometheus scrapes those metrics every 15 seconds and stores them as time-series data. In this step, you'll define both services in a Docker Compose file and verify that Prometheus is collecting data.

In this step, get ready to:

Create the project directory structure and Docker Compose configuration.
Write the Prometheus scrape configuration.
Start the stack and verify metrics collection.

Create the project directory and Docker Compose file

Every file for this project lives inside a single folder. You'll create that folder on your Desktop so it's easy to find.

Press the Windows key to open your search bar.
Type Terminal and press Enter to open it.

Move to your Desktop and create the project folder by running these commands:

cd ~/Desktop
mkdir monitoring-stack
cd monitoring-stack
mkdir prometheus

Confirm the folders exist by running:

ls

You should see a prometheus folder listed in the output.

Don't see the prometheus folder?

Make sure you ran all four commands in order. If you see an error on the cd ~/Desktop line, try cd $HOME\Desktop instead.

Why can't I create the project directory?

Now you'll define the two services that make up the initial stack. The first service is Prometheus itself.

Create a new file called docker-compose.yml in your monitoring-stack folder using your preferred text editor, and add the Prometheus service definition:

# Define the services that make up the monitoring stack
services:
  prometheus:
    image: prom/prometheus:v3.14.0
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      # Mount the Prometheus config file (read-only)
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      # Persist collected metrics data across container restarts
      - prometheus-data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.retention.time=7d'
      - '--web.enable-lifecycle'
    restart: unless-stopped
    networks:
      - monitoring

What does this code do?

image: prom/prometheus:v3.14.0 pulls the official Prometheus Docker image at a pinned version, so your stack is reproducible.
The volumes section mounts your local config file into the container and creates a named volume for storing scraped metrics.
The command flags tell Prometheus where its config lives, how long to keep data (7 days), and to allow configuration reloads via HTTP.
networks: monitoring puts the container on a shared network so it can reach other services by name.

Below the Prometheus service block in the same file, add the Node Exporter service, volumes, and networks:

  # Exposes host-level system metrics for Prometheus to scrape
  node-exporter:
    image: prom/node-exporter:v1.12.1
    container_name: node-exporter
    ports:
      - "9100:9100"
    restart: unless-stopped
    networks:
      - monitoring

# Named volume to persist Prometheus time-series data
volumes:
  prometheus-data:

# Shared network so containers can reach each other by name
networks:
  monitoring:
    driver: bridge

What does this code do?

The node-exporter service runs a lightweight agent that exposes CPU, memory, and disk metrics on port 9100.
The volumes section at the bottom declares the prometheus-data named volume so your metrics survive container restarts.
The networks section creates a bridge network called monitoring. Both services join this network, letting Prometheus reach Node Exporter by its service name.

Save docker-compose.yml.

Seeing a YAML syntax error later?

YAML is sensitive to indentation. Make sure every level uses exactly 2 spaces, not tabs. The node-exporter block should be indented at the same level as prometheus (2 spaces under services).

Help me fix the YAML syntax error.

✔️ Awesome, I've got everything!

Great. Make sure you've saved docker-compose.yml before moving on.

ⓧ I'd like to double check the full code

Here is the complete docker-compose.yml file. Compare yours against it:

services:
  prometheus:
    image: prom/prometheus:v3.14.0
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.retention.time=7d'
      - '--web.enable-lifecycle'
    restart: unless-stopped
    networks:
      - monitoring

  node-exporter:
    image: prom/node-exporter:v1.12.1
    container_name: node-exporter
    ports:
      - "9100:9100"
    restart: unless-stopped
    networks:
      - monitoring

volumes:
  prometheus-data:

networks:
  monitoring:
    driver: bridge

Write the Prometheus configuration

Prometheus needs a configuration file that tells it what to scrape and how often. This file defines the scrape interval and the list of targets.

Create a new file called prometheus.yml inside the prometheus folder (the one you created earlier inside monitoring-stack), and add the following content:

global:
  # How often Prometheus scrapes each target
  scrape_interval: 15s
  # How often Prometheus evaluates alert rules
  evaluation_interval: 15s

scrape_configs:
  # Prometheus scrapes its own metrics
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Scrape system metrics from Node Exporter
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']

What does this configuration do?

scrape_interval: 15s tells Prometheus to pull metrics from every target once every 15 seconds. This is the pull-based model that makes Prometheus different from push-based monitoring tools.
The first job, prometheus, scrapes Prometheus's own internal metrics at localhost:9090. Monitoring your monitoring tool is standard practice.
The second job, node-exporter, scrapes system metrics at node-exporter:9100. The hostname node-exporter resolves to the container because both services share the monitoring network.

What about the metrics on Windows?

Node Exporter is designed for Linux systems. On Docker Desktop for Windows, containers run inside a WSL2 Linux VM. Node Exporter reports metrics from that VM, not Windows itself.

This is perfectly fine for learning. The entire scraping, querying, and alerting pipeline works identically. The metrics are real. They just come from the WSL2 VM rather than Windows directly.

Save prometheus/prometheus.yml.
Confirm the file exists by running:

ls prometheus

You should see prometheus.yml listed in the output.

Don't see prometheus.yml?

Make sure you saved the file inside the prometheus subfolder, not in the monitoring-stack root. The full path should be monitoring-stack/prometheus/prometheus.yml.

Help me find where my file went.

✔️ Awesome, I've got everything!

Great. Make sure you've saved prometheus/prometheus.yml before moving on.

ⓧ I'd like to double check the full code

Here is the complete prometheus/prometheus.yml file. Compare yours against it:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']

Start the stack and verify metrics collection

Both configuration files are in place. Time to start the containers and see Prometheus in action.

Start the stack by running this command:

docker compose up -d

What does this command do?

docker compose up reads your docker-compose.yml, pulls the required images, creates the containers, and starts them. The -d flag runs everything in the background so you keep your terminal.

The first run pulls both Docker images, which may take a minute or two depending on your internet speed.

Verify both containers are running by running:

docker compose ps

You should see two rows: prometheus and node-exporter, both with a STATUS that includes Up.

Seeing a container exit or not starting?

If the prometheus container exited, check for YAML errors in your config by running docker compose logs prometheus. A common cause is incorrect indentation in prometheus.yml.
If neither container appears, make sure Docker Desktop is running. You should see the Docker whale icon in your system tray.
If you see a port conflict error, another process is already using port 9090 or 9100. Close the conflicting application and run docker compose up -d again.

Help me get my containers running.

Now open your browser and go to http://localhost:9090. You should see the Prometheus web UI load.

Before you check the targets, think about this: Prometheus has been running for a few seconds. Do you think it has already discovered and successfully scraped Node Exporter?

In the Prometheus UI, click Status in the top navigation bar.
Select Targets from the dropdown.

You should see two target groups: node-exporter and prometheus. Both should show a State of UP in green, with a recent Last Scrape time. Prometheus discovered Node Exporter on the shared network and is already collecting system metrics every 15 seconds.

Seeing a target with State DOWN?

If node-exporter shows DOWN, run docker compose ps in your terminal to confirm the container is still running.
If the node-exporter container is up but the target is down, check that both services are on the same network by running docker network inspect monitoring-stack_monitoring.
Wait 15-30 seconds and refresh the Targets page. The first scrape may not have completed yet.

Help me debug why my target is down.

Your metrics pipeline is live. Prometheus is scraping real system data from Node Exporter every 15 seconds. Next up, you'll query that data with PromQL and see what your machine is actually doing.

Query Live Metrics with PromQL

Your monitoring stack is running and Prometheus is scraping metrics from Node Exporter every 15 seconds. But raw metric names like node_cpu_seconds_total are just numbers without context.

PromQL is the query language that transforms raw time-series data into meaningful information. In this step, you'll write queries that turn counters into percentages and rates. You'll also discover a key limitation of the Prometheus UI that sets up the next step.

In this step, get ready to:

Confirm your scrape targets are healthy and explore raw metric data.
Write PromQL queries that transform raw counters into CPU and memory percentages.
Experience the Prometheus UI's single-query limitation firsthand.

Confirm your targets and explore raw metrics

Before writing any queries, you need to confirm that Prometheus is successfully scraping data from your targets. The up metric is the simplest way to check this.

Return to the Prometheus UI at http://localhost:9090 in your browser.
Click Status in the top navigation bar, then click Targets.
Confirm that node-exporter shows a State of UP with a recent Last Scrape time.

Now head to the expression browser where you'll run your first queries.

Click Query in the top navigation bar to open the expression browser.
Paste the following query into the expression input field and click Execute:

up

You should see two rows in the Table tab. Each row has a value of 1, meaning the target is up and being scraped successfully.

What does the up metric tell you?

Prometheus automatically generates a special metric called up for every scrape target. A value of 1 means the target was reachable at the last scrape. A value of 0 means it was down.

Each row also has labels like job and instance that identify which target the metric belongs to. These labels come from your scrape_configs in prometheus.yml.

Next, look at a real metric that Node Exporter is collecting from your machine.

Clear the expression input field and paste this query, then click Execute:

node_cpu_seconds_total

You should see many rows appear, each with different labels like mode="idle", mode="system", and mode="user". The values are large and keep growing.

What is a counter?

The metric node_cpu_seconds_total is a counter. Counters only go up. They track the total cumulative number of CPU seconds spent in each mode since the system started.

A raw counter value like 54321.7 is not useful on its own. To get a meaningful reading, you need to calculate the rate of change over time. That's what PromQL's rate() function does.

Transform raw counters into useful metrics

Raw counters are meaningless on their own. PromQL functions like rate() transform them into per-second rates that you can actually use for monitoring.

Clear the expression input field and paste this query to see the CPU idle rate, then click Execute:

rate(node_cpu_seconds_total{mode="idle"}[5m])

Click the Graph tab (below the expression field) to see the result as a line chart over time.

You should see a line hovering near 1. A value of 1 means one full CPU second spent idle per wall-clock second.

What does rate() do?

The rate() function calculates the per-second average rate of increase of a counter over a time window. Here, [5m] tells Prometheus to look at the last 5 minutes of samples and compute the average rate across that window.

The {mode="idle"} part is a label matcher. It filters the time series to only include samples where the mode label equals idle.

Now try a different metric. Instead of CPU, check how much memory is available.

Switch back to the Table tab.
Clear the expression input field and paste this query, then click Execute:

node_memory_MemAvailable_bytes / 1024 / 1024

You should see a single row with a number in the hundreds or thousands. That number is your available memory in megabytes.

What does this query do?

Node Exporter reports memory in bytes. Dividing by 1024 twice converts bytes to kilobytes, then kilobytes to megabytes. PromQL supports basic arithmetic on metrics, so you can transform values inline without any external tools.

Now combine everything you've learned into the most useful query: CPU usage as a percentage. Before you run it, think about this: if you know the idle CPU rate, how would you calculate the used percentage?

Clear the expression input field and paste this query, then click Execute:

100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

Click the Graph tab to see your CPU usage percentage over time.

You should see a line chart showing your CPU usage as a percentage. The value will likely hover in the low single digits unless your machine is under load.

How does this query work?

This query works from the inside out. First, rate(node_cpu_seconds_total{mode="idle"}[5m]) calculates how much idle time each CPU core is spending. Then avg by(instance) averages across all CPU cores for the same machine.

Multiplying by 100 converts the decimal to a percentage. Subtracting from 100 flips "idle" into "used." If your CPU is 95% idle, it's 5% used.

Seeing "no data" or an empty graph?

Make sure you copied the entire query including all parentheses. A missing closing bracket will cause a parse error.

If the graph is flat at zero, your stack may not have been running long enough to accumulate 5 minutes of data. Wait a few minutes and try again.

Still stuck? Help me troubleshoot my PromQL CPU usage query in Prometheus.

Discover the Prometheus UI limitation

You now have three useful queries: CPU usage, memory available, and the raw idle rate. In a real monitoring setup, you would want to see all three side by side on a single screen. Try to do that now.

With the CPU usage percentage graph still displayed, try running the memory query from earlier:

node_memory_MemAvailable_bytes / 1024 / 1024

The CPU graph disappears. The expression browser replaces your previous query result with the new one. You can only view one query at a time.

Try switching between the Table and Graph tabs to see the same data in different formats.
Try to save this view or bookmark the current query. Notice there is no save button or dashboard feature.

This is the designed limitation. The Prometheus expression browser is built for ad-hoc queries and debugging. It cannot combine multiple metrics into a single view, save your queries, or format panels with custom layouts.

Why not just use the Prometheus UI for monitoring?

The Prometheus expression browser is great for exploring data and debugging queries. But a real monitoring setup needs a persistent dashboard where you can see CPU, memory, and disk usage at a glance without re-typing queries.

Prometheus was designed to do one thing well: collect and store metrics. The visualization layer is a separate concern. That's exactly the problem Grafana solves.

You've written real PromQL queries that pull live data from your machine and transform it into useful metrics. You know how rate() works, why counters need transformation, and where the Prometheus UI falls short. Next up, you'll solve that limitation by adding Grafana and building a dashboard that displays all your metrics side by side.

Build a Grafana Dashboard

You've seen the Prometheus UI in action. It can run individual queries and display results, but you can only view one query at a time. There is no way to combine multiple metrics into a single view, customize the display, or save your work.

Grafana is the visualization layer that solves this. It connects to Prometheus as a data source, lets you build dashboards with multiple panels, and saves everything so you can come back later. In this step, you'll add Grafana to your stack and build a dashboard with live CPU, memory, and disk panels.

In this step, get ready to:

Add Grafana to your Docker Compose stack with an auto-provisioned Prometheus datasource.
Log in to Grafana and verify the Prometheus connection.
Build a dashboard with three panels showing CPU, memory, and filesystem metrics.

Create the Grafana provisioning file

Grafana supports auto-provisioning. Instead of manually adding a data source through the UI every time, you write a YAML file that configures it on startup.

Create the provisioning directory structure by running these commands:

mkdir -p grafana/provisioning/datasources

Confirm the directories were created by running:

ls grafana/provisioning

You should see datasources listed in the output.

Create the datasource provisioning file by adding this code to grafana/provisioning/datasources/prometheus.yml:

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: true

What does this file do?

This YAML file tells Grafana to automatically create a Prometheus data source on startup. The url uses http://prometheus:9090 because both containers are on the same Docker network, so Grafana can reach Prometheus by its service name.

Setting isDefault: true means every new panel will use this data source automatically. Setting editable: true lets you modify it from the Grafana UI if needed.

Save grafana/provisioning/datasources/prometheus.yml.

Add Grafana to Docker Compose

Now you need to tell Docker Compose about Grafana so it runs alongside Prometheus and Node Exporter.

In docker-compose.yml, find the node-exporter service block. Add the following grafana service below it, before the volumes: section:

  grafana:
    image: grafana/grafana:13.2.0
    container_name: grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - grafana-data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
    restart: unless-stopped
    networks:
      - monitoring

What does this service do?

The grafana service runs Grafana on port 3000. The GF_SECURITY_ADMIN_PASSWORD environment variable pre-sets the admin password to admin, so Grafana skips the forced password change on first login.

Two volumes are mounted. grafana-data persists dashboards and settings between restarts. The provisioning directory mounts your datasource config file so Grafana auto-configures Prometheus on startup.

Also add grafana-data: to the volumes: section at the bottom of the file, so it looks like this:

volumes:
  prometheus-data:
  grafana-data:

Save docker-compose.yml.
Bring up the updated stack by running:

docker compose up -d

Verify all three containers are running by running:

docker compose ps

You should see three containers listed: prometheus, node-exporter, and grafana. All three should show a status of running.

Grafana container not starting?

Check that the grafana service is indented at the same level as prometheus and node-exporter in docker-compose.yml. YAML is sensitive to indentation.
Make sure grafana-data: is listed under the volumes: section at the bottom of the file. A missing volume definition causes a startup error.
Run docker compose logs grafana to see the container's log output for more details.

Still stuck? Help me troubleshoot why my Grafana container isn't starting in Docker Compose.

✔️ Awesome, I've got everything!

Great. Double check you've saved both files.

ⓧ I'd like to double check the full code

Here is the full grafana/provisioning/datasources/prometheus.yml:

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: true

Here is the full docker-compose.yml:

services:
  prometheus:
    image: prom/prometheus:v3.14.0
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.retention.time=7d'
      - '--web.enable-lifecycle'
    restart: unless-stopped
    networks:
      - monitoring

  node-exporter:
    image: prom/node-exporter:v1.12.1
    container_name: node-exporter
    ports:
      - "9100:9100"
    restart: unless-stopped
    networks:
      - monitoring

  grafana:
    image: grafana/grafana:13.2.0
    container_name: grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - grafana-data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
    restart: unless-stopped
    networks:
      - monitoring

volumes:
  prometheus-data:
  grafana-data:

networks:
  monitoring:
    driver: bridge

Log in to Grafana and verify the datasource

Grafana is now running. Your provisioning file should have auto-configured Prometheus as a data source. Let's confirm the connection works.

Open http://localhost:3000 in your browser.
Log in with username admin and password admin.
In the left-side menu, click Connections.
Click Data sources.
Confirm that Prometheus appears in the list with the URL http://prometheus:9090.

Why auto-provisioning?

In production, teams manage Grafana configuration as code. Provisioning files mean you can version-control your datasource setup, deploy it to any environment, and never rely on someone manually clicking through a UI. This is the same pattern used in real DevOps workflows.

Prometheus not showing in Data sources?

Check that the file path is exactly grafana/provisioning/datasources/prometheus.yml. A misnamed folder or file means Grafana won't find the provisioning config.
Check the volume mount in docker-compose.yml. It should be ./grafana/provisioning:/etc/grafana/provisioning:ro.
Try restarting the Grafana container by running docker compose restart grafana and refreshing the page.

Still stuck? Help me figure out why my provisioned Prometheus datasource isn't appearing in Grafana.

Create a dashboard with CPU, memory, and filesystem panels

Now for the payoff. You'll create a dashboard with three panels that show your machine's live metrics side by side. Each panel uses a PromQL query you already explored in the Prometheus UI.

In the left-side menu, click Dashboards.
Click New and select New Dashboard.
Click the blue Add new element icon.
Click or drag a panel onto the dashboard.
On the new panel, click Configure visualization.

The panel editor opens with the Prometheus data source pre-selected. You'll see a query area at the bottom of the screen.

In the query editor, look for the mode toggle and switch to Code mode. This lets you paste raw PromQL directly.
Clear any existing query text and paste in this CPU usage query:

100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

Click Refresh (or press Shift+Enter) to run the query.

You should see a time-series graph appear showing your CPU usage as a percentage.

In the panel title field at the top of the editor, enter CPU Usage (%).
Click Save in the top-right corner.
When prompted, enter a dashboard title such as System Monitoring and click Save.

Your first panel is live. You should see a graph of CPU usage updating in real time.

Seeing "No data" in the panel?

Make sure you switched to Code mode in the query editor. Builder mode may not accept raw PromQL correctly.
Check that the Prometheus data source is selected in the query editor. If another data source is shown, select Prometheus from the data source dropdown.
Verify that the node-exporter container is still running by checking docker compose ps in your terminal.

Still stuck? Help me troubleshoot why my Grafana panel shows no data for a Prometheus PromQL query.

Now add the second panel for memory usage.

Click Back to dashboard to return to the dashboard view.
Click the blue Add new element icon again and click or drag a new panel onto the dashboard.
Click Configure visualization on the new panel.
Switch to Code mode in the query editor and paste this memory usage query:

(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100

Click Refresh to run the query.
Enter Memory Usage (%) as the panel title.
Click Save.

You should see a second graph appear showing memory usage as a percentage.

Add the third and final panel for filesystem usage.

Click Back to dashboard again.
Click the blue Add new element icon, click or drag a new panel, and click Configure visualization.
Switch to Code mode and paste this filesystem usage query:

(1 - node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100

Click Refresh to run the query.
Enter Filesystem Usage (%) as the panel title.
Click Save.
Click Back to dashboard and then click Exit edit to leave edit mode.

Before you look at the result, what do you expect to see? You've added three panels, each with a different PromQL query. Think about what the dashboard should look like.

You should now see all three panels side by side: CPU usage, memory usage, and filesystem usage, each displaying a live time-series graph that updates every 15 seconds. This is your first real monitoring dashboard.

Why Grafana over the Prometheus UI?

Compare what you're looking at now to what you had in the Prometheus UI. There, you could view one query at a time, with no way to save your work. Grafana lets you combine all your key metrics into a single view, customize how they look, and persist everything to disk.

This is the same pattern production teams use. Dashboards become the first place engineers look during an incident, because they show the health of the system at a glance.

Your monitoring stack now has eyes. Prometheus collects the data, and Grafana turns it into a dashboard you can actually watch. Next up, you'll give it a voice by configuring alerts that fire when something goes wrong.

Configure Alerting with Alertmanager

Your Grafana dashboard gives you a live view of CPU, memory, and disk usage. But a dashboard only works when someone is watching it.

In production, SRE teams solve this with alerting rules and alert routing. Prometheus evaluates rules on every scrape cycle and fires alerts when conditions are met. Alertmanager receives those alerts, groups them, and routes them to the right people. In this step, you will write alert rules, deploy Alertmanager, and trigger a real alert to prove the full pipeline works.

In this step, get ready to:

Write alert rules that detect high CPU, high memory, and downed instances.
Deploy Alertmanager and connect it to Prometheus.
Trigger and verify a real alert end-to-end.

Create the alert rules file

Before Alertmanager can route anything, Prometheus needs rules that define what to alert on. Each rule is a PromQL expression paired with a threshold and a duration.

Create a new file called alert-rules.yml inside the prometheus folder, and paste in the following code:

groups:
  - name: node-alerts
    rules:
      - alert: HighCpuUsage
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: "CPU usage is above 80% for more than 2 minutes (current value: {{ $value }}%)"

What does this code do?

This defines a rule group called node-alerts with one alert rule so far.

HighCpuUsage reuses the CPU percentage query from your dashboard. It fires when CPU usage stays above 80% for 2 continuous minutes.
for: 2m prevents a short spike from triggering an alert. The condition must hold for the entire duration before the alert fires.
annotations use Go template syntax ({{ $labels.instance }}) to include the affected machine in the alert message.

Now add the next two alert rules directly below the HighCpuUsage rule. Make sure they are at the same indentation level (aligned with the first - alert: line):

      - alert: HighMemoryUsage
        expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 85
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage on {{ $labels.instance }}"
          description: "Memory usage is above 85% for more than 2 minutes (current value: {{ $value }}%)"

      - alert: InstanceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Instance {{ $labels.instance }} is down"
          description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 1 minute."

What do these two rules do?

HighMemoryUsage fires when memory usage exceeds 85% for 2 minutes. It uses the same memory formula from your Grafana panel.
InstanceDown fires when any scrape target stops responding (up == 0) for just 1 minute. It has a critical severity because a downed instance needs immediate attention.

Save prometheus/alert-rules.yml.

✔️ Awesome, I've got everything!

Great. Make sure you have saved the file before moving on.

ⓧ I'd like to double check the full code

Here is the complete prometheus/alert-rules.yml file. Make sure yours matches exactly:

groups:
  - name: node-alerts
    rules:
      - alert: HighCpuUsage
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: "CPU usage is above 80% for more than 2 minutes (current value: {{ $value }}%)"

      - alert: HighMemoryUsage
        expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 85
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage on {{ $labels.instance }}"
          description: "Memory usage is above 85% for more than 2 minutes (current value: {{ $value }}%)"

      - alert: InstanceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Instance {{ $labels.instance }} is down"
          description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 1 minute."

Create the Alertmanager configuration

Alertmanager needs its own configuration file to know how to group and route incoming alerts.

Create a new folder called alertmanager inside your monitoring-stack directory by running this command:

mkdir alertmanager

Create a new file called alertmanager.yml inside the alertmanager folder, and paste in the following code:

global:
  resolve_timeout: 5m

route:
  receiver: 'default'
  group_by: ['alertname', 'instance']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h

receivers:
  - name: 'default'

What does this configuration do?

resolve_timeout: 5m tells Alertmanager to mark an alert as resolved if Prometheus stops sending it for 5 minutes.
group_by: ['alertname', 'instance'] batches related alerts together. If three instances go down at once, you get one grouped notification instead of three separate ones.
The default receiver has no notification integrations configured. Alerts will appear in the Alertmanager UI without being sent anywhere. In production, you would add Slack, PagerDuty, or email integrations here.

Save alertmanager/alertmanager.yml.
Confirm both new files exist by running:

ls prometheus/alert-rules.yml alertmanager/alertmanager.yml

You should see both file paths listed in the output with no errors.

Seeing 'No such file or directory'?

Make sure you are running the command from inside the monitoring-stack directory.
Check that the file name is exactly alert-rules.yml (with a hyphen, not an underscore) inside the prometheus folder.
Check that the alertmanager folder exists and contains alertmanager.yml.

Why aren't my alert config files showing up?

Add Alertmanager to the stack and wire it to Prometheus

The alert rules and Alertmanager config files exist on disk, but nothing is using them yet. You need to tell Prometheus where to find the rules and where to send fired alerts, then add Alertmanager as a new service in Docker Compose.

Open prometheus/prometheus.yml and add the following two sections between the global: block and the scrape_configs: block:

# Tell Prometheus where to send fired alerts
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - 'alertmanager:9093'

# Load the alert rules file
rule_files:
  - 'alert-rules.yml'

What do these sections do?

alerting: tells Prometheus the network address of the Alertmanager instance. When a rule fires, Prometheus sends the alert here.
rule_files: tells Prometheus which YAML files contain alert rules. Prometheus re-evaluates every rule on each evaluation_interval (15 seconds in your config).

Save prometheus/prometheus.yml.

✔️ Awesome, I've got everything!

Great. Make sure you have saved the file before moving on.

ⓧ I'd like to double check the full code

Here is the complete prometheus/prometheus.yml file. Make sure yours matches exactly:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - 'alertmanager:9093'

rule_files:
  - 'alert-rules.yml'

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']

Now you need to update docker-compose.yml with two changes: mount the new alert rules file into the Prometheus container, and add an Alertmanager service.

In docker-compose.yml, find the volumes: section under the prometheus service. Find this line:

      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus

Add a new volume line between these two lines, so it looks like this:

      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./prometheus/alert-rules.yml:/etc/prometheus/alert-rules.yml:ro
      - prometheus-data:/prometheus

Next, scroll down to the end of the grafana service block. Add the following Alertmanager service below it, before the volumes: section at the bottom of the file:

  # Routes alerts from Prometheus
  alertmanager:
    image: prom/alertmanager:v0.34.0
    container_name: alertmanager
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
    command:
      - '--config.file=/etc/alertmanager/alertmanager.yml'
    restart: unless-stopped
    networks:
      - monitoring

What does this service do?

This adds Alertmanager as the fourth container in your stack. It listens on port 9093, mounts the configuration file you just created, and joins the same monitoring network so Prometheus can reach it by container name.

Save docker-compose.yml.

✔️ Awesome, I've got everything!

Great. Make sure you have saved the file before continuing.

ⓧ I'd like to double check the full code

Your docker-compose.yml is now over 30 lines, so confirm each section is present by scrolling through the file. You should have, in order:

The prometheus service with three volume mounts (including the new alert-rules.yml line).
The node-exporter service.
The grafana service.
The new alertmanager service with image prom/alertmanager:v0.34.0 and port 9093.
The volumes: and networks: sections at the bottom.

Restart the stack with the new configuration by running:

docker compose up -d

Docker Compose detects the changed and new services. It recreates the Prometheus container (new volume mount) and creates the new Alertmanager container.

Confirm all four containers are running by running:

docker compose ps

You should see four services listed: prometheus, node-exporter, grafana, and alertmanager, all with a State of Up.

Alertmanager container not starting?

Run docker compose logs alertmanager to see the error output. A YAML syntax error in alertmanager/alertmanager.yml is the most common cause.
Make sure the alertmanager service in docker-compose.yml is indented at the same level as grafana (two spaces under services:).
If Prometheus is restarting, check docker compose logs prometheus. A missing or misnamed alert-rules.yml volume mount causes Prometheus to fail on startup.

My Alertmanager or Prometheus container won't start after updating docker-compose.yml. Help me debug.

Trigger and verify a real alert

Your alert rules are loaded and Alertmanager is running. Time to prove the pipeline works by deliberately causing a failure and watching the alert fire.

Open http://localhost:9090/alerts in your browser.

You should see all three alert rules listed: HighCpuUsage, HighMemoryUsage, and InstanceDown. All three should be in an inactive (green) state, meaning no conditions are currently met.

Don't see any alert rules?

Check that prometheus/prometheus.yml has the rule_files: section with 'alert-rules.yml' listed.
Check that the alert-rules.yml volume mount exists in the prometheus service in docker-compose.yml.
Run docker compose restart prometheus after fixing any issues, then refresh the alerts page.

My Prometheus alerts page is empty. Help me figure out why.

Now for the real test. You are going to stop the Node Exporter container so Prometheus can no longer scrape it. This triggers the InstanceDown alert.

Before you run the next command, think about what you expect to happen. The InstanceDown rule has for: 1m. Will the alert fire immediately, or will there be a delay?

Stop the Node Exporter container by running:

docker compose stop node-exporter

Wait about 30 seconds, then refresh http://localhost:9090/alerts in your browser.

You should see InstanceDown change from inactive to PENDING (yellow). The alert is not firing yet because the for: 1m duration has not elapsed. Prometheus detected the problem but is waiting to confirm it is not a brief blip.

Wait another 30-60 seconds and refresh the page again.

The InstanceDown alert should now show FIRING (red). Prometheus has confirmed the target is down for over 1 minute and sent the alert to Alertmanager.

Open http://localhost:9093 in your browser to see the Alertmanager UI.

You should see the InstanceDown alert listed here too. This confirms the full pipeline is working: Prometheus detected the problem, evaluated the rule, and sent the alert to Alertmanager for routing.

Why two places to see alerts?

The Prometheus Alerts page at /alerts shows rule evaluation state (inactive, pending, firing). Alertmanager at port 9093 shows alerts that have been received and are being routed.

In production, Alertmanager is where teams manage silences, see grouping, and configure notification channels. Prometheus is where you debug whether a rule is actually evaluating as expected.

Restart Node Exporter to resolve the alert by running:

docker compose start node-exporter

Wait about a minute and refresh http://localhost:9090/alerts.

The InstanceDown alert should return to inactive (green). You just triggered and resolved a real alert through the full observability pipeline.

Alert still showing as FIRING after restarting?

Run docker compose ps to confirm node-exporter is back up and running.
The alert takes up to a minute to resolve after the target comes back. Give it another 60 seconds and refresh again.
Check the Status > Targets page in Prometheus to confirm node-exporter shows as UP again.

My InstanceDown alert won't resolve after restarting node-exporter. Help me debug.

Your monitoring stack is now complete. Prometheus scrapes metrics, Grafana visualizes them, and Alertmanager catches problems automatically. Next up, a Secret Mission that takes your alerting to the next level with predictive alerts.

Secret mission

Predict the Future with PromQL

Your current alerts only fire when something is already broken. In this mission, you will write a predictive alert using PromQL's predict_linear function that forecasts disk space exhaustion before it happens. This is a technique used by experienced SRE teams to prevent incidents instead of reacting to them.

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally, so there are no ongoing costs.

Resources you used:

Four Docker containers: prometheus, node-exporter, grafana, and alertmanager.
Two Docker volumes: prometheus-data and grafana-data storing persistent metric and dashboard data.
The monitoring-stack project directory containing all configuration files.

Keep everything running

No action needed. Choose this if you're still actively building or want to keep experimenting with your monitoring stack.

Your four containers continue running and collecting metrics.
Open http://localhost:3000 to access your Grafana dashboard at any time.
Open http://localhost:9090 to run PromQL queries against your metrics.
Open http://localhost:9093 to view the Alertmanager UI.

Pause - I'll come back to this later

Stop all containers to free up system resources. Your configuration files and volume data remain intact so you can resume later.

Navigate to the monitoring-stack directory and stop all containers by running:

docker compose stop

When you're ready to resume, start the containers again from the same directory by running:

docker compose start

All four services pick up exactly where they left off, with your dashboards and alert rules intact.

Delete - I don't want to use this again

Remove all project containers, volumes, and configuration files to start fresh.

Navigate to the monitoring-stack directory and remove all containers and volumes by running:

docker compose down -v

What does this command do?

The docker compose down -v command stops and removes all four containers, their networks, and the two named volumes (prometheus-data and grafana-data). All stored metrics and dashboard data are deleted permanently.

Move up one directory by running:

cd ..

Delete the monitoring-stack folder to remove all configuration files.

macOS

rm -rf monitoring-stack

Windows

rmdir /s /q monitoring-stack

Nice Work!

Nice work! You just built a production-style monitoring stack that collects, visualizes, and alerts on real system metrics from your own machine.

You've learned how to:

Deploy a full observability pipeline with Docker Compose, launching Prometheus, Node Exporter, Grafana, and Alertmanager as a connected four-service stack with persistent storage, shared networking, and auto-provisioned configuration.
Write PromQL queries that transform raw counter data into meaningful percentage-based metrics and display them on a custom Grafana dashboard with live CPU, memory, and filesystem panels.
Configure alert rules with thresholds and for-durations to avoid false positives, then trigger and resolve a real InstanceDown alert end-to-end, from detection in Prometheus through routing in Alertmanager.
Secret Mission: Use predict_linear to build a predictive alert that forecasts disk exhaustion before it happens. This is a key SRE technique for proactive incident prevention.

Ready to quiz yourself?