Deploy a Server Monitoring Stack
Run Prometheus, Grafana, and exporters locally with Docker Compose.
Introduction
30 Second Summary
When a website goes down or slows to a crawl, the first question is always "what changed?" Production teams answer that question with monitoring dashboards that show exactly what every server is doing, in real time.
In this project, you will deploy a complete server monitoring stack using Docker Compose. You'll configure Prometheus to collect metrics, connect Grafana to visualize them, and build a dashboard that tracks CPU, memory, and network activity.
What You'll Build
A containerized monitoring stack with Prometheus, Grafana, Node Exporter, and cAdvisor, all orchestrated by a single Docker Compose file running locally in WSL.
By the end of this project, you'll have:
- A docker-compose.yml that orchestrates four interconnected monitoring services with pinned image versions and persistent storage.
- A configured Prometheus instance that scrapes host and container metrics every 15 seconds.
- A live Grafana dashboard displaying real-time CPU, memory, disk, and network metrics using PromQL queries.
- Secret Mission: Add Alertmanager with a custom alerting rule that fires when CPU usage exceeds 80%.
Are there any prerequisites?
This project assumes you have WSL and Docker Desktop installed with WSL integration enabled. You should be comfortable pulling images, running containers, and working with volumes and networks in Docker.
Set Up the Project Structure
Before you can deploy any containers, you need a clear set of configuration files that define your monitoring stack. These files are the infrastructure-as-code that tells Docker Compose exactly what to run and how to wire it together.
In this step, you will create the project directory and all the configuration files your stack needs. By the end, you will have a complete file tree ready for the service definitions and scraping rules you will write in the next steps.
In this step, get ready to:
- Create the project directory tree with all required files.
- Write the Grafana datasource provisioning file.
- Understand the monitoring architecture and data flow.
Create the project directory tree
Your monitoring stack needs four files organized into directories that separate each service's configuration. Creating this structure first gives you a clear map of the entire project.
- In your WSL terminal, move to your Desktop by running this command:
cd ~/Desktop
- Create the project directories by running these commands:
mkdir -p monitoring-stack/prometheus
mkdir -p monitoring-stack/grafana/provisioning/datasources
What do these commands do?
The -p flag creates parent directories as needed. This means mkdir -p monitoring-stack/grafana/provisioning/datasources creates all four nested folders in one command.
- Create the empty configuration files by running these commands:
touch monitoring-stack/docker-compose.yml
touch monitoring-stack/prometheus/prometheus.yml
touch monitoring-stack/grafana/provisioning/datasources/datasource.yml
- Confirm the files exist by running:
find monitoring-stack -type f
- You should see all four files listed in the output:
monitoring-stack/docker-compose.yml
monitoring-stack/prometheus/prometheus.yml
monitoring-stack/grafana/provisioning/datasources/datasource.yml
Not seeing all four files?
Make sure you ran the mkdir -p commands before the touch commands. The touch command cannot create a file if its parent directory does not exist.
help me troubleshoot my directory structure
Write the Grafana datasource provisioning file
Grafana can automatically configure data sources on startup using provisioning files. This means you will not need to manually click through the UI to connect Prometheus as a data source later.
- Write the datasource provisioning file by running this command:
cat > monitoring-stack/grafana/provisioning/datasources/datasource.yml << 'EOF'
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: true
editable: false
EOF
What does this configuration do?
This YAML file tells Grafana to create a Prometheus data source automatically when the container starts.
- access: proxy means Grafana's backend makes the HTTP request to Prometheus, not your browser. This works because both containers share a Docker network.
- url: http://prometheus:9090 uses the container name as the hostname. Inside a Docker Compose network, services reach each other by name.
- isDefault: true makes this the default data source for new dashboard panels.
- Verify the file was written correctly by running:
cat monitoring-stack/grafana/provisioning/datasources/datasource.yml
- You should see the full YAML content with http://prometheus:9090 as the URL.
Seeing an empty file or wrong content?
If the file is empty, the heredoc command may not have completed. Make sure you copied the entire block including the final EOF on its own line with no leading spaces.
help me fix my datasource provisioning file
✔️ Awesome, I've got everything!
Great. Double check you see the correct YAML output from the cat command, then move on.
ⓧ I'd like to double check the full code
Here is the complete content of datasource.yml for comparison:
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: true
editable: false
Understand the architecture
Before writing any service definitions, it helps to see how data flows through your monitoring stack. Each component has a single job, and they connect in a one-directional chain.
How does the monitoring data flow?
Node Exporter and cAdvisor expose metrics at HTTP endpoints. Node Exporter reports host-level Linux metrics (CPU, memory, disk, network). cAdvisor reports per-container resource usage.
Prometheus scrapes those endpoints every 15 seconds and stores the data as time-series. It is a pull-based system, meaning Prometheus reaches out to exporters rather than exporters pushing data to it.
Grafana queries Prometheus to render dashboards. Each component runs as its own container on a shared Docker network, which is why your datasource provisioning file uses the container name prometheus as the hostname.
At this point you have 4 files in your project directory and no containers running yet. In the next step, you will fill in docker-compose.yml to define all four services.
Write the Docker Compose Stack
Your project structure is in place with all the files ready to fill in.
Now you need to define what actually runs inside those files. A single docker-compose.yml will declare four services, wire them together on a shared network, and attach persistent storage. This is infrastructure-as-code: your entire monitoring stack described in one file that anyone can deploy with a single command.
In this step, get ready to:
- Define the Prometheus service with config-loading, retention, and lifecycle flags.
- Define Node Exporter and cAdvisor to expose host and container metrics.
- Define the Grafana service with auto-provisioned datasource and shared infrastructure.
Define the Prometheus service
Prometheus is the core of your monitoring stack. It pulls (scrapes) metrics from other services on a schedule, stores them as time-series data, and serves queries. Your service definition needs to load a config file, set a data retention period, and enable hot-reloading so you can update the config without restarting the container.
- Open docker-compose.yml in your editor and add the Prometheus service definition:
services:
prometheus:
image: prom/prometheus:v3.12.0
container_name: prometheus
restart: unless-stopped
ports:
- "9090:9090"
volumes:
- ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
- prometheus-data:/prometheus
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.path=/prometheus"
- "--storage.tsdb.retention.time=15d"
- "--web.enable-lifecycle"
networks:
- monitoring
depends_on:
- node-exporter
- cadvisor
What does this code do?
This defines the Prometheus container with a pinned image version (prom/prometheus:v3.12.0). The command section passes flags that control Prometheus behavior at startup.
- --config.file tells Prometheus where to find its scrape configuration inside the container.
- --storage.tsdb.retention.time=15d keeps 15 days of metrics before automatically deleting old data.
- --web.enable-lifecycle exposes an endpoint that lets you reload the config without restarting the container.
- The volumes mount your local config file as read-only (:ro) and persist scraped data in a named volume so it survives container restarts.
- Save docker-compose.yml.
- You won't be able to validate the file yet. The networks and volumes sections referenced here need to be defined at the bottom of the file first. You will add those in the final substep.
Define Node Exporter and cAdvisor services
Node Exporter exposes host-level Linux metrics (CPU, memory, disk, network). cAdvisor exposes per-container resource usage. Together they give Prometheus visibility into both the host machine and individual Docker containers.
- Below the Prometheus service block in docker-compose.yml, add the Node Exporter service:
node-exporter:
image: prom/node-exporter:v1.11.1
container_name: node-exporter
restart: unless-stopped
ports:
- "9100:9100"
volumes:
- /proc:/host/proc:ro
- /sys:/host/sys:ro
- /:/rootfs:ro
command:
- "--path.procfs=/host/proc"
- "--path.sysfs=/host/sys"
- "--path.rootfs=/rootfs"
- "--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)"
networks:
- monitoring
Why does Node Exporter mount host paths?
A container is isolated from the host by default. To report host-level metrics, Node Exporter needs read-only access to the host's /proc, /sys, and root filesystem. The command flags tell Node Exporter where to find these inside the container.
The $$ in the exclude pattern is Docker Compose's escape for a literal $ character. Without it, Compose would try to interpret $| as a variable reference.
- Directly below the Node Exporter block, add the cAdvisor service:
cadvisor:
image: ghcr.io/google/cadvisor:v0.57.0
container_name: cadvisor
restart: unless-stopped
ports:
- "8080:8080"
volumes:
- /:/rootfs:ro
- /var/run:/var/run:ro
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
- /dev/disk/:/dev/disk:ro
privileged: true
devices:
- /dev/kmsg:/dev/kmsg
networks:
- monitoring
Why does cAdvisor need privileged mode?
cAdvisor needs deep access to the Docker runtime and kernel metrics. The privileged: true flag grants the container extended Linux capabilities. The /dev/kmsg device gives access to kernel log messages for additional system-level data.
Notice the image uses ghcr.io/google/cadvisor instead of the older gcr.io registry. As of v0.53.0, cAdvisor images moved to GitHub Container Registry.
- Save docker-compose.yml.
- You won't see a change yet. The next substep adds the Grafana service and the shared network and volume definitions that all services reference.
Define the Grafana service and shared infrastructure
Grafana is your visualization layer. It queries Prometheus and renders dashboards. You also need to define the shared bridge network and named volumes that every service references.
- Below the cAdvisor block in docker-compose.yml, add the Grafana service followed by the network and volume definitions:
grafana:
image: grafana/grafana-oss:13.1.0
container_name: grafana
restart: unless-stopped
ports:
- "3000:3000"
volumes:
- grafana-data:/var/lib/grafana
- ./grafana/provisioning:/etc/grafana/provisioning:ro
environment:
GF_SECURITY_ADMIN_USER: admin
GF_SECURITY_ADMIN_PASSWORD: admin
GF_USERS_ALLOW_SIGN_UP: "false"
networks:
- monitoring
depends_on:
- prometheus
networks:
monitoring:
driver: bridge
volumes:
prometheus-data:
grafana-data:
What does this code do?
- The environment block sets the admin credentials and disables public sign-up. These are Grafana's built-in configuration variables.
- The provisioning volume mount (./grafana/provisioning:/etc/grafana/provisioning:ro) loads the datasource YAML you created in Step 1. Grafana reads it on startup and auto-connects to Prometheus.
- The networks section at the bottom creates a bridge network called monitoring. All services join this network so they can reach each other by container name.
- The volumes section declares named volumes for Prometheus data and Grafana data. Named volumes persist data even when containers are removed and recreated.
- Save docker-compose.yml.
- Validate your complete Compose file by running this command from the monitoring-stack directory:
docker compose config
You should see the fully parsed YAML output printed to your terminal with all services, networks, and volumes expanded. No error messages should appear.
Seeing a YAML error?
- Check your indentation. YAML uses spaces (not tabs), and each nested level is exactly 2 spaces deeper.
- Make sure there is only one services: key at the top. Each service definition must be indented under it.
- Verify that port mappings use quotes around the values (e.g. "9090:9090").
Help me debug a YAML parsing error in my docker-compose.yml
✔️ Awesome, I've got everything!
Great. Double check you have saved docker-compose.yml and that docker compose config shows all four services with no errors.
ⓧ I'd like to double check the full code
The order of services within the file does not affect functionality. Docker Compose uses depends_on (not file position) to determine startup order. Here is the canonical version:
services:
prometheus:
image: prom/prometheus:v3.12.0
container_name: prometheus
restart: unless-stopped
ports:
- "9090:9090"
volumes:
- ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
- prometheus-data:/prometheus
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.path=/prometheus"
- "--storage.tsdb.retention.time=15d"
- "--web.enable-lifecycle"
networks:
- monitoring
depends_on:
- node-exporter
- cadvisor
grafana:
image: grafana/grafana-oss:13.1.0
container_name: grafana
restart: unless-stopped
ports:
- "3000:3000"
volumes:
- grafana-data:/var/lib/grafana
- ./grafana/provisioning:/etc/grafana/provisioning:ro
environment:
GF_SECURITY_ADMIN_USER: admin
GF_SECURITY_ADMIN_PASSWORD: admin
GF_USERS_ALLOW_SIGN_UP: "false"
networks:
- monitoring
depends_on:
- prometheus
node-exporter:
image: prom/node-exporter:v1.11.1
container_name: node-exporter
restart: unless-stopped
ports:
- "9100:9100"
volumes:
- /proc:/host/proc:ro
- /sys:/host/sys:ro
- /:/rootfs:ro
command:
- "--path.procfs=/host/proc"
- "--path.sysfs=/host/sys"
- "--path.rootfs=/rootfs"
- "--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)"
networks:
- monitoring
cadvisor:
image: ghcr.io/google/cadvisor:v0.57.0
container_name: cadvisor
restart: unless-stopped
ports:
- "8080:8080"
volumes:
- /:/rootfs:ro
- /var/run:/var/run:ro
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
- /dev/disk/:/dev/disk:ro
privileged: true
devices:
- /dev/kmsg:/dev/kmsg
networks:
- monitoring
networks:
monitoring:
driver: bridge
volumes:
prometheus-data:
grafana-data:
Your entire monitoring stack is now defined as code in a single file. Next up, you will write the Prometheus configuration that tells it exactly which targets to scrape and how often.
Configure Prometheus Scraping
Your Docker Compose file defines the containers, but Prometheus still needs to know what to scrape.
Without a configuration file telling it where to find metrics endpoints, Prometheus starts with no targets and collects nothing. In this step, you will write the Prometheus configuration that defines how often to scrape and where to find each target.
In this step, get ready to:
- Write the global scrape and evaluation intervals.
- Define three scrape jobs targeting Prometheus, Node Exporter, and cAdvisor.
- Validate the configuration with Prometheus's built-in tool.
Write the global configuration
The global section sets default behavior that applies to all scrape jobs. Two values matter here: how often Prometheus pulls metrics from targets, and how often it evaluates alerting rules.
- Open prometheus/prometheus.yml in your editor and add the global configuration:
global:
scrape_interval: 15s # How often Prometheus pulls metrics from targets
evaluation_interval: 15s # How often alerting rules are evaluated
What does this configure?
- scrape_interval: 15s means Prometheus sends an HTTP request to every target every 15 seconds to collect new metric values.
- evaluation_interval: 15s means Prometheus checks alerting and recording rules every 15 seconds (you will use this in the Secret Mission).
You won't see a change yet. The next chunk adds the scrape targets that complete this file.
Define scrape jobs
Each scrape job tells Prometheus where to find a specific metrics endpoint. You need three jobs: one for Prometheus itself, one for Node Exporter, and one for cAdvisor.
- Below the global block (leave one blank line), add the three scrape jobs:
scrape_configs:
# Prometheus monitors its own health
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
# Host-level Linux metrics (CPU, memory, disk, network)
- job_name: "node-exporter"
static_configs:
- targets: ["node-exporter:9100"]
# Per-container resource usage metrics
- job_name: "cadvisor"
static_configs:
- targets: ["cadvisor:8080"]
Why container names instead of localhost?
Inside a Docker Compose network, services reference each other by container name. The node-exporter and cadvisor targets use their service names because Prometheus reaches them over the shared monitoring network you defined in docker-compose.yml.
The prometheus job uses localhost:9090 because Prometheus scrapes itself from within its own container.
- Save prometheus/prometheus.yml.
✔️ Awesome, I've got everything!
Great. Double check you have saved the file before moving on to validation.
ⓧ I'd like to double check the full code
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
- job_name: "node-exporter"
static_configs:
- targets: ["node-exporter:9100"]
- job_name: "cadvisor"
static_configs:
- targets: ["cadvisor:8080"]
Verify the configuration syntax
Prometheus ships with a built-in validation tool called promtool. You can run it inside a Docker container without installing anything extra on your machine.
- Validate your configuration by running this command from the monitoring-stack directory:
docker run --rm -v ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus:v3.12.0 promtool check config /etc/prometheus/prometheus.yml
What does this command do?
- docker run --rm creates a temporary container that deletes itself after the command finishes.
- -v ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml mounts your local config file into the container at the path Prometheus expects.
- promtool check config is the built-in validator that parses the file and reports any errors.
- You should see output containing SUCCESS confirming your configuration is valid.
Seeing FAILED or a syntax error?
- Check that you are running the command from inside the monitoring-stack directory. The -v ./prometheus/prometheus.yml path is relative to your current location.
- YAML is indentation-sensitive. Make sure you are using spaces (not tabs) and that each level is indented by exactly 2 spaces.
- Verify that the targets values are enclosed in both square brackets and double quotes, like ["localhost:9090"].
Still stuck? Why is promtool failing on my prometheus.yml?
Your Prometheus configuration is validated and all project files are complete. Next up, you will bring the entire stack online and confirm all services are healthy and communicating.
Deploy and Verify the Stack
Your configuration is validated and ready to go. Now it is time to bring the entire stack to life with a single command.
In this step, you will launch all four containers, confirm they are communicating correctly, and explore the raw metrics that Prometheus collects. By the end, your monitoring pipeline will be live and actively collecting data every 15 seconds.
In this step, get ready to:
- Deploy all four containers with a single Docker Compose command.
- Confirm Prometheus is scraping all three targets successfully.
- Explore raw metric data from the exporters.
Start the stack
Docker Compose reads your docker-compose.yml and orchestrates the creation of all resources (network, volumes, containers) in the correct dependency order.
- Deploy the entire monitoring stack by running this command from inside your monitoring-stack directory:
docker compose up -d
What does this command do?
The -d flag runs containers in detached mode, meaning they run in the background and return control of your terminal to you.
Docker will pull all four images from their registries, create the monitoring bridge network, create the named volumes, and start all containers in dependency order.
First run takes a few minutes
Docker needs to download all four container images on the first run. You will see download progress bars for each layer. Subsequent runs will start almost instantly because the images are cached locally.
- Confirm all four containers are running by running this command:
docker compose ps
You should see a table listing all four containers (prometheus, grafana, node-exporter, cadvisor) with STATUS showing "Up" for each one.
One or more containers not showing Up?
- If a container shows "Restarting" or "Exited", check its logs by running docker compose logs <service-name> (replace <service-name> with the name of the failing service, e.g. cadvisor).
- A port conflict ("bind: address already in use") means another process is using one of the required ports (9090, 3000, 9100, or 8080). Stop that process or change the host port in your Compose file.
- Make sure Docker Desktop is running with WSL integration enabled. Without it, volume mounts to /proc and /sys will fail.
Why are my Docker Compose containers not starting? I see errors in docker compose ps showing exited or restarting status.
Verify Prometheus is scraping targets
Prometheus exposes a web UI where you can check whether it successfully reaches each scrape target. This is the first place to look when diagnosing data collection issues.
- Open your web browser.
- Navigate to http://localhost:9090/targets.
You should see three targets listed: prometheus, node-exporter, and cadvisor. All three should show state "UP" with a last scrape time within the last 15 seconds.
What does the targets page tell you?
Each row on the targets page represents one scrape job from your prometheus.yml. The "UP" state means Prometheus successfully connected to that endpoint and pulled metrics. "DOWN" means the target is unreachable.
The "Last Scrape" column shows how long ago Prometheus last collected data. Since your scrape interval is 15 seconds, this value should never exceed 15s during normal operation.
Seeing a target with state DOWN?
- Check that the target container is actually running with docker compose ps. A stopped container cannot respond to scrape requests.
- Verify the target addresses in prometheus.yml use container names (like node-exporter:9100), not localhost. Inside the Docker network, each service resolves by its container name.
- If you just started the stack, wait 15-30 seconds and refresh the page. The first scrape might not have completed yet.
Why is one of my Prometheus scrape targets showing DOWN?
Explore raw metrics
Before building dashboards, it helps to see what raw metric data looks like. Node Exporter exposes hundreds of metrics about your host system at a plain HTTP endpoint.
- Open a new browser tab and navigate to http://localhost:9100/metrics.
You should see a wall of plain text. This is the Prometheus exposition format. Each line is a metric with labels and a numeric value.
- Use your browser's find feature (Ctrl+F) to search for node_cpu_seconds_total. This metric tracks CPU time spent in each mode (user, system, idle).
- Search for node_memory_MemAvailable_bytes. This metric shows how much RAM is currently available on your host.
What is this data?
This is exactly what Prometheus pulls every 15 seconds. Each metric has a name, optional labels in curly braces, and a numeric value. For example, node_cpu_seconds_total{cpu="0",mode="idle"} 12345.67 means CPU 0 has spent 12345.67 seconds in idle mode since boot.
These raw numbers are what Grafana will transform into charts and gauges in the next step. The labels let you filter and group data by CPU core, disk device, network interface, and more.
What about cAdvisor metrics?
You can also visit http://localhost:8080/metrics to see cAdvisor metrics. These track per-container resource usage (CPU, memory, network I/O) for every Docker container on your host.
Your monitoring pipeline is live. Prometheus is actively scraping metrics from both exporters and storing them as time-series data. Next up, you will connect Grafana to Prometheus and build a dashboard that turns all those raw numbers into visual insights.
Build a Grafana Dashboard
Your monitoring stack is running and all three scrape targets are healthy. Prometheus is collecting metrics every 15 seconds, but raw time-series data on its own is hard to interpret.
In this step, you will connect to Grafana, verify the data pipeline is working end-to-end, and build dashboards that turn those metrics into visual panels you can read at a glance.
In this step, get ready to:
- Log into Grafana and verify the Prometheus datasource connection.
- Import a community dashboard to visualize host metrics.
- Write a custom PromQL query to create a CPU usage gauge.
Log into Grafana and verify the datasource
Before building dashboards, you need to confirm that Grafana can talk to Prometheus. The datasource provisioning file you created earlier should have already configured this connection automatically.
- Open http://localhost:3000 in your web browser.
- Log in with username admin and password admin.
Password change prompt
Grafana will immediately ask you to change the default password. Choose a new password you can remember, or click Skip if the option appears. Either way, you will land on the Grafana home page.
- Click Connections in the left-side menu.
- Click Data sources.
- You should see Prometheus listed as a datasource. Click it to open its settings.
- Scroll to the bottom and click Test.
You should see a green banner confirming the datasource is working. This proves the full data pipeline: Node Exporter and cAdvisor expose metrics, Prometheus scrapes them, and Grafana can query Prometheus.
Datasource test failing?
- Make sure all four containers are still running. Switch to your WSL terminal and run docker compose ps to confirm.
- Verify the URL in the datasource settings shows http://prometheus:9090. This is the Docker internal address, not localhost.
- If you edited the provisioning file, restart the Grafana container with docker compose restart grafana.
Why is my Grafana datasource test failing?
Import the Node Exporter Full dashboard
Instead of building every panel from scratch, you can import a community dashboard that already visualizes dozens of Node Exporter metrics. Dashboard ID 1860 is the popular "Node Exporter Full" dashboard.
- Click Dashboards in the left-side menu.
- Click New and select Import from the drop-down menu.
- In the import field, enter 1860 and click Load.
- In the Prometheus dropdown at the bottom, select your Prometheus datasource.
- Click Import.
The dashboard loads immediately with dozens of panels showing CPU, memory, disk, network, and filesystem metrics. The data updates every 15 seconds as Prometheus scrapes new values.
Why import a community dashboard?
Community dashboards give you instant visibility while you learn how Grafana works. The "Node Exporter Full" dashboard has been refined by thousands of contributors to cover every useful host metric. You can explore each panel's query to learn PromQL patterns.
Dashboard shows No Data?
- Make sure you selected the correct Prometheus datasource during import. If you skipped it, edit the dashboard settings and assign the datasource.
- Check the time range in the top-right corner of the dashboard. Set it to Last 5 minutes to see recent data.
- Confirm Node Exporter is still scraping by visiting http://localhost:9090/targets and checking all targets show UP.
Why is my imported Grafana dashboard showing no data?
Write a custom PromQL query
Imported dashboards are useful, but writing your own queries teaches you how the visualization layer actually works. PromQL is Prometheus's query language for selecting, filtering, and transforming time-series metrics.
- Click Dashboards in the left-side menu.
- Click New and select New dashboard.
- Click Add visualization.
- Select your Prometheus datasource when prompted.
- In the query editor at the bottom, switch to Code mode if it shows the visual builder.
- Enter the following PromQL query:
100 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
What does this query do?
- node_cpu_seconds_total{mode="idle"} selects the CPU idle time metric, filtered by the idle label.
- irate(...[5m]) calculates the per-second rate of change over a 5-minute window.
- avg(...) averages across all CPU cores.
- 100 - (... * 100) inverts the idle percentage into a usage percentage. If the CPU is 90% idle, this returns 10% usage.
- In the top-right panel area, open the visualization type dropdown (it may default to "Time series").
- Select Gauge as the visualization type.
- In the panel options on the right side, find the Standard options section.
- Set Max to 100.
- Click Apply in the top-right to save the panel to your dashboard.
Your gauge now displays a real-time CPU usage percentage that updates every 15 seconds.
Gauge showing no data or NaN?
- Double-check the query for typos. The metric name is node_cpu_seconds_total with underscores, not hyphens.
- Make sure you are in Code mode in the query editor. The visual builder may not support this query structure.
- If the gauge shows NaN, Node Exporter may not have collected enough data yet. Wait 5 minutes for the 5-minute rate window to fill.
Why is my PromQL gauge query returning no data?
You now have both a full community dashboard and a custom panel you built yourself. In the Secret Mission, you will push this stack further by adding automated alerting when CPU usage crosses a threshold.
Secret mission
Add Alertmanager with CPU Alert Rules
Your monitoring stack collects and visualizes metrics. But what happens when something goes wrong at 3 AM and nobody is watching the dashboard? In this mission, you will add Alertmanager as a fifth service and write a custom alerting rule that fires when CPU usage exceeds 80%.
Clean Up Your Resources
Clean Up Your Resources
This project runs entirely locally, so there are no ongoing financial costs. However, the running containers consume RAM and disk space for stored metrics data.
Decide whether to keep your resources running, pause them to come back later, or delete them entirely.
Resources you used:
- 5 Docker containers (prometheus, grafana, node-exporter, cadvisor, alertmanager).
- 3 named Docker volumes (prometheus-data, grafana-data, alertmanager-data).
- 1 Docker bridge network (monitoring).
- The monitoring-stack project directory and all configuration files.
Keep everything running
No action needed. Choose this if you are still actively building or want to observe metrics over time. The longer the stack runs, the more historical data you accumulate in Prometheus.
Pause - I'll come back to this later
Stop the containers to free up RAM without removing any data. Your metrics history and Grafana dashboards stay intact on disk.
- In your WSL terminal, navigate to the monitoring-stack directory and stop all containers by running:
cd ~/Desktop/monitoring-stack
docker compose stop
When you are ready to resume, start the containers again by running:
docker compose start
All five containers will restart with your existing data and dashboards intact.
Delete - I don't want to use this again
Remove all containers, networks, volumes, and project files. This frees all disk space but permanently deletes your stored metrics and Grafana dashboard configurations.
- In your WSL terminal, navigate to the monitoring-stack directory and tear down the entire stack (including named volumes) by running:
cd ~/Desktop/monitoring-stack
docker compose down -v
What does this command do?
The docker compose down -v command removes all containers, the monitoring bridge network, and all three named volumes (prometheus-data, grafana-data, alertmanager-data). This is a destructive operation with no confirmation prompt.
- Navigate up one directory and remove the project folder by running:
cd ..
rm -rf monitoring-stack
- Confirm the directory is gone by running:
ls ~/Desktop | grep monitoring
You should see no output, confirming the project files are deleted.
Nice Work!
Nice Work!
Nice work! You just deployed a production-style server monitoring stack using Docker Compose, complete with metric collection, time-series storage, and dashboard visualization.
You've learned how to:
- Deployed a 4-service monitoring stack using Docker Compose with pinned image versions and persistent storage.
- Configured Prometheus to scrape host metrics via Node Exporter and container metrics via cAdvisor on a 15-second interval.
- Built a Grafana dashboard with both imported community dashboards and custom PromQL queries.
- Secret Mission: Added Alertmanager with a custom CPU alerting rule and verified the full alert lifecycle (inactive, pending, firing).
Ready to quiz yourself?