Build a Monitoring Stack with Prometheus

Deploy Prometheus, Grafana, and Alertmanager to monitor a Flask app locally.

Introduction

30 Second Summary

Every app has a heartbeat. In production, if nobody is watching that heartbeat, problems snowball into outages before anyone notices.

In this project, you will deploy a full monitoring stack using Prometheus, Grafana, and Alertmanager on your local machine with Docker Compose. You'll instrument a Flask application with custom metrics, build dashboards to visualize performance, and configure alerts that fire when your app's error rate spikes.

What You'll Build

A complete local monitoring pipeline that scrapes metrics from a custom-instrumented Flask application, visualizes them in Grafana dashboards, and triggers alerts through Alertmanager when error rates exceed thresholds.

By the end of this project, you'll have:

  • A Docker Compose stack running Prometheus, Grafana, Alertmanager, Node Exporter, and a Flask app as five orchestrated services.
  • A custom-instrumented Flask application exposing Counter, Histogram, and Gauge metrics that Prometheus scrapes automatically.
  • A Grafana dashboard visualizing request latency percentiles and error rates, plus alerting rules that fire when your app misbehaves.
  • Secret Mission: Add recording rules for pre-computed metrics and per-endpoint alerting with inhibition rules.

Are there any prerequisites?

You'll need basic familiarity with the Linux command line (navigating directories, running commands). You also need Docker and Docker Compose installed on your machine.

Step 1 will walk you through verifying your Docker installation before you build anything.

Set Up Your Project Structure

A monitoring stack has multiple components that each need their own configuration. Prometheus needs scrape targets, Alertmanager needs routing rules, and your Flask app needs its own directory for source code and a Dockerfile.

In this step, you'll scaffold the project directory, confirm that Docker is ready to orchestrate your containers, and install the Python packages your Flask app depends on.

In this step, get ready to:
  • Create the project directory with subdirectories for Prometheus, Alertmanager, and the Flask app.
  • Verify Docker and Docker Compose are installed and running.
  • Set up a Python virtual environment with Flask and prometheus-client.
Create the project directory

Every service in your monitoring stack will have its own folder for configuration files or source code. Creating this structure now keeps things organized as you add more files in later steps.

  • Open your terminal. Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or PowerShell (Windows) and press Enter to open it.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir monitoring-stack
cd monitoring-stack
  • Create the subdirectories for each component by running:
mkdir prometheus alertmanager flask-app
  • Confirm the directories exist by running:
ls

You should see prometheus, alertmanager, and flask-app listed in the output.

Don't see the folders?

Make sure you ran cd monitoring-stack before running ls. If you're still in ~/Desktop, you'll see everything on your Desktop instead.

help me figure out why my project directories aren't showing up

Verify Docker and Docker Compose are installed

Your monitoring stack runs as Docker containers orchestrated by Docker Compose. Docker Compose V2 is a plugin built into Docker Desktop, and this project uses the docker compose command (without a hyphen).

  • Check your Docker installation by running:
docker --version
  • Then check Docker Compose by running:
docker compose version

✔️ Both commands show version numbers

Docker and Docker Compose are ready. You can move on to the next substep.

ⓧ docker compose says command not found

You have Docker installed, but you're missing the Compose V2 plugin. This happens when you installed Docker Engine standalone without Docker Desktop.

The easiest fix is to install Docker Desktop, which bundles the Compose plugin automatically.

  • Download Docker Desktop from https://www.docker.com/products/docker-desktop/.
  • Run the installer and follow the setup wizard.
  • After installation, open Docker Desktop and wait for the engine to start.
  • Run docker compose version again to confirm it now works.

ⓧ Docker not installed

You need to install Docker Desktop, which includes both Docker Engine and the Compose plugin.

  • Download Docker Desktop from https://www.docker.com/products/docker-desktop/.
  • Run the installer and follow the setup wizard.
  • After installation, open Docker Desktop and wait for the whale icon in your system tray to show "Docker Desktop is running".
  • Run both commands again to confirm they output version numbers.
Create a Python virtual environment

You'll install Flask and the prometheus-client library in a virtual environment. This keeps your project's dependencies isolated from other Python projects on your machine.

  • Check that Python 3 is available by running:
python3 --version

✔️ I see a version number

Python 3 is ready. As long as you see 3.9 or higher, you're good to go.

ⓧ Command not found

You need Python 3.9 or higher installed. Flask 3.1.1 requires it.

  • Download Python from https://www.python.org/downloads/.
  • Run the installer. On Windows, check the box that says Add python.exe to PATH before clicking Install Now.
  • Close and reopen your terminal, then run python3 --version again.
  • Make sure you are inside the monitoring-stack folder, then create the virtual environment by running:
python3 -m venv venv
  • Activate the virtual environment:

macOS

source venv/bin/activate

Windows

venv\Scripts\activate

You should see (venv) appear at the beginning of your terminal prompt. This confirms the virtual environment is active.

  • Install Flask and prometheus-client by running:
pip install flask==3.1.1 prometheus-client==0.25.0

What does this command do?

pip install downloads and installs Python packages into your active virtual environment.

The == syntax pins exact versions so everyone working on this project uses the same dependencies. flask==3.1.1 is the web framework and prometheus-client==0.25.0 is the official Python library for exposing Prometheus metrics.

  • Verify the packages installed correctly by running:
pip show flask prometheus-client

You should see package details for both Flask and prometheus-client, including their version numbers matching what you installed.

Seeing a permissions error or install failure?

Make sure your virtual environment is active. Check for (venv) at the start of your terminal prompt. If it's missing, run the activate command again.

If you see an error about pip not being found, try python3 -m pip install instead.

help me fix my pip install error for flask and prometheus-client

Your project structure is ready and your development environment is set up. Next up, you'll build the instrumented Flask application that exposes custom Prometheus metrics.

Build the Instrumented Flask Application

Your project structure is in place and your Python environment is ready. Now it's time to build the application that Prometheus will actually monitor.

In this step, you will create a Flask web application that exposes three types of Prometheus metrics: a Counter for total requests, a Histogram for request duration, and a Gauge for in-flight requests. You will then containerize it with Docker so it can run alongside the rest of your monitoring stack.

In this step, get ready to:
  • Create an instrumented Flask app with Counter, Histogram, and Gauge metrics.
  • Mount the Prometheus metrics endpoint at /metrics using DispatcherMiddleware.
  • Containerize the Flask app with a Dockerfile and verify metrics output.
Create the Flask app with instrumented endpoints

The prometheus-client library provides three core metric types that cover most monitoring needs. A Counter tracks cumulative totals (like total requests). A Histogram records the distribution of values (like request latency). A Gauge tracks values that go up and down (like in-flight requests).

  • Create a new file called app.py inside the flask-app folder by running:
touch flask-app/app.py
  • Open flask-app/app.py in your editor and add the imports and metric definitions by pasting this code:
import time
import random

from flask import Flask, Response, jsonify
from werkzeug.middleware.dispatcher import DispatcherMiddleware
from prometheus_client import (
    Counter,
    Histogram,
    Gauge,
    make_wsgi_app,
)

app = Flask(__name__)

# Metrics
REQUEST_COUNT = Counter(
    'flask_http_requests_total',
    'Total HTTP requests',
    ['endpoint', 'status']
)

REQUEST_DURATION = Histogram(
    'flask_http_request_duration_seconds',
    'HTTP request duration in seconds',
    ['endpoint'],
    buckets=[0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0]
)

IN_FLIGHT = Gauge(
    'flask_http_requests_in_flight',
    'Number of HTTP requests currently being processed'
)

What does this code do?

The imports bring in Flask for the web server, Werkzeug's DispatcherMiddleware for routing, and the three Prometheus metric types plus make_wsgi_app for serving metrics.

  • REQUEST_COUNT is a Counter with two labels: endpoint and status. Labels let you filter and group metrics later in PromQL queries.
  • REQUEST_DURATION is a Histogram with custom buckets tuned for web request latencies (10ms through 5s).
  • IN_FLIGHT is a Gauge that tracks how many requests are being processed right now.
  • Now add the three route handlers below the metric definitions. These are the endpoints Prometheus will monitor. Paste this code at the bottom of app.py:

@app.route('/')
@IN_FLIGHT.track_inprogress()
def home():
    start = time.time()
    time.sleep(random.uniform(0.01, 0.05))
    duration = time.time() - start
    REQUEST_DURATION.labels(endpoint='/').observe(duration)
    REQUEST_COUNT.labels(endpoint='/', status='200').inc()
    return jsonify({"message": "Hello from the monitored Flask app!"})


@app.route('/slow')
@IN_FLIGHT.track_inprogress()
def slow():
    start = time.time()
    time.sleep(random.uniform(0.5, 2.0))
    duration = time.time() - start
    REQUEST_DURATION.labels(endpoint='/slow').observe(duration)
    REQUEST_COUNT.labels(endpoint='/slow', status='200').inc()
    return jsonify({"message": "This was a slow request"})


@app.route('/error')
@IN_FLIGHT.track_inprogress()
def error():
    start = time.time()
    time.sleep(random.uniform(0.01, 0.1))
    duration = time.time() - start
    REQUEST_DURATION.labels(endpoint='/error').observe(duration)
    REQUEST_COUNT.labels(endpoint='/error', status='500').inc()
    return Response("Internal Server Error", status=500)

What does this code do?

Each endpoint follows the same instrumentation pattern: start a timer, simulate work with time.sleep(), record the duration in the Histogram, increment the Counter with labels, and return a response.

  • / simulates a fast request (10-50ms). It always returns a 200 status.
  • /slow simulates a slow request (0.5-2 seconds). This helps you see latency distribution later.
  • /error always returns a 500 status. This endpoint lets you trigger alerts when error rates spike.
  • The @IN_FLIGHT.track_inprogress() decorator automatically increments the Gauge when a request starts and decrements it when the request finishes.
  • Save flask-app/app.py.

Seeing a syntax error?

Make sure there are two blank lines between the IN_FLIGHT Gauge definition and the first @app.route decorator. Python is sensitive to indentation and spacing around function definitions.

Help me debug a syntax error in my Flask app with Prometheus metrics.

Mount the Prometheus metrics endpoint

Prometheus scrapes metrics by making HTTP GET requests to a /metrics endpoint on your application. The prometheus-client library provides make_wsgi_app() which creates a WSGI application that serves all registered metrics in Prometheus format.

  • Add the webhook endpoint, the DispatcherMiddleware mount, and the main block at the bottom of flask-app/app.py:

@app.route('/webhook', methods=['POST'])
def webhook():
    """Receives alert notifications from Alertmanager"""
    return jsonify({"status": "received"}), 200


# Mount prometheus metrics endpoint at /metrics
app.wsgi_app = DispatcherMiddleware(app.wsgi_app, {
    '/metrics': make_wsgi_app()
})

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000, debug=False)

What does this code do?

  • The /webhook endpoint receives POST requests from Alertmanager when an alert fires. You will configure this in a later step.
  • DispatcherMiddleware intercepts requests to /metrics and routes them to the Prometheus WSGI app instead of Flask. All other requests pass through to Flask normally.
  • debug=False is required because Flask's reloader conflicts with prometheus-client. The reloader spawns a second process that tries to re-register metrics, causing errors.
  • Now create the requirements.txt file that the Docker image will use. Create it inside the flask-app folder with this content:
flask==3.1.1
prometheus-client==0.25.0
werkzeug==3.1.3

Why pin Werkzeug separately?

Werkzeug is Flask's underlying WSGI toolkit and provides the DispatcherMiddleware class. Pinning it explicitly ensures the Docker container uses the same version you tested with locally.

  • Save both flask-app/app.py and flask-app/requirements.txt.

✔️ Awesome, I've got everything!

Great. Double check you have saved both files before moving on.

ⓧ I'd like to double check the full code

Here is the complete flask-app/app.py file:

import time
import random

from flask import Flask, Response, jsonify
from werkzeug.middleware.dispatcher import DispatcherMiddleware
from prometheus_client import (
    Counter,
    Histogram,
    Gauge,
    make_wsgi_app,
)

app = Flask(__name__)

# Metrics
REQUEST_COUNT = Counter(
    'flask_http_requests_total',
    'Total HTTP requests',
    ['endpoint', 'status']
)

REQUEST_DURATION = Histogram(
    'flask_http_request_duration_seconds',
    'HTTP request duration in seconds',
    ['endpoint'],
    buckets=[0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0]
)

IN_FLIGHT = Gauge(
    'flask_http_requests_in_flight',
    'Number of HTTP requests currently being processed'
)


@app.route('/')
@IN_FLIGHT.track_inprogress()
def home():
    start = time.time()
    time.sleep(random.uniform(0.01, 0.05))
    duration = time.time() - start
    REQUEST_DURATION.labels(endpoint='/').observe(duration)
    REQUEST_COUNT.labels(endpoint='/', status='200').inc()
    return jsonify({"message": "Hello from the monitored Flask app!"})


@app.route('/slow')
@IN_FLIGHT.track_inprogress()
def slow():
    start = time.time()
    time.sleep(random.uniform(0.5, 2.0))
    duration = time.time() - start
    REQUEST_DURATION.labels(endpoint='/slow').observe(duration)
    REQUEST_COUNT.labels(endpoint='/slow', status='200').inc()
    return jsonify({"message": "This was a slow request"})


@app.route('/error')
@IN_FLIGHT.track_inprogress()
def error():
    start = time.time()
    time.sleep(random.uniform(0.01, 0.1))
    duration = time.time() - start
    REQUEST_DURATION.labels(endpoint='/error').observe(duration)
    REQUEST_COUNT.labels(endpoint='/error', status='500').inc()
    return Response("Internal Server Error", status=500)


@app.route('/webhook', methods=['POST'])
def webhook():
    """Receives alert notifications from Alertmanager"""
    return jsonify({"status": "received"}), 200


# Mount prometheus metrics endpoint at /metrics
app.wsgi_app = DispatcherMiddleware(app.wsgi_app, {
    '/metrics': make_wsgi_app()
})

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000, debug=False)

And here is flask-app/requirements.txt:

flask==3.1.1
prometheus-client==0.25.0
werkzeug==3.1.3

App won't start or imports fail?

Make sure your virtual environment is still active (you should see the environment name in your terminal prompt). If it is deactivated, re-activate it before running the app.

Help me fix an import error in my Flask Prometheus app.

Create a Dockerfile and test the app

The Flask app will run as a Docker container alongside Prometheus and Grafana. You need a Dockerfile that packages the app and its dependencies into an image.

  • Create a file called Dockerfile inside the flask-app folder with this content:
FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

EXPOSE 5000

CMD ["python", "app.py"]

What does this Dockerfile do?

  • FROM python:3.12-slim uses a lightweight Python base image to keep the container small.
  • COPY requirements.txt is copied first so Docker can cache the pip install layer. If only your code changes, Docker skips reinstalling dependencies.
  • EXPOSE 5000 documents the port the app listens on. Docker Compose will map this to your host.
  • Save flask-app/Dockerfile.
  • Test the Flask app locally before containerizing it. Run the app directly with Python by running:
python flask-app/app.py

You should see output indicating the Flask server is running on port 5000.

  • Open a second terminal and verify the metrics endpoint by running:
curl http://localhost:5000/metrics

You should see Prometheus-format output with lines like # HELP flask_http_requests_total and # TYPE flask_http_requests_total counter. You will also see default Python process metrics.

  • Stop the Flask server by pressing Ctrl+C in the first terminal.

Curl returns connection refused?

Make sure the Flask app is still running in the first terminal. If it crashed on startup, check the error message for import issues or syntax errors in app.py.

Also verify you are using localhost:5000 and not a different port. The app binds to 0.0.0.0:5000 as specified in the code.

Help me troubleshoot why curl to localhost:5000/metrics is returning connection refused.

Your instrumented Flask app is ready and verified. Next up, you will configure Prometheus to scrape these metrics and deploy the entire monitoring stack with Docker Compose.

Configure Prometheus and Deploy the Stack

Your instrumented Flask app is ready to report metrics. Now you need something to collect those metrics, store them as time series data, and alert you when things go wrong.

In this step, you'll configure Prometheus as your metrics collector, Alertmanager as your notification router, and Docker Compose to orchestrate the entire stack.

In this step, get ready to:
  • Configure Prometheus with scrape targets for your Flask app, Node Exporter, and itself.
  • Set up Alertmanager with a webhook receiver for alert notifications.
  • Deploy all 5 services with Docker Compose.
Write the Prometheus configuration

Prometheus uses a YAML configuration file to know which services to monitor and how frequently to pull their metrics. This is called scraping.

  • Create a file called prometheus.yml inside your prometheus/ folder and paste in the following configuration:
global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093

rule_files:
  - /etc/prometheus/alert_rules.yml

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']

  - job_name: 'flask-app'
    static_configs:
      - targets: ['flask-app:5000']

What does this configuration do?

  • scrape_interval: 15s tells Prometheus to pull metrics from every target every 15 seconds.
  • evaluation_interval: 15s controls how often Prometheus checks alerting rules.
  • The alerting section tells Prometheus where to send fired alerts (Alertmanager on port 9093).
  • Three scrape_configs jobs define the targets: Prometheus itself, Node Exporter for host metrics, and your Flask app.

Prometheus also needs an alert rules file, even if it's empty for now. You'll populate this with real rules in Step 6.

  • Create a file called alert_rules.yml inside your prometheus/ folder and paste in this placeholder:
groups: []
  • Confirm both files exist by running:
ls prometheus/

You should see alert_rules.yml and prometheus.yml in the output.

Don't see the files?

  • Make sure you're in the project root directory (monitoring-stack), not inside the flask-app/ folder.
  • Check that the prometheus/ folder exists. If not, create it with mkdir prometheus.

Help me troubleshoot my prometheus config files

✔️ Awesome, I've got everything!

Great. Make sure both files are saved before moving on.

ⓧ I'd like to double check the full code

global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093

rule_files:
  - /etc/prometheus/alert_rules.yml

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']

  - job_name: 'flask-app'
    static_configs:
      - targets: ['flask-app:5000']
Write the Alertmanager configuration

Alertmanager handles what happens after Prometheus fires an alert. It groups related alerts, waits for duplicates, and routes notifications to the right receiver.

  • Create a file called alertmanager.yml inside your alertmanager/ folder and paste in the following configuration:
global:
  resolve_timeout: 5m

route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'webhook'

receivers:
  - name: 'webhook'
    webhook_configs:
      - url: 'http://flask-app:5000/webhook'
        send_resolved: true

What does this configuration do?

  • resolve_timeout: 5m means if an alert stops firing for 5 minutes, Alertmanager considers it resolved.
  • group_by: ['alertname'] groups related alerts together so you don't get spammed with one notification per instance.
  • The webhook receiver sends alert notifications as HTTP POST requests to your Flask app's /webhook endpoint (the one you built in Step 2).
  • send_resolved: true means you also get notified when an alert clears.
  • Confirm the file exists by running:
ls alertmanager/

You should see alertmanager.yml in the output.

File not showing up?

  • Make sure the alertmanager/ folder exists at the project root. Create it with mkdir alertmanager if needed.
  • Double check that the filename is exactly alertmanager.yml (not alertmanager.yaml or with a typo).

Help me fix my Alertmanager config setup

✔️ Awesome, I've got everything!

Great. Make sure the file is saved before moving on.

ⓧ I'd like to double check the full code

global:
  resolve_timeout: 5m

route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'webhook'

receivers:
  - name: 'webhook'
    webhook_configs:
      - url: 'http://flask-app:5000/webhook'
        send_resolved: true
Write Docker Compose and deploy the stack

Docker Compose lets you define and run multi-container applications with a single file. Your docker-compose.yml will orchestrate all 5 services so they can communicate with each other on the same network.

  • Create a file called docker-compose.yml in your project root and paste in the following:
services:
  prometheus:
    image: prom/prometheus:v3.12.0
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./prometheus/alert_rules.yml:/etc/prometheus/alert_rules.yml:ro
      - prometheus-data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=15d'
      - '--web.enable-lifecycle'
    restart: unless-stopped

  grafana:
    image: grafana/grafana-enterprise:13.0.2
    container_name: grafana
    ports:
      - "3000:3000"
    volumes:
      - grafana-storage:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    restart: unless-stopped

  alertmanager:
    image: prom/alertmanager:v0.32.1
    container_name: alertmanager
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
    command:
      - '--config.file=/etc/alertmanager/alertmanager.yml'
      - '--storage.path=/alertmanager'
    restart: unless-stopped

  node-exporter:
    image: prom/node-exporter:v1.11.1
    container_name: node-exporter
    ports:
      - "9100:9100"
    restart: unless-stopped

  flask-app:
    build: ./flask-app
    container_name: flask-app
    ports:
      - "5000:5000"
    restart: unless-stopped

volumes:
  prometheus-data:
  grafana-storage:

What does this Docker Compose file do?

  • Five services are defined: Prometheus (port 9090), Grafana (port 3000), Alertmanager (port 9093), Node Exporter (port 9100), and your Flask app (port 5000).
  • Config files are mounted as read-only (:ro) so containers can read them but not modify them.
  • Named volumes (prometheus-data and grafana-storage) persist data between container restarts.
  • The --web.enable-lifecycle flag lets you reload Prometheus configuration without restarting the container.
  • The Flask app uses build: ./flask-app which tells Docker Compose to build the image from your Dockerfile.
  • Save docker-compose.yml.
  • Deploy the entire stack by running:
docker compose up -d

What does this command do?

The -d flag runs all containers in detached mode (in the background). Docker Compose will pull the required images, build the Flask app image from your Dockerfile, create a shared network, and start all 5 services.

This may take a minute or two on the first run while Docker downloads the images.

  • Verify all containers are running by running:
docker compose ps

You should see all 5 containers listed with a STATUS of Up.

Seeing containers with Exited status?

  • Check the logs for the failing container. For example, run docker compose logs prometheus to see what went wrong.
  • A common cause is a YAML syntax error in your config files. Make sure indentation uses spaces (not tabs) and colons are followed by a space.
  • If the Flask app fails, check that your flask-app/ folder contains both app.py and requirements.txt.

Help me fix my Docker Compose containers

✔️ Awesome, I've got everything!

Great. All 5 containers should show Up in the STATUS column.

ⓧ I'd like to double check the full code

services:
  prometheus:
    image: prom/prometheus:v3.12.0
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./prometheus/alert_rules.yml:/etc/prometheus/alert_rules.yml:ro
      - prometheus-data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=15d'
      - '--web.enable-lifecycle'
    restart: unless-stopped

  grafana:
    image: grafana/grafana-enterprise:13.0.2
    container_name: grafana
    ports:
      - "3000:3000"
    volumes:
      - grafana-storage:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    restart: unless-stopped

  alertmanager:
    image: prom/alertmanager:v0.32.1
    container_name: alertmanager
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
    command:
      - '--config.file=/etc/alertmanager/alertmanager.yml'
      - '--storage.path=/alertmanager'
    restart: unless-stopped

  node-exporter:
    image: prom/node-exporter:v1.11.1
    container_name: node-exporter
    ports:
      - "9100:9100"
    restart: unless-stopped

  flask-app:
    build: ./flask-app
    container_name: flask-app
    ports:
      - "5000:5000"
    restart: unless-stopped

volumes:
  prometheus-data:
  grafana-storage:

Your entire monitoring stack is now live. Next up, you'll explore the Prometheus UI and learn to query your metrics with PromQL.

Explore Prometheus and Write PromQL Queries

Your monitoring stack is running. All five containers are up, scraping metrics every 15 seconds.

Now it is time to verify the data is flowing and learn how to ask questions about it. PromQL is the query language that Prometheus uses to slice, aggregate, and transform time-series data. In this step, you will verify your targets, write real queries, and generate traffic so you have something meaningful to look at.

In this step, get ready to:
  • Confirm all scrape targets are healthy in the Prometheus UI.
  • Write PromQL queries for request rate, latency percentiles, and error rate.
  • Generate traffic to your Flask app and watch metrics update in real time.
Verify all targets are UP in the Prometheus UI

Before you write queries, you need to confirm Prometheus is successfully scraping all three targets. The Targets page shows the health status of every scrape job you defined in prometheus.yml.

  • Open your browser and navigate to http://localhost:9090.
  • Click **Status** in the top navigation bar.
  • Select **Targets** from the dropdown menu.

You should see three scrape jobs listed: prometheus, node-exporter, and flask-app. Each one should show a green **UP** status.

What does UP mean?

A target marked UP means Prometheus successfully connected to it and scraped its /metrics endpoint within the last scrape interval (15 seconds in your config).

If a target shows DOWN, Prometheus could not reach it. This usually means the container is not running or there is a network issue between containers.

  • Click **Graph** in the top navigation bar to open the expression browser.
  • Type up in the expression input field.
  • Click **Execute**.

You should see three results, each with a value of 1. A value of 1 means the target is up. A value of 0 means it is down.

What is the up metric?

The up metric is automatically generated by Prometheus for every scrape target. It is a quick health check you can query at any time to verify your monitoring pipeline is working end-to-end.

Seeing a target marked DOWN?

Make sure all containers are running by checking docker ps in your terminal. If a container is missing, run docker compose up -d from your monitoring-stack folder to restart it.

Prometheus scrapes every 15 seconds. If you just started the stack, wait 30 seconds and reload the Targets page.

Why is my target showing DOWN in Prometheus?

Write PromQL queries to calculate rate, latency, and error rate

With your targets healthy, you can now query the metrics your Flask app is exposing. Each query answers a real operational question you would ask in production.

  • In the Prometheus expression browser, clear the expression field.
  • Paste the following query to calculate requests per second over the last 5 minutes:
rate(flask_http_requests_total[5m])

What does rate() do?

The rate() function calculates the per-second average rate of increase of a counter over a time window. Counters only go up, so rate() converts raw totals into a per-second speed.

The [5m] part is the lookback window. It tells Prometheus to average the rate over the last 5 minutes of data points.

  • Click **Execute** and review the results in the table.
  • Now clear the expression and paste this query to compute the 95th percentile of request latency:
histogram_quantile(0.95, rate(flask_http_request_duration_seconds_bucket[5m]))

What does histogram_quantile() do?

The histogram_quantile() function estimates a percentile from histogram bucket data. The first argument 0.95 means the 95th percentile.

This tells you the response time that 95% of requests complete within. If the p95 is 1.5 seconds, it means only 5% of requests take longer than that.

  • Click **Execute** to see the p95 latency result.
  • Clear the expression one more time and paste this query to calculate the error rate:
sum(rate(flask_http_requests_total{status="500"}[5m])) / sum(rate(flask_http_requests_total[5m]))

What does this error rate query do?

The numerator counts only requests where status="500" (errors). The denominator counts all requests regardless of status. Dividing them gives you the fraction of requests that are failing.

A result of 0.33 means 33% of requests are returning errors. This is the exact query you will use later to trigger alerts.

  • Click **Execute** to see the current error rate.

If you have not sent any traffic to the Flask app yet, you may see empty results or NaN (not a number). This is normal. The next substep generates traffic so these queries return real data.

Seeing no data or empty results?

Prometheus needs at least two scrape cycles of data before rate() can calculate a rate. Wait at least 30 seconds after sending traffic, then re-execute the query.

If you see a syntax error, make sure you copied the entire query including the closing parentheses and brackets.

My PromQL query returns empty or NaN results. What is wrong?

Generate traffic and watch metrics update in real time

Metrics are only interesting when there is actual data flowing through your app. You will send 100 requests to each of the three Flask endpoints to give Prometheus meaningful data to aggregate.

  • Open a terminal window and generate traffic by running this command:
for i in $(seq 1 100); do curl -s localhost:5000/ > /dev/null; curl -s localhost:5000/slow > /dev/null; curl -s localhost:5000/error > /dev/null; done

What does this loop do?

This loop sends 100 requests to each endpoint: / (fast, returns 200), /slow (deliberately delayed, returns 200), and /error (returns 500). Together, they produce a realistic mix of fast, slow, and failing requests.

The -s flag silences curl's progress output. The > /dev/null discards the response body so your terminal stays clean.

This command will take a few minutes to complete because the /slow endpoint sleeps for 0.5 to 2 seconds per request. Let it run.

  • Switch back to the Prometheus expression browser in your browser.
  • Re-enter the request rate query and click **Execute**:
rate(flask_http_requests_total[5m])
  • Click the **Graph** tab (below the expression field) to visualize the rate as a time-series chart.

You should see lines appearing for each endpoint/status combination. The / and /error endpoints will show higher rates than /slow because they respond faster.

  • Try the error rate query again to confirm it now returns a decimal value:
sum(rate(flask_http_requests_total{status="500"}[5m])) / sum(rate(flask_http_requests_total[5m]))

You should see a value around 0.33 because one out of three endpoints (/error) always returns a 500 status code.

Still seeing empty results after sending traffic?

Wait at least 30 seconds after the traffic loop starts. Prometheus needs time to scrape and collect enough data points for rate() to work.

If curl fails with a connection refused error, verify the Flask app container is running with docker ps. Look for the flask-app container on port 5000.

Why is my Prometheus query still empty after generating traffic?

You now know how to verify your monitoring pipeline and query real metrics with PromQL. Next up, you will build a Grafana dashboard to visualize these queries in polished, auto-refreshing panels.

Build a Grafana Dashboard

You have Prometheus scraping metrics and you can query them in the Prometheus UI. That gets the job done for ad-hoc debugging, but it is not how teams monitor production services.

Production teams build persistent dashboards that visualize key signals at a glance. Grafana is the industry-standard tool for this. In this step, you will connect Grafana to your Prometheus instance and build a multi-panel dashboard that shows request rate, latency percentiles, error rate, and CPU usage.

In this step, get ready to:
  • Connect Grafana to Prometheus as a data source.
  • Build a four-panel dashboard with request rate, latency, error rate, and CPU usage.
  • Save the dashboard for ongoing monitoring.
Log into Grafana and add the Prometheus data source

Grafana needs to know where to find your metrics before it can display them. You will point it at the Prometheus container running inside the same Docker network.

  • Open your browser and navigate to http://localhost:3000.
  • On the sign-in page, enter admin for both the username and password.
  • Click Sign in.
  • Grafana will prompt you to change the default password. You can click Skip for now since this is a local learning environment.

Why admin/admin?

The docker-compose.yml sets GF_SECURITY_ADMIN_PASSWORD=admin as an environment variable. This overrides Grafana's default and locks the admin password to admin. In production, you would use a strong, unique password.

  • Click Connections in the left-side menu.
  • Under Connections, click Add new connection.
  • Enter Prometheus in the search bar.
  • Click Prometheus data source.
  • Click Add new data source in the upper right.
  • In the Connection section, enter the following URL in the Prometheus server URL field:
http://prometheus:9090

Why prometheus instead of localhost?

Grafana runs inside the Docker network alongside Prometheus. Inside that network, containers reference each other by their service name defined in docker-compose.yml. The service name prometheus resolves to the Prometheus container's internal IP.

Using localhost here would point Grafana at its own container, not at Prometheus.

  • Scroll to the bottom of the page and click Save & test.
  • You should see a green success message confirming the data source is working.

Connection failed?

  • Make sure all Docker containers are still running. Switch back to your terminal and run docker compose ps to check.
  • Confirm the URL is exactly http://prometheus:9090 with no trailing slash.
  • If you accidentally typed localhost instead of prometheus, replace it with the Docker service name.

Help me debug my Grafana data source connection to Prometheus.

Create dashboard panels for request rate and latency

A good monitoring dashboard answers questions at a glance. You will create panels that answer: how many requests is each endpoint getting, and how fast are they responding?

  • Click Dashboards in the left-side menu.
  • Click New and select New Dashboard.
  • Click the Add new element icon (blue plus sign) and drag a panel onto the dashboard.
  • On the new panel, click Configure visualization to open the panel editor.

You are now in the panel editor. Your Prometheus data source should be pre-selected since it is the only one configured.

  • In the query editor at the bottom of the screen, enter this PromQL query:
sum by (endpoint)(rate(flask_http_requests_total[5m]))
  • Click Refresh to run the query.
  • Confirm that the visualization type is set to Time series (this is the default for Prometheus).
  • In the panel options on the right, set the title to Request Rate by Endpoint.
  • Click Save in the top right to save the panel.

What does this query do?

The rate() function calculates the per-second rate of increase over the last 5 minutes. Wrapping it with sum by (endpoint) groups the results by endpoint label, giving you one line per route (/, /slow, /error).

Now add a second panel for latency percentiles.

  • Click the Add new element icon (blue plus sign) again and drag another panel onto the dashboard.
  • Click Configure visualization on the new panel.
  • In the query editor, enter the p50 latency query:
histogram_quantile(0.5, rate(flask_http_request_duration_seconds_bucket[5m]))
  • Click + Query to add a second query row.
  • In the second query row, enter the p95 latency query:
histogram_quantile(0.95, rate(flask_http_request_duration_seconds_bucket[5m]))
  • Click + Query again to add a third query row.
  • In the third query row, enter the p99 latency query:
histogram_quantile(0.99, rate(flask_http_request_duration_seconds_bucket[5m]))
  • Click Refresh to see all three percentile lines on the graph.
  • Keep the visualization type as Time series.
  • Set the panel title to Request Latency (p50/p95/p99).
  • Click Save.

What is histogram_quantile?

The histogram_quantile() function estimates percentile values from histogram bucket data. The first argument (0.5, 0.95, 0.99) specifies which percentile to calculate.

p50 is the median latency. p95 means 95% of requests were faster than this value. p99 captures the slowest tail. Comparing all three on one graph reveals whether your slowness is widespread or limited to outliers.

You should now see two panels on your dashboard: one showing request rate per endpoint and another showing three latency percentile lines.

Seeing 'No data' on a panel?

  • Make sure you generated traffic to the Flask app earlier. If the traffic has stopped, Prometheus needs recent data points within the 5-minute window.
  • Check the time range picker in the top-right corner of the dashboard. Set it to Last 15 minutes to see recent data.
  • Verify the data source dropdown in the query editor shows your Prometheus instance.

Help me figure out why my Grafana panel shows no data for Prometheus queries.

Add error rate and CPU usage panels, then save the dashboard

Two more panels will complete your operational dashboard. An error rate stat gives you a single number to glance at for app health. A CPU panel from Node Exporter shows infrastructure health.

  • Click the Add new element icon and drag a new panel onto the dashboard.
  • Click Configure visualization.
  • In the query editor, enter the error rate query:
sum(rate(flask_http_requests_total{status="500"}[5m])) / sum(rate(flask_http_requests_total[5m]))
  • Click Refresh.
  • Change the visualization type from Time series to Stat. Click All visualizations to see the full list and select Stat.
  • Under Standard options in the panel options, set the Unit field to Percent (0.0-1.0). This formats the decimal as a percentage.
  • Set the panel title to Error Rate.
  • Click Save.

Why use a Stat panel?

A Stat panel displays a single large number. For error rate, you want an instant-read health indicator rather than a line graph. If the number is high, something is wrong right now.

Finally, add the infrastructure panel.

  • Click the Add new element icon and drag one more panel onto the dashboard.
  • Click Configure visualization.
  • In the query editor, enter the CPU usage query:
100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
  • Click Refresh.
  • Keep the visualization type as Time series.
  • Set the panel title to CPU Usage %.
  • Click Save.

What does this CPU query do?

Node Exporter tracks CPU time spent in each mode (idle, user, system, etc.). The query calculates the rate of idle CPU seconds, averages across all cores, multiplies by 100 to get a percentage, then subtracts from 100 to flip it into a "percentage busy" metric.

Because Node Exporter is running inside a Docker container in this project, the values reflect container-reported CPU metrics rather than full host metrics.

Now save the entire dashboard.

  • Click Save in the top right corner.
  • Enter Flask Monitoring as the dashboard title.
  • Click Save.
  • Click Exit edit to return to the dashboard view.

You should now see a four-panel dashboard with request rate, latency percentiles, error rate, and CPU usage all displaying live data from your monitoring stack.

CPU panel showing 0% or no data?

  • Node Exporter running in Docker without host mounts reports limited metrics. As long as you see a value (even a low one), the panel is working correctly.
  • If you see no data at all, confirm Node Exporter is still running by checking docker compose ps in your terminal.

Help me troubleshoot my Node Exporter CPU usage panel in Grafana.

Your monitoring stack now has a visual layer. You can see request patterns, latency trends, error spikes, and infrastructure load all in one place. Next up, you will configure alerting rules so Prometheus can notify you automatically when things go wrong.

Configure Alerting Rules

Your Grafana dashboard gives you visibility into what's happening right now. But dashboards require someone to be watching them.

Alerting solves this by automatically notifying you when something goes wrong. In this step, you'll write a Prometheus alerting rule that fires when your Flask app's error rate exceeds 10%, then trigger it intentionally to watch the full alert lifecycle in action.

In this step, get ready to:
  • Write an alerting rule that detects high error rates.
  • Reload Prometheus to pick up the new rule.
  • Trigger the alert and observe it move from inactive to pending to firing.
Create the alert rules file

Prometheus evaluates alerting rules on every evaluation cycle (every 15 seconds in your configuration). Each rule contains a PromQL expression and a duration. If the expression evaluates to true for the specified duration, the alert fires and gets forwarded to Alertmanager.

  • Open prometheus/alert_rules.yml in your editor.
  • Replace the entire contents of the file with the following alert rule:
groups:
  - name: flask_app_alerts
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(flask_http_requests_total{status="500"}[5m]))
          /
          sum(rate(flask_http_requests_total[5m]))
          > 0.1
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "High error rate detected"
          description: "Flask app error rate is above 10% for more than 1 minute. Current value: {{ $value | humanizePercentage }}"

What does this rule do?

  • alert: HighErrorRate names the alert so you can identify it in the Prometheus and Alertmanager UIs.
  • expr uses the same error rate PromQL query from Step 4. It divides 500-status requests by total requests over a 5-minute window. If that ratio exceeds 0.1 (10%), the condition is true.
  • for: 1m means the condition must stay true for a full minute before the alert fires. This prevents brief spikes from triggering false alarms.
  • labels attach metadata (like severity) that Alertmanager uses for routing decisions.
  • annotations provide human-readable context. The {{ $value | humanizePercentage }} template inserts the current error rate value into the description.
  • Save prometheus/alert_rules.yml.

✔️ Awesome, I've got everything!

Great. Double check you've saved the file before moving on.

ⓧ I'd like to double check the full code

groups:
  - name: flask_app_alerts
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(flask_http_requests_total{status="500"}[5m]))
          /
          sum(rate(flask_http_requests_total[5m]))
          > 0.1
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "High error rate detected"
          description: "Flask app error rate is above 10% for more than 1 minute. Current value: {{ $value | humanizePercentage }}"
Reload Prometheus and verify the rule

Your docker-compose.yml already mounts alert_rules.yml into the Prometheus container and includes the --web.enable-lifecycle flag. This flag exposes a reload endpoint that lets you apply configuration changes without restarting the container.

  • Reload the Prometheus configuration by running this command:
curl -X POST http://localhost:9090/-/reload

What does this command do?

This sends an HTTP POST to Prometheus's management API at /-/reload. Prometheus re-reads its configuration file and all referenced rule files without restarting. If the new configuration has syntax errors, Prometheus rejects the reload and keeps running with the previous valid config.

  • Open your browser and navigate to http://localhost:9090/alerts.
  • You should see HighErrorRate listed under the flask_app_alerts group with a state of inactive (shown in green).

Don't see HighErrorRate listed?

  • Check that your YAML indentation uses spaces, not tabs. YAML is sensitive to whitespace.
  • Verify the file is saved as prometheus/alert_rules.yml (not a different filename or location).
  • Check Prometheus logs for syntax errors by running docker logs prometheus. Look for lines mentioning rule file parsing failures.

Why isn't my alert rule showing up in Prometheus after reloading?

Trigger the alert and observe the lifecycle

Prometheus alerts move through three states: inactive (condition is false), pending (condition is true but hasn't lasted long enough), and firing (condition held for the full for duration). Let's push the error rate above 10% and watch the transition.

  • Generate a burst of error traffic by running this command:
for i in $(seq 1 200); do curl -s http://localhost:5000/error > /dev/null; done

What does this command do?

This sends 200 requests to the /error endpoint in rapid succession. Each request returns a 500 status code, which the Flask app counts via the flask_http_requests_total counter with status="500". This pushes the error rate well above the 10% threshold.

  • Return to the Prometheus Alerts page at http://localhost:9090/alerts.
  • Refresh the page. Within 15-30 seconds, you should see HighErrorRate transition from inactive (green) to pending (yellow).
  • Keep refreshing. After 1 minute in pending state, the alert transitions to firing (red).

Why does it take a minute to fire?

The for: 1m clause in your rule creates this delay intentionally. In production, a brief spike in errors might resolve on its own. The for duration prevents your team from getting paged for transient issues that auto-recover.

  • Once the alert reaches firing state, open the Alertmanager UI at http://localhost:9093.
  • You should see the HighErrorRate alert displayed with its labels and annotations. This confirms the full pipeline: Prometheus detected the problem, evaluated the rule, and forwarded the alert to Alertmanager.

Alert stuck in pending or not appearing in Alertmanager?

  • If the alert stays in pending, it means the error rate dropped below 10% before the 1-minute window elapsed. Run the traffic generation command again and keep refreshing.
  • If the alert fires in Prometheus but doesn't appear in Alertmanager, check that Alertmanager is running with docker ps and verify the container named alertmanager is up.
  • Check Alertmanager's logs with docker logs alertmanager for connection errors.

My alert fired in Prometheus but I can't see it in Alertmanager. How do I debug the connection?

You've just completed the full observability loop. Your stack can now detect problems, evaluate conditions over time, and route notifications to the right place. Next up, you'll extend this with recording rules and per-endpoint alerts in the Secret Mission.

Secret mission

Recording Rules and Per-Endpoint Alerts

Your current alert fires on the global error rate. But what if only one endpoint is failing while others are healthy? In this mission, you'll add recording rules for pre-computed metrics and build granular per-endpoint alerts with intelligent inhibition rules that prevent notification storms.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely on local Docker containers, so there are no ongoing cloud costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Docker containers: prometheus, grafana, alertmanager, node-exporter, flask-app.
  • Docker volumes: prometheus-data and grafana-storage.
  • Docker images pulled and built during the project.
  • Project folder on your Desktop containing all configuration files and the Flask app.
  • Python virtual environment created inside the project folder.

Keep everything running

No action needed. Choose this if you are still actively building or experimenting with your monitoring stack.

  • Your containers will continue running and restart automatically thanks to the restart: unless-stopped policy in your Compose file.
  • Prometheus will keep scraping metrics and retaining data for 15 days.
  • Your Grafana dashboard and data source configuration persist in the grafana-storage volume.

Pause - I'll come back to this later

Shut down running containers to free up CPU and memory, but keep all your data and configuration intact for later.

  • Open your terminal and navigate to your project folder on the Desktop.
  • Stop all containers by running:
docker compose stop
  • This stops the containers without removing them or their volumes. Your Prometheus data and Grafana dashboards remain intact.
  • When you are ready to resume, navigate back to the project folder and run:
docker compose start

Delete - I don't want to use this again

Remove all containers, volumes, and project files to start completely fresh.

  • Open your terminal and navigate to your project folder on the Desktop.
  • Remove all containers and volumes by running:
docker compose down -v

What does this command do?

The -v flag removes the named volumes (prometheus-data and grafana-storage) along with the containers and networks. Without it, your stored metrics and Grafana configuration would persist even after the containers are gone.

  • Remove the Docker images downloaded for this project by running:
docker image prune -a
  • Type y when prompted to confirm. This removes all unused images from your system.
  • Navigate up one level from your project folder by running:
cd ~/Desktop
  • Delete the project folder and all its contents by running:
rm -rf monitoring-stack
  • On Windows, use this command instead:
Remove-Item -Recurse -Force monitoring-stack
  • Confirm the folder is gone by running:
ls
  • You should no longer see monitoring-stack in the output.

That's a Wrap!

That's a Wrap!

Nice work! You just built a production-grade monitoring stack from scratch using Prometheus, Grafana, Alertmanager, and a custom-instrumented Flask application.

You've learned how to:

  • Deployed Prometheus, Grafana, Alertmanager, and Node Exporter as a multi-service stack using Docker Compose.
  • Instrumented a Flask application with Counter, Histogram, and Gauge metrics, then wrote PromQL queries including rate(), histogram_quantile(), and aggregations to answer real operational questions.
  • Built a multi-panel Grafana dashboard visualizing request rate, latency percentiles, and error rate, then configured alerting rules and verified the full alert lifecycle from inactive to pending to firing.
  • Created recording rules for pre-computed metrics and built per-endpoint alerting with inhibition rules to prevent duplicate notifications.

Ready to quiz yourself?