SRE Monitoring with Prometheus and Grafana

Deploy a live monitoring stack and watch alerts fire when your app breaks.

Introduction

30 Second Summary

When a website goes down, the best teams know about it before their users do. They have dashboards showing real-time health data and alerts that fire the moment something breaks.

In this project, you will build a complete monitoring setup on your own machine using Prometheus and Grafana, the same open-source tools used by SRE teams worldwide. You will deploy an app, monitor it, break it on purpose, and watch an alert fire.

What You'll Build

You will open a browser tab to a Grafana dashboard, see live graphs updating every few seconds as traffic flows through your app, then intentionally crash the app and watch the metrics react in real time.

By the end of this project, you'll have:

A live Grafana dashboard that updates every 5 seconds, showing request rates and errors as colorful real-time graphs.
A Python Flask app running in Docker that generates Prometheus metrics you can see by visiting a /metrics endpoint in your browser.
A designed failure scenario where you crash the app on purpose and watch a Prometheus alerting rule fire automatically, proving your monitoring catches outages before users do.
Secret Mission: Add a response time panel and configure a Grafana alert that sends a notification when latency exceeds a threshold.

Do I need any prior experience?

No prerequisites are needed. This guide walks you through installing every tool from scratch, including Docker Desktop and VS Code. All you need is a Windows PC with 8 GB of RAM.

Before We Start

Before We Start...

Before we dive in, let's lock in what you're about to build and why it matters. This step is your moment to commit to the goal before any hands-on work begins.

Install Docker Desktop and VS Code

To build your monitoring stack, you need a way to run multiple services side by side without installing each one individually on your machine. Prometheus, Grafana, and your Python app each need their own environment, and managing that manually gets complicated fast.

Docker solves this by packaging each service into a container that runs in isolation. In this step, you'll install Docker Desktop and VS Code so your machine is ready to build and run the project.

In this step, get ready to:

Install VS Code for editing your project files.
Install Docker Desktop for running containers.
Verify Docker is working from the terminal.

Download and install VS Code

VS Code is a free code editor you'll use to write and manage all your project files.

Open your browser and go to https://code.visualstudio.com/.
Click the large Download for Windows button.
Once the installer downloads, double-click it to run.
Accept the license agreement and click Next through each screen, keeping the defaults.
Click Install.
Click Finish when the installation completes.

Press the Windows key to open your search bar.
Type Visual Studio Code and press Enter to open it.

You should see the VS Code Welcome tab appear. This confirms the installation worked.

VS Code not showing up in search?

Try searching for just Code instead. If it still does not appear, restart your computer so Windows registers the new application in search, then try again.

Help me fix VS Code not appearing in Windows search

Download and install Docker Desktop

Docker Desktop is the application that lets you run containers on Windows. It uses a technology called WSL 2 (Windows Subsystem for Linux) to run a lightweight Linux environment behind the scenes.

Open your browser and go to https://docs.docker.com/desktop/setup/install/windows-install/.
Click Docker Desktop for Windows - x86_64 to download the installer.
Double-click the downloaded Docker Desktop Installer.exe to start installation.

What will the installer ask?

The installer will show a configuration screen with a checkbox for Use WSL 2 instead of Hyper-V. Keep this checked (it should be selected by default). WSL 2 is the recommended backend and works on Windows Home, Pro, and Enterprise editions.

Keep the default WSL 2 option selected.
Click Ok to begin installation.
When the installer finishes, it will prompt you to restart your computer.
Save any open work, then restart.

After your computer restarts, Docker Desktop may launch automatically. If it does not, open it manually:

Press the Windows key to open your search bar.
Type Docker Desktop and press Enter to open it.
Accept the Docker service agreement if prompted.
Wait for Docker Desktop to finish starting up. You'll see a green status indicator in the bottom-left corner of the Docker Desktop window when the engine is ready.

What is Docker?

Docker lets you run applications inside containers. A container is like a lightweight, isolated mini-computer that has everything an application needs to run.

Instead of installing Prometheus, Grafana, and Python directly on your machine (and dealing with version conflicts), you'll run each one in its own container. Docker Desktop provides the engine that makes this possible on Windows.

Docker Desktop stuck on starting?

If Docker Desktop seems stuck or shows an error about WSL, open a terminal and run:

wsl --update

Then restart Docker Desktop. If you see a message about enabling virtualization, you'll need to enable it in your computer's BIOS settings (the steps vary by manufacturer).

Help me fix Docker Desktop not starting

Verify Docker in your VS Code terminal

Now that both tools are installed, you'll confirm Docker is accessible from VS Code's built-in terminal. This is where you'll run all your project commands going forward.

Switch back to VS Code.
Click Terminal in the top menu bar.
Click New Terminal.

Before you run the next command, what do you think you'll see if Docker installed correctly?

Check your Docker installation by running this command:

docker --version

✔️ I see a version number

You should see output like Docker version 27.x.x (your exact numbers may differ). This confirms Docker is installed and accessible from your terminal.

You're all set to start building.

ⓧ Command not found or error

If you see an error or the command is not recognized, try these fixes:

Make sure Docker Desktop is running (check for the Docker whale icon in your system tray at the bottom-right of your screen).
Close your VS Code terminal and open a new one by clicking Terminal then New Terminal. The terminal needs to reload after Docker is installed.
If it still does not work, restart your computer one more time and try again.

Help me fix the docker command not being recognized

Your machine is ready. Next up, you'll create your Python web app and get it running inside a Docker container.

Build and Run Your Python App

Your development environment is set up with Docker Desktop and VS Code ready to go. Now you need something to actually monitor.

Before you can track metrics or build dashboards, you need a running application that generates data. In this step, you'll build a small Python web app using Flask and run it inside a Docker container.

In this step, get ready to:

Create the sre-monitoring project folder and open it in VS Code.
Write a Flask web app instrumented with Prometheus metrics.
Package and run the app in a Docker container using Docker Compose.

Create the project folder and Flask app

Every project needs a home. You'll create a dedicated folder on your Desktop so the project is easy to find later.

Move to your Desktop by running this command in your VS Code terminal:

cd ~/Desktop

Create your project folder and move into it by running these commands:

mkdir sre-monitoring
cd sre-monitoring

Open this folder in VS Code by clicking File in the top menu bar.
Click Open Folder.
Select the sre-monitoring folder on your Desktop, then click Select Folder.

VS Code reloads with your empty project folder in the Explorer sidebar on the left.

Don't see the Explorer sidebar?

Press Ctrl+Shift+E to toggle the Explorer panel. You should see SRE-MONITORING as the folder name at the top.

Still stuck? Help me open my project folder in VS Code.

Now you'll create the Flask application file. Flask is a lightweight Python web framework that lets you create web servers in just a few lines of code.

In VS Code's Explorer sidebar, right-click in the empty area and select New File.
Name the file app.py and press Enter.
Paste the following code into app.py:

import time
from flask import Flask
from prometheus_flask_exporter import PrometheusMetrics

# Create the Flask app and attach Prometheus metrics
app = Flask(__name__)
metrics = PrometheusMetrics(app)

# Register app metadata as a Prometheus gauge
metrics.info('app_info', 'Application info', version='1.0.0')

# Homepage route
@app.route('/')
def hello():
    return 'Hello, SRE!'

# Health check endpoint
@app.route('/healthy')
def healthy():
    return 'OK', 200

# Start the server on all network interfaces, port 5000
if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)

What does this code do?

PrometheusMetrics(app) automatically instruments your Flask app so that every request is tracked. It also exposes a /metrics endpoint where Prometheus can read the collected data.
The / route returns a simple greeting. The /healthy route is a health check that returns OK with a 200 status code.
host='0.0.0.0' tells Flask to listen on all network interfaces, which is required for the app to be reachable from outside its Docker container.

Save app.py by pressing Ctrl+S.
Confirm the file appears in VS Code's Explorer sidebar on the left.

Don't see app.py in the sidebar?

Make sure you created the file inside the sre-monitoring folder and not somewhere else. If the Explorer shows a different folder name at the top, go back to File > Open Folder and select sre-monitoring from your Desktop.

Need more help? Help me create app.py in the right location.

Package the app for Docker

Docker needs two files to build your app into a container image: a list of Python packages to install, and a Dockerfile that describes how to assemble the image.

Create a new file called requirements.txt (right-click in Explorer > New File).
Paste these two dependencies into requirements.txt:

flask==3.1.3
prometheus-flask-exporter==0.23.2

Save requirements.txt.
Create another new file called Dockerfile (no file extension).
Paste the following into your Dockerfile:

# Use a lightweight Python image
FROM python:3.12-slim

# Set the working directory inside the container
WORKDIR /app

# Install Python dependencies first (for Docker layer caching)
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy the application code
COPY app.py .

# Document the port the app listens on
EXPOSE 5000

# Start the Flask app
CMD ["python", "app.py"]

What does this Dockerfile do?

FROM python:3.12-slim starts from a small, pre-built Python image so you don't have to install Python yourself.
COPY requirements.txt and RUN pip install install Flask and the Prometheus exporter inside the container.
CMD ["python", "app.py"] is the command that runs when the container starts.

Save the Dockerfile.
Confirm all your files exist by running this in the terminal:

ls

You should see app.py, Dockerfile, and requirements.txt listed in the output.

Missing a file in the output?

Make sure your terminal is inside the sre-monitoring folder. If you opened a new terminal after reloading VS Code, it should default to your project folder. If not, run cd ~/Desktop/sre-monitoring to get back.

Need help? Help me find my project files in the terminal.

Run the app with Docker Compose

Docker Compose lets you define and run multi-container apps from a single configuration file. You'll start with just one service (your web app) and add more in later steps.

Create a new file called compose.yaml in your project folder.
Paste the following into compose.yaml:

services:
  webapp:
    build: .  # Build the Docker image from the Dockerfile in this folder
    ports:
      - "5000:5000"  # Map host port 5000 to container port 5000

What does this compose file do?

build: . tells Docker Compose to build an image using the Dockerfile in the current folder.
ports: "5000:5000" maps port 5000 on your machine to port 5000 inside the container, so you can visit the app in your browser.

Save compose.yaml.
Build and start the app by running this command:

docker compose up --build -d

What does this command do?

docker compose up starts all services defined in your compose.yaml.
--build forces Docker to build a fresh image from your Dockerfile.
-d runs the container in detached mode so your terminal stays free.

Docker will download the Python base image, install your dependencies, and start the app. This may take 1-2 minutes on the first run.

Before you check the result, what do you think you'll see if you visit http://localhost:5000 in your browser?

Open your browser and go to http://localhost:5000.

You should see the text Hello, SRE! displayed on the page. Your Flask app is live and serving traffic from inside a Docker container.

Seeing "This site can't be reached"?

The container might still be starting up. Run the following to check its status:

docker compose ps

If the STATE column shows running, wait a few seconds and refresh. If it shows exited, check the logs:

docker compose logs webapp

Look for error messages about missing imports or syntax errors. Double-check that your app.py and requirements.txt match the code above exactly.

Still stuck? Help me debug why my Docker container isn't running.

✔️ Awesome, I've got everything!

Great work. Double check you've saved all four files in your sre-monitoring folder.

ⓧ I'd like to double check the full code

Here are all four files. Make sure yours match exactly.

import time
from flask import Flask
from prometheus_flask_exporter import PrometheusMetrics

app = Flask(__name__)
metrics = PrometheusMetrics(app)

metrics.info('app_info', 'Application info', version='1.0.0')

@app.route('/')
def hello():
    return 'Hello, SRE!'

@app.route('/healthy')
def healthy():
    return 'OK', 200

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)

flask==3.1.3
prometheus-flask-exporter==0.23.2

FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

EXPOSE 5000

CMD ["python", "app.py"]

services:
  webapp:
    build: .
    ports:
      - "5000:5000"

Your app is live and serving traffic. Next up, you'll connect Prometheus to scrape those metrics and store them as time-series data.

Add Prometheus and See Metrics

Your Flask app is running and responding to requests. But right now, if it started failing or slowing down, you would have no way to know until users complained.

Prometheus is a time-series database that collects and stores metrics from applications. In this step, you will wire it up to scrape your app's /metrics endpoint every 5 seconds, giving you real-time visibility into how your app is performing.

In this step, get ready to:

Create a Prometheus configuration file that targets your Flask app.
Add a Prometheus container to your Docker Compose stack.
Query live metrics in the Prometheus web UI.

Create the Prometheus configuration file

Prometheus needs a configuration file to know which applications to monitor and how often to check them. You will create this file in your project folder.

Create a new file called prometheus.yml in your sre-monitoring folder by running this command in your VS Code terminal:

New-Item prometheus.yml

Open prometheus.yml in the VS Code editor and paste the following configuration:

global:
  scrape_interval: 5s
  evaluation_interval: 5s

rule_files:
  - "alert_rules.yml"

scrape_configs:
  - job_name: 'webapp'
    static_configs:
      - targets: ['webapp:5000']

What does this configuration do?

scrape_interval: 5s tells Prometheus to pull metrics from your app every 5 seconds. The default is 15 seconds, but 5 seconds gives you faster feedback while learning.
rule_files prepares Prometheus for alerting rules you will create in a later step. Prometheus ignores this entry for now since the file does not exist yet.
job_name: 'webapp' groups your app's metrics under the label "webapp" so you can identify them easily.
targets: ['webapp:5000'] tells Prometheus where to find your app. Inside Docker Compose, containers reach each other by service name, so webapp resolves to your Flask container on port 5000.

Save prometheus.yml.
Confirm the file exists by running:

ls

You should see prometheus.yml listed alongside app.py, Dockerfile, compose.yaml, and requirements.txt.

Don't see prometheus.yml in the list?

Make sure you created the file inside the sre-monitoring folder and not a subfolder.
Check for typos in the filename. It must be prometheus.yml (not .yaml or .ymal).

Still stuck? Help me figure out why my prometheus.yml file isn't showing up in my project folder.

✔️ Awesome, I've got everything!

Great. Double check you have saved prometheus.yml before moving on.

ⓧ I'd like to double check the full code

global:
  scrape_interval: 5s
  evaluation_interval: 5s

rule_files:
  - "alert_rules.yml"

scrape_configs:
  - job_name: 'webapp'
    static_configs:
      - targets: ['webapp:5000']

Add Prometheus to your Docker Compose file

Now you need to tell Docker Compose to run a Prometheus container alongside your Flask app. This container will use the official Prometheus image and mount the configuration file you just created.

Open compose.yaml in VS Code.
Find the end of the webapp service (after the line - "5000:5000").
Add the following Prometheus service below the webapp service:

  prometheus:
    image: prom/prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    depends_on:
      - webapp

What does this service definition do?

image: prom/prometheus pulls the official Prometheus Docker image. No local build needed.
ports: - "9090:9090" exposes the Prometheus web UI on your machine at port 9090.
volumes mounts your local prometheus.yml into the container at /etc/prometheus/prometheus.yml, which is where Prometheus looks for its config by default.
depends_on: - webapp ensures the Flask app starts before Prometheus, so there is something to scrape right away.

Save compose.yaml.

Getting YAML indentation errors later?

The prometheus key must be indented at the same level as webapp (two spaces in from services).
Make sure there is a blank line between the webapp service and the prometheus service for readability.
YAML uses spaces, not tabs. If you mixed them, VS Code will show a red squiggle.

Need help? Help me fix my compose.yaml indentation so the Prometheus service is recognized.

✔️ Awesome, I've got everything!

Great. Double check you have saved compose.yaml before moving on.

ⓧ I'd like to double check the full code

services:
  webapp:
    build: .
    ports:
      - "5000:5000"

  prometheus:
    image: prom/prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    depends_on:
      - webapp

Start Prometheus and query your first metrics

With the configuration in place, you can now launch Prometheus alongside your app and see real metrics flowing in.

Start the updated stack by running this command:

docker compose up -d

Docker Compose will pull the Prometheus image (this may take a minute the first time) and start both containers.

Confirm both containers are running:

docker compose ps

You should see two services listed with a status of "Up": webapp and prometheus.

Before you check the targets page, think about this: do you think Prometheus has already connected to your Flask app? What would it need to succeed?

Open http://localhost:9090/targets in your browser.

You should see your webapp listed with State = "UP" in green. This means Prometheus successfully reached your Flask app's /metrics endpoint and is actively scraping data every 5 seconds.

What does UP mean on the targets page?

The Prometheus targets page shows the health status of every application it monitors. A green "UP" means the last scrape succeeded. A red "DOWN" means Prometheus could not reach the target. Since your Flask app and Prometheus are on the same Docker Compose network, they can reach each other by service name.

Now query the metrics. Open http://localhost:9090 in your browser to access the Prometheus expression browser.
Type flask_http_request_total into the query box at the top.
Click Execute.

You should see the metric appear in the results table. It may show a value of 0 or a small number if no requests have been made to your app recently.

Open a new browser tab and visit http://localhost:5000 several times to generate some traffic.
Switch back to the Prometheus tab and click Execute again.

The counter value for flask_http_request_total increases each time you visit your app. This proves Prometheus is collecting and storing your app's metrics as time-series data.

Targets page shows DOWN instead of UP?

Make sure your webapp container is still running. Run docker compose ps and check that both services show "Up".
Verify the target in prometheus.yml is exactly webapp:5000 (not localhost:5000). Docker Compose containers use service names, not localhost.
If you just started the containers, wait 10 seconds and refresh. Prometheus needs at least one scrape interval (5 seconds) to attempt the first connection.

Still stuck? Help me figure out why Prometheus shows my webapp target as DOWN.

Your monitoring stack is taking shape. Prometheus is now collecting metrics from your Flask app every 5 seconds and storing them as time-series data. Next up, you will build a Grafana dashboard that turns these raw numbers into visual graphs that update in real time.

Build a Grafana Dashboard

Prometheus is collecting metrics from your Flask app every 5 seconds. You can query those numbers manually, but staring at raw counters and text tables is not how real SRE teams monitor production.

Grafana turns those raw metrics into live visual graphs that update in real time. In this step, you will add Grafana to your stack, connect it to Prometheus, and build a dashboard that shows your app's request rate at a glance.

In this step, get ready to:

Add Grafana to your Docker Compose stack and sign in.
Connect Prometheus as a Grafana data source.
Build a real-time Request Rate dashboard.

Add Grafana to Docker Compose

Grafana needs to run alongside Prometheus and your web app. You will add it as a new service in your compose.yaml file.

In VS Code, open compose.yaml and add the following Grafana service below the prometheus service:

  grafana:
    image: grafana/grafana-enterprise
    ports:
      - "3000:3000"
    depends_on:
      - prometheus

What does this code do?

grafana/grafana-enterprise is the official Grafana Docker image. It includes all open-source features and is free to use.
Port 3000:3000 maps Grafana's web UI to your browser at http://localhost:3000.
depends_on: prometheus ensures Prometheus starts before Grafana so the data source is ready.

Save compose.yaml.
Start the updated stack by running this command in your terminal:

docker compose up -d

Docker will pull the Grafana image and start it alongside your existing containers.

Open http://localhost:3000 in your browser.
Log in with username admin and password admin.
Grafana will prompt you to change the password. Click Skip for now.

You should see the Grafana home page with a welcome message and navigation menu on the left.

Not seeing the login page?

Make sure Docker Desktop is running and shows "Engine running" in the bottom-left corner.
Run docker compose ps in your terminal to check all three containers (webapp, prometheus, grafana) are listed with status "Up".
If the grafana container shows "Exited", check for port conflicts. Another app may already be using port 3000.

Help me troubleshoot why Grafana is not starting in my Docker Compose setup.

Add Prometheus as a data source

Grafana does not know where your metrics live yet. You need to point it at Prometheus so it can query the time-series data your app is generating.

In Grafana, click Connections in the left-side menu.
Click Add new connection.
Type Prometheus in the search bar and select the Prometheus data source.
Click Add new data source in the upper right.
In the Connection section, find the Prometheus server URL field and enter:

http://prometheus:9090

Why not localhost?

Grafana runs inside its own Docker container. From Grafana's perspective, localhost means the Grafana container itself, not your computer. The Docker Compose service name prometheus resolves to the Prometheus container's internal IP address.

Scroll to the bottom and click Save & test.

You should see a green banner confirming the connection worked.

Connection test failed?

Double-check the URL is exactly http://prometheus:9090 with no trailing slash.
Make sure your Prometheus container is running. Run docker compose ps and confirm prometheus shows status "Up".
Verify you did not accidentally rename the prometheus service in compose.yaml. The URL must match the service name exactly.

Help me fix the Prometheus data source connection error in Grafana.

Create a Request Rate panel

With the data source connected, you can now build a dashboard panel that visualizes how many requests per second your Flask app is handling.

Click Dashboards in the left-side menu.
Click New and select New Dashboard.
Click + Add visualization.
In the dialog that opens, select your Prometheus data source.

The panel editor opens with a query area at the bottom and a visualization preview at the top.

In the query area, switch to the Code view (if not already selected) so you can type raw PromQL.
Enter this query in the metrics field:

rate(flask_http_request_total[1m])

What does this query do?

rate() calculates the per-second average rate of increase over a time window. flask_http_request_total is the counter metric your Flask app exposes (thanks to prometheus-flask-exporter).

The [1m] window means Prometheus averages the rate over the last 1 minute of data points. This smooths out spikes and gives you a readable line on the graph.

Click Refresh (or press Shift+Enter) to run the query.
Under Panel options on the right side, change the title to Request Rate.
Click Save in the top-right corner of the panel editor.
In the save dialog, enter SRE Monitoring as the dashboard title.
Click Save.

Your dashboard is now live. Before you confirm it is working, generate some traffic so the graph has data to show.

Open a new browser tab and visit http://localhost:5000.
Refresh that page 5-10 times to generate requests.
Switch back to your Grafana dashboard tab.

What do you expect to see on the graph after generating those requests?

Within a few seconds, the Request Rate panel should show a line spike upward as Prometheus picks up your new requests. The graph updates automatically as fresh metrics arrive.

Graph shows no data?

Make sure you visited http://localhost:5000 at least a few times. Without traffic, the rate is zero and the graph may appear empty.
Check the time range in the top-right corner of the dashboard. Set it to Last 5 minutes so recent data is visible.
Confirm your data source test passed earlier. If not, revisit the Prometheus URL setting.

Help me understand why my Grafana Request Rate panel shows no data.

✔️ Awesome, I've got everything!

Great. Double check you have saved your compose.yaml file and your Grafana dashboard.

ⓧ I'd like to double check the full code

Here is the complete compose.yaml at this point in the project:

services:
  webapp:
    build: .
    ports:
      - "5000:5000"

  prometheus:
    image: prom/prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    depends_on:
      - webapp

  grafana:
    image: grafana/grafana-enterprise
    ports:
      - "3000:3000"
    depends_on:
      - prometheus

All other files (app.py, requirements.txt, Dockerfile, prometheus.yml) remain unchanged from the previous step.

You now have a complete monitoring dashboard showing live request data. Next up, you will intentionally break the app and watch the metrics react in real time.

Break the App and Watch It Burn

Your Grafana dashboard is live and updating with real-time request data. That is great for watching healthy traffic, but the real power of monitoring shows up when things go wrong.

In this step, you will add a Prometheus alerting rule that automatically detects when your app goes down. Then you will intentionally kill the app and watch your entire monitoring stack react in real time.

In this step, get ready to:

Add a Prometheus alerting rule that fires when the app is unreachable for 30 seconds.
Stop the app container and observe the request rate drop to zero on your dashboard.
Watch the alert transition to "firing" state and then bring the app back to life.

Create an alerting rule for Prometheus

Prometheus can evaluate expressions on a schedule and fire alerts when a condition is met. You will create a rule that says "if the app has been unreachable for 30 seconds, something is wrong."

In VS Code, create a new file called alert_rules.yml inside your sre-monitoring folder.
Add the following alerting rule configuration:

groups:
  - name: app_alerts
    rules:
      - alert: AppDown
        expr: up{job="webapp"} == 0
        for: 30s
        labels:
          severity: critical
        annotations:
          summary: "Web application is down"
          description: "The webapp target has been unreachable for more than 30 seconds."

What does this rule do?

The expression up{job="webapp"} == 0 checks whether Prometheus can reach your Flask app. When a scrape target is unreachable, the up metric drops to 0.

The for: 30s line means the condition must be true for 30 continuous seconds before the alert fires. This prevents false alarms from brief network blips.

Save alert_rules.yml.

Your prometheus.yml already references this file in its rule_files section from the previous step. Now you need to mount the file into the Prometheus container so it can actually read it.

In compose.yaml, find the volumes section under the prometheus service. It currently looks like this:

    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml

Add a second volume mount line directly below it, so the section now looks like this:

    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./alert_rules.yml:/etc/prometheus/alert_rules.yml

Save compose.yaml.

Apply the changes and confirm the rule loads

Apply the updated configuration by running this command in your VS Code terminal:

docker compose up -d

Prometheus will restart and pick up the new alert rule file.

Open http://localhost:9090/rules in your browser.
Confirm you see AppDown listed under the app_alerts group.

Don't see AppDown listed?

Check that alert_rules.yml is saved in the same folder as compose.yaml (not inside a subfolder).
Verify the volume mount path in compose.yaml matches exactly: ./alert_rules.yml:/etc/prometheus/alert_rules.yml.
Make sure there are no indentation errors in your YAML files. YAML is whitespace-sensitive.

help me debug why my Prometheus alert rule isn't loading

Now generate some traffic so your dashboard shows activity. Open http://localhost:5000 in your browser and refresh it 5-10 times.
Switch to your Grafana dashboard at http://localhost:3000 and confirm the Request Rate graph is showing the traffic you just generated.

Stop the app and watch the alert fire

Everything is healthy right now. Your dashboard shows traffic flowing and Prometheus is happily scraping metrics. Time to break something on purpose.

Before you run the next command, think about what you expect to happen. If you stop the webapp container, what will happen to the Request Rate graph on your Grafana dashboard? What will Prometheus do after 30 seconds of not being able to reach the app?

Stop the webapp container by running:

docker compose stop webapp

Switch to your Grafana dashboard and watch the Request Rate panel for 30-60 seconds.

The request rate drops to zero. No traffic is flowing because the app is gone. This is exactly what an SRE would see during a real outage.

Now open http://localhost:9090/alerts in your browser.
Wait 30 seconds if needed. You should see the AppDown alert transition from "pending" to "firing" state (shown in red).

What do the alert states mean?

Prometheus alerts move through states. Inactive means the condition is not met. Pending means the condition is met but the for duration has not elapsed yet.

Firing means the condition has been true for the full for duration. In your case, the app has been unreachable for at least 30 seconds.

You just completed the core SRE loop: deploy an app, monitor it, detect a failure automatically. In a real production environment, this firing alert would notify an on-call engineer before users even notice the outage.

Bring the app back to life by running:

docker compose start webapp

Switch back to your Grafana dashboard and refresh http://localhost:5000 a few times.
Watch the Request Rate graph climb back up as traffic resumes.
Check http://localhost:9090/alerts again. The AppDown alert should return to inactive state.

Alert still showing as firing after restarting?

Give it 30-60 seconds. Prometheus evaluates rules every 5 seconds, but the alert needs several successful scrapes before it resolves.
Make sure the app is actually running. Visit http://localhost:5000 and confirm you see "Hello, SRE!" in your browser.

help me troubleshoot why my Prometheus alert is still firing

✔️ Awesome, I've got everything!

Great. Double check that you have saved both alert_rules.yml and compose.yaml.

ⓧ I'd like to double check the full code

Here is the full alert_rules.yml file:

groups:
  - name: app_alerts
    rules:
      - alert: AppDown
        expr: up{job="webapp"} == 0
        for: 30s
        labels:
          severity: critical
        annotations:
          summary: "Web application is down"
          description: "The webapp target has been unreachable for more than 30 seconds."

Here is the full compose.yaml file:

services:
  webapp:
    build: .
    ports:
      - "5000:5000"

  prometheus:
    image: prom/prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./alert_rules.yml:/etc/prometheus/alert_rules.yml
    depends_on:
      - webapp

  grafana:
    image: grafana/grafana-enterprise
    ports:
      - "3000:3000"
    depends_on:
      - prometheus

Secret mission

Add Latency Monitoring with a Grafana Alert

Request rate tells you if traffic is flowing, but response time tells you if users are suffering. Real SREs monitor latency and get alerted before users notice degradation. In this secret mission, you will add a slow endpoint to your app, build a response time panel, and configure a Grafana alert that fires when latency exceeds a threshold.

Clean Up Your Resources

This project runs entirely locally using Docker containers, so there are no ongoing cloud costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

Docker containers (webapp, prometheus, grafana) running on your machine.
Docker images (python:3.12-slim, prom/prometheus, grafana/grafana-enterprise, and your custom webapp image) stored locally.
The sre-monitoring project folder on your Desktop.

Keep everything running

No action needed. Choose this if you are still actively building or experimenting with your monitoring stack.

Leave Docker Desktop running and your containers stay live.
Your Grafana dashboard remains accessible at http://localhost:3000.
Prometheus keeps scraping metrics at http://localhost:9090.
Your Flask app stays available at http://localhost:5000.

Pause - I'll come back to this later

Stop all containers to free up memory and CPU without deleting anything. You can resume later with a single command.

Open a terminal in VS Code and make sure you are in the sre-monitoring folder.
Stop all containers by running:

docker compose stop

When you are ready to resume, run:

docker compose start

Your containers, images, and data all remain intact.

Delete - I don't want to use this again

Remove all project containers, networks, volumes, and images. This frees up disk space completely.

Open a terminal in VS Code and make sure you are in the sre-monitoring folder.
Remove all containers, networks, volumes, and images by running:

docker compose down -v --rmi all

Delete the project folder by running:

cd ~/Desktop
rmdir /s /q sre-monitoring

macOS or Linux users

Use this command instead to delete the project folder:

cd ~/Desktop
rm -rf sre-monitoring

All Docker resources and local files are now removed from your machine.

That's a Wrap!

Nice work! You just built a complete SRE monitoring setup from scratch, deployed it with Docker Compose, and proved you can detect failures before users even notice.

Here's what you accomplished:

Built a Python Flask app instrumented with Prometheus metrics, and deployed it alongside Prometheus and Grafana using Docker Compose.
Created a live Grafana dashboard showing real-time request rate data updating every 5 seconds.
Completed the SRE observability loop: deployed an app, monitored it, intentionally broke it, and watched a Prometheus alerting rule fire to detect the failure automatically.
Secret Mission: Added latency monitoring with an Average Response Time panel and configured a Grafana threshold alert that fires when response time exceeds 1 second.

Ready to quiz yourself?