Monitor Jenkins Pipelines Like an SRE

Build a Jenkins observability stack with Prometheus and Grafana dashboards.

Introduction

30 Second Summary

When a CI/CD pipeline breaks at 2am, most teams find out from angry messages. SREs find out from their monitoring dashboards minutes before anyone notices.

In this project, you will build a complete Jenkins monitoring stack using Prometheus and Grafana that detects pipeline failures, queue saturation, and memory pressure before they cause outages.

What You'll Build

A local observability stack running Jenkins, Prometheus, and Grafana in Docker containers, connected and configured the way SREs monitor production CI/CD systems.

By the end of this project, you'll have:

  • A Docker Compose stack that launches Jenkins, Prometheus, and Grafana as a connected monitoring system.
  • A Grafana dashboard displaying real-time pipeline health metrics including build success rate, queue depth, executor utilisation, and JVM memory.
  • Prometheus alerting rules that catch common Jenkins failures before they become full outages.
  • Secret Mission: Track P95 build duration and create an SLO breach alert that fires when builds exceed your performance target.

Are there any prerequisites?

A basic understanding of CI/CD pipelines and Docker containers is helpful. You do not need prior experience with Prometheus, Grafana, or Jenkins configuration.

Set Up Docker and Project Structure

Before you can monitor Jenkins pipelines, you need a clean project folder that holds all the configuration files for your monitoring stack.

In this step, you will create the project directory, set up placeholder files for each service configuration, and confirm that Docker can reach the internet to pull images.

In this step, get ready to:
  • Create the project directory with all configuration file placeholders.
  • Pull a test Docker image to confirm networking works.
Create the project directory and placeholder files

Your monitoring stack will use five configuration files. Creating them now as empty placeholders means you can focus on one file at a time in the steps ahead.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir jenkins-monitoring
cd jenkins-monitoring
  • Create the five placeholder configuration files by running this command:
touch docker-compose.yml prometheus.yml alert_rules.yml Jenkinsfile Jenkinsfile-failing

What are these files for?

  • docker-compose.yml defines the Jenkins, Prometheus, and Grafana containers and how they connect to each other.
  • prometheus.yml tells Prometheus where to find Jenkins metrics.
  • alert_rules.yml defines the alerting thresholds that detect pipeline problems.
  • Jenkinsfile and Jenkinsfile-failing are sample pipelines you will create to generate metrics.
  • Confirm the files exist by running:
ls

You should see docker-compose.yml, prometheus.yml, alert_rules.yml, Jenkinsfile, and Jenkinsfile-failing listed in the output.

Don't see all five files?

Make sure you ran the touch command from inside the jenkins-monitoring folder. Run pwd to check your current directory.

If you see fewer than five files, re-run the touch command. It is safe to run multiple times.

Still stuck? Help me figure out why my files weren't created in the jenkins-monitoring folder.

Verify Docker networking with a test pull

Docker needs internet access to download container images. A quick pull test confirms everything is connected before you bring up the full monitoring stack.

  • Pull the official test image by running this command:
docker pull hello-world

You should see Docker download the image layers, finishing with a line that reads Status: Downloaded newer image for hello-world:latest.

Seeing a connection error?

Confirm Docker Desktop is running. Look for the Docker whale icon in your system tray (Windows) or menu bar (macOS). If it shows as stopped, start it and wait for the status to show "Engine running".

If Docker is running but you see a timeout, check your internet connection and try again.

Need more help? Help me troubleshoot why docker pull hello-world is failing.

Your project skeleton is in place and Docker is confirmed working. Next up, you will fill in docker-compose.yml and prometheus.yml to bring the full monitoring stack online.

Launch the Monitoring Stack with Docker Compose

Your project structure is in place. Now it's time to bring the monitoring stack to life.

In this step, you'll define three services in a single Docker Compose file: Jenkins (your CI/CD server), Prometheus (your metrics collector), and Grafana (your dashboard tool). You'll also configure Prometheus to know where Jenkins lives so it can start scraping metrics.

In this step, get ready to:
  • Write the Docker Compose configuration for all three services.
  • Configure Prometheus to scrape Jenkins metrics.
  • Launch the stack and verify all services are accessible.
Write the Docker Compose configuration

Docker Compose lets you define multiple containers in a single YAML file and launch them together with one command. Your three services need to communicate with each other over a shared network.

  • Open docker-compose.yml in your text editor and paste the following configuration:
services:
  jenkins:
    image: jenkins/jenkins:2.555.2-lts-jdk21
    container_name: jenkins
    restart: unless-stopped
    ports:
      - "8080:8080"
      - "50000:50000"
    volumes:
      - jenkins_home:/var/jenkins_home
    networks:
      - monitoring

  prometheus:
    image: prom/prometheus:v3.11.3
    container_name: prometheus
    restart: unless-stopped
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./alert_rules.yml:/etc/prometheus/alert_rules.yml
      - prometheus_data:/prometheus
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--web.enable-lifecycle"
    networks:
      - monitoring

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: unless-stopped
    ports:
      - "3000:3000"
    volumes:
      - grafana_data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    networks:
      - monitoring

volumes:
  jenkins_home:
  prometheus_data:
  grafana_data:

networks:
  monitoring:
    driver: bridge

What does this configuration do?

This file defines three services that run as containers on your machine:

  • jenkins uses the official Jenkins LTS image (2.555.2 with Java 21 bundled). Port 8080 is the web UI. Port 50000 is for build agents.
  • prometheus mounts your local prometheus.yml and alert_rules.yml into the container. The --web.enable-lifecycle flag lets you reload its config without restarting the container.
  • grafana runs the dashboard UI on port 3000. The environment variable sets the admin password to admin.

All three services join a shared monitoring network, which means they can reach each other by container name (e.g., Prometheus can call jenkins:8080).

  • Save docker-compose.yml.

Seeing a YAML syntax error later?

YAML is strict about indentation. Every level uses exactly 2 spaces. Make sure you have no tabs and no trailing whitespace.

Help me fix a YAML indentation issue in my docker-compose.yml

✔️ Awesome, I've got everything!

Great. Make sure the file is saved before moving on.

ⓧ I'd like to double check the full code

services:
  jenkins:
    image: jenkins/jenkins:2.555.2-lts-jdk21
    container_name: jenkins
    restart: unless-stopped
    ports:
      - "8080:8080"
      - "50000:50000"
    volumes:
      - jenkins_home:/var/jenkins_home
    networks:
      - monitoring

  prometheus:
    image: prom/prometheus:v3.11.3
    container_name: prometheus
    restart: unless-stopped
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./alert_rules.yml:/etc/prometheus/alert_rules.yml
      - prometheus_data:/prometheus
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--web.enable-lifecycle"
    networks:
      - monitoring

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: unless-stopped
    ports:
      - "3000:3000"
    volumes:
      - grafana_data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    networks:
      - monitoring

volumes:
  jenkins_home:
  prometheus_data:
  grafana_data:

networks:
  monitoring:
    driver: bridge
Configure Prometheus to scrape Jenkins

Prometheus needs a configuration file telling it which targets to scrape and how often. You'll point it at the Jenkins metrics endpoint that the Prometheus plugin exposes.

  • Open prometheus.yml in your text editor and paste the following configuration:
global:
  scrape_interval: 60s
  scrape_timeout: 10s

rule_files:
  - "alert_rules.yml"

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "jenkins"
    metrics_path: "/prometheus/"
    scrape_interval: 60s
    static_configs:
      - targets: ["jenkins:8080"]

What does this configuration do?

  • scrape_interval: 60s tells Prometheus to collect metrics from all targets every 60 seconds.
  • rule_files points to alert_rules.yml where you'll define alerting rules in a later step. The file is empty for now, which is fine.
  • The jenkins job uses metrics_path: "/prometheus/" because the Jenkins Prometheus plugin serves metrics at that specific path (the trailing slash is required).
  • targets: ["jenkins:8080"] uses the container name as the hostname. Docker's shared network resolves this automatically.
  • Save prometheus.yml.

Prometheus won't start later?

The most common cause is a typo in prometheus.yml. Check that metrics_path has a trailing slash (/prometheus/ not /prometheus) and that indentation is consistent with 2 spaces.

Help me debug my prometheus.yml configuration

✔️ Awesome, I've got everything!

Great. Make sure prometheus.yml is saved before continuing.

ⓧ I'd like to double check the full code

global:
  scrape_interval: 60s
  scrape_timeout: 10s

rule_files:
  - "alert_rules.yml"

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "jenkins"
    metrics_path: "/prometheus/"
    scrape_interval: 60s
    static_configs:
      - targets: ["jenkins:8080"]
Launch the stack and verify all services

Both configuration files are ready. Time to pull the container images and start all three services.

  • Launch the entire monitoring stack by running this command from your jenkins-monitoring folder:
docker compose up -d

What does this command do?

docker compose up reads your docker-compose.yml, pulls the required images (if they aren't already cached), creates the containers, and starts them. The -d flag runs everything in detached mode so you get your terminal back.

The first run downloads about 1-2 GB of container images, which may take a few minutes depending on your connection speed. You'll see each container name printed as it's created.

  • Once the command finishes, verify all three containers are running:
docker compose ps

You should see three containers (jenkins, prometheus, grafana) all showing a status of Up.

  • Open your browser and verify each service is accessible:
  • Visit http://localhost:8080 for Jenkins. You should see the Jenkins setup wizard ("Unlock Jenkins" page).
  • Visit http://localhost:9090 for Prometheus. You should see the Prometheus expression browser.
  • Visit http://localhost:3000 for Grafana. You should see the Grafana login page.

One or more services won't start?

  • If a container exits immediately, check its logs by running docker compose logs jenkins (replace jenkins with the failing service name).
  • If port 8080, 9090, or 3000 is already in use by another application, you'll see a bind error. Close the conflicting application or change the host port in docker-compose.yml.
  • If Docker itself is not responding, make sure Docker Desktop is running (check for the whale icon in your system tray or menu bar).

Help me troubleshoot my Docker Compose stack

Your monitoring stack is live. Next up, you'll configure Jenkins itself by completing its setup wizard and installing the Prometheus metrics plugin.

Configure Jenkins with the Prometheus Plugin

Your monitoring stack is running, but Jenkins is still locked behind a setup wizard. Before Prometheus can scrape any metrics, Jenkins needs to be configured and told to expose them.

In this step, you will complete the Jenkins first-run wizard, then install the Prometheus metrics plugin that exposes an endpoint Prometheus already knows how to find.

In this step, get ready to:
  • Complete the Jenkins setup wizard and create an admin user.
  • Install the Prometheus metrics plugin.
  • Verify the metrics endpoint is accessible.
Complete the Jenkins setup wizard

Jenkins generates a one-time password on first boot that you need to unlock the setup wizard. The password is stored inside the container's filesystem.

  • Retrieve the initial admin password by running this command:
docker exec jenkins cat /var/jenkins_home/secrets/initialAdminPassword

You should see a long alphanumeric string printed in the terminal. Copy this string to your clipboard.

Don't see a password?

Jenkins may still be starting up. Wait 30 seconds and try the command again. You can check the container status by running docker ps and confirming the jenkins container shows a healthy status.

Help me troubleshoot the Jenkins initial admin password

  • Open your browser and navigate to http://localhost:8080.
  • On the Unlock Jenkins page, paste the password you copied into the Administrator password field.
  • Click Continue.

What is the Unlock Jenkins page?

Jenkins generates a random password on first boot to prevent unauthorized access. This is a one-time gate that proves you have access to the server. After the setup wizard completes, you will use the admin account you create instead.

  • On the Customize Jenkins page, click Install suggested plugins.

Why Install suggested plugins?

This installs a curated set of commonly used plugins (Git, Pipeline, Credentials, etc.) that most Jenkins setups need. The Prometheus plugin is not included in the suggested set, so you will install it separately in the next substep.

The plugin installation may take a few minutes. Wait until all plugins show a green checkmark.

  • On the Create First Admin User page, fill in any username, password, and email you like (this is local only).
  • Click Save and Finish.
  • On the Jenkins is ready page, click Start using Jenkins.

You should now see the Jenkins dashboard with a welcome message and an empty job list.

Install the Prometheus metrics plugin

The Prometheus metrics plugin exposes a /prometheus/ endpoint on Jenkins that outputs all internal metrics in a format Prometheus can scrape directly.

  • In the Jenkins sidebar, click Manage Jenkins.
  • Click Plugins.
  • Click the Available plugins tab on the left.
  • In the search box, type Prometheus metrics.
  • Check the box next to Prometheus metrics in the results.
  • Click Install (the button at the top right).

Jenkins will download and install the plugin. You should see a success message once it finishes.

What does the Prometheus metrics plugin do?

This plugin collects internal Jenkins data (build counts, queue sizes, executor usage, JVM memory) and formats it as Prometheus-compatible metrics. Every metric is prefixed with default_jenkins_ by default.

Your prometheus.yml already points at this endpoint. Once the plugin is active, Prometheus starts collecting data automatically on its next scrape cycle.

Plugin not showing up in search?

If the available plugins list is empty or outdated, Jenkins may not have downloaded the plugin catalog yet. Wait a minute for Jenkins to contact the Update Center, then refresh the page and search again.

Help me find the Prometheus metrics plugin in Jenkins

Verify the metrics endpoint

The plugin activates immediately after installation. You can confirm it is working by visiting the metrics endpoint in your browser.

  • Open a new browser tab and navigate to http://localhost:8080/prometheus/.

You should see a page full of plain-text metrics. Look for lines starting with default_jenkins_ to confirm the plugin is active.

What am I looking at?

This page is the raw Prometheus metrics endpoint. Each line is a metric name followed by its current value. Prometheus reads this page every 60 seconds (as configured in your prometheus.yml) and stores the values as time-series data.

You will query these metrics in Grafana later to build dashboard panels.

Seeing a 404 or blank page?

Make sure the URL ends with a trailing slash: /prometheus/ not /prometheus. Without the trailing slash, Jenkins returns a redirect that some browsers do not follow automatically.

If you still see nothing, try restarting Jenkins from the terminal:

docker restart jenkins

Wait 30 seconds, then visit the URL again.

Help me troubleshoot the Jenkins Prometheus metrics endpoint

Jenkins is now exposing metrics and Prometheus is already configured to scrape them. Next up, you will create pipelines that generate real build data for your monitoring stack to observe.

Create Sample Pipelines and Generate Metrics

Your monitoring stack is running and Prometheus is scraping the Jenkins endpoint. But right now there is nothing to monitor.

A dashboard without data is just an empty canvas. In this step, you will create two pipeline jobs that simulate real CI/CD work. One pipeline succeeds (Build, Test, Deploy) and the other deliberately fails. Running them multiple times produces the build history that Prometheus needs to surface meaningful metrics.

In this step, get ready to:
  • Create a sample pipeline with Build, Test, and Deploy stages that produce observable metrics.
  • Create a deliberately failing pipeline to generate failure metrics.
  • Verify that Prometheus is scraping Jenkins metrics by querying the Prometheus UI.
Create and run the sample pipeline

The first pipeline simulates a healthy CI/CD workflow with three stages. Each stage includes a sleep step so that build duration metrics have meaningful, non-zero values for Prometheus to scrape.

  • Open your text editor and create a file called Jenkinsfile inside your jenkins-monitoring project folder.
  • Paste the following pipeline script into Jenkinsfile:
pipeline {
    agent any
    options {
        timeout(time: 10, unit: 'MINUTES')
        buildDiscarder(logRotator(numToKeepStr: '10'))
        timestamps()
    }
    stages {
        stage('Build') {
            steps {
                echo 'Building application...'
                sleep(time: 5, unit: 'SECONDS')
                echo 'Build complete.'
            }
        }
        stage('Test') {
            steps {
                echo 'Running tests...'
                sleep(time: 3, unit: 'SECONDS')
                echo 'All tests passed.'
            }
        }
        stage('Deploy') {
            steps {
                echo 'Deploying to staging...'
                sleep(time: 2, unit: 'SECONDS')
                echo 'Deployment complete.'
            }
        }
    }
    post {
        always {
            echo "Build duration: ${currentBuild.durationString.replace(' and counting', '')}"
            echo "Build result: ${currentBuild.currentResult}"
        }
        failure {
            echo 'ALERT: Pipeline failed! Check monitoring dashboard.'
        }
    }
}

What does this pipeline do?

The options block sets a 10-minute timeout, keeps only the last 10 builds, and adds timestamps to the console log.

Each stage uses sleep to simulate real work. This produces non-trivial duration values that Prometheus can scrape as histogram data.

The post block runs after every build. It prints the total duration and result, which is useful for quick console-level monitoring.

  • Save Jenkinsfile.

Now create the pipeline job inside Jenkins.

  • Switch back to Jenkins in your browser at http://localhost:8080.
  • Click New Item in the left sidebar.
  • In the Enter an item name field, type sample-pipeline.
  • Select Pipeline from the list of item types.
  • Click OK.
  • Scroll down to the Pipeline section at the bottom of the configuration page.
  • Make sure the Definition dropdown is set to Pipeline script.
  • Paste the entire contents of your Jenkinsfile into the Script text area.
  • Click Save.

You are now on the job page for sample-pipeline. Run it several times to build up history.

  • Click Build Now in the left sidebar.
  • Wait about 10-15 seconds for the build to complete (the sleep steps add up to 10 seconds total).
  • Click Build Now two more times, waiting for each build to finish before starting the next.

After three builds, you should see three green checkmarks under Build History in the left sidebar.

Build stuck in queue or failing?

  • If the build says "Waiting for next available executor", Jenkins may still be initialising its built-in node. Wait 30 seconds and try again.
  • If the build fails with a script error, double-check that you pasted the entire pipeline script including the opening pipeline { and closing }.
  • If Jenkins reports "Timestamps plugin not found", go to Manage Jenkins > Plugins > Available plugins and install the Timestamper plugin.

Why is my sample-pipeline build failing or stuck in Jenkins?

Create and run the failing pipeline

In production, not every pipeline succeeds. SREs need to detect failure patterns quickly. This second pipeline deliberately fails during the Test stage so your monitoring stack has failure data to work with.

  • In your text editor, create a new file called Jenkinsfile-failing in the same project folder.
  • Paste the following pipeline script into Jenkinsfile-failing:
pipeline {
    agent any
    options {
        timeout(time: 10, unit: 'MINUTES')
        buildDiscarder(logRotator(numToKeepStr: '10'))
        timestamps()
    }
    stages {
        stage('Build') {
            steps {
                echo 'Building application...'
                sleep(time: 2, unit: 'SECONDS')
            }
        }
        stage('Test') {
            steps {
                echo 'Running tests...'
                error('Simulated test failure for monitoring exercise')
            }
        }
    }
    post {
        always {
            echo "Build duration: ${currentBuild.durationString.replace(' and counting', '')}"
            echo "Build result: ${currentBuild.currentResult}"
        }
        failure {
            echo 'ALERT: Pipeline failed! This is expected for the monitoring exercise.'
        }
    }
}

What does this pipeline do differently?

The error('Simulated test failure for monitoring exercise') step forces the pipeline to fail immediately during the Test stage. This is a controlled failure that produces failure-count metrics in Prometheus.

The post.failure block prints a message confirming the failure was intentional. This helps you distinguish expected failures from real problems in the console output.

  • Save Jenkinsfile-failing.

Now create the second pipeline job in Jenkins.

  • In Jenkins, click the Jenkins logo in the top-left to return to the Dashboard.
  • Click New Item in the left sidebar.
  • In the Enter an item name field, type failing-pipeline.
  • Select Pipeline and click OK.
  • Scroll down to the Pipeline section.
  • Make sure Definition is set to Pipeline script.
  • Paste the entire contents of Jenkinsfile-failing into the Script text area.
  • Click Save.
  • Click Build Now three times, waiting for each build to finish before starting the next.

Each build should fail with a red X icon under Build History. This is expected.

Seeing unexpected errors in the failing pipeline?

The only error should come from the error() step in the Test stage. If you see a different error (like a syntax issue), verify the script is pasted exactly. The error() line should NOT be inside quotes.

Help me troubleshoot my failing-pipeline Jenkins job

Verify Prometheus is scraping Jenkins metrics

With six builds now complete (three successes, three failures), Prometheus should have scraped build metrics from the /prometheus/ endpoint. It can take up to 60 seconds for new metrics to appear because that is the scrape interval you configured in prometheus.yml.

  • Open the Prometheus UI at http://localhost:9090 in your browser.
  • Click inside the expression editor at the top of the page.
  • Type the following query:
default_jenkins_builds_duration_milliseconds_summary_count
  • Click the Execute button (or press Enter).

You should see one or more result rows in the table below the query box. Each row represents a scraped metric with labels like jenkins_job showing your pipeline names.

What does this metric tell you?

The metric default_jenkins_builds_duration_milliseconds_summary_count tracks how many builds have completed. If you see values like 3 or 6 in the results, Prometheus is successfully scraping your Jenkins build data.

The default_ prefix comes from the PROMETHEUS_NAMESPACE environment variable that the Prometheus metrics plugin uses. Its default value is default.

No results showing up?

  • Wait 60 seconds and try the query again. Prometheus scrapes Jenkins every 60 seconds based on your prometheus.yml configuration.
  • Check that the Jenkins target is healthy. In Prometheus, click Status in the top nav, then Targets. The jenkins job should show a state of UP.
  • If the target shows DOWN, verify that the Prometheus metrics plugin is installed in Jenkins and that http://localhost:8080/prometheus/ returns metric data in your browser.

Why is Prometheus not scraping my Jenkins metrics?

✔️ Awesome, I've got everything!

Great. Double check you have saved both Jenkinsfile and Jenkinsfile-failing in your project folder.

ⓧ I'd like to double check the full code

Here is the complete Jenkinsfile:

pipeline {
    agent any
    options {
        timeout(time: 10, unit: 'MINUTES')
        buildDiscarder(logRotator(numToKeepStr: '10'))
        timestamps()
    }
    stages {
        stage('Build') {
            steps {
                echo 'Building application...'
                sleep(time: 5, unit: 'SECONDS')
                echo 'Build complete.'
            }
        }
        stage('Test') {
            steps {
                echo 'Running tests...'
                sleep(time: 3, unit: 'SECONDS')
                echo 'All tests passed.'
            }
        }
        stage('Deploy') {
            steps {
                echo 'Deploying to staging...'
                sleep(time: 2, unit: 'SECONDS')
                echo 'Deployment complete.'
            }
        }
    }
    post {
        always {
            echo "Build duration: ${currentBuild.durationString.replace(' and counting', '')}"
            echo "Build result: ${currentBuild.currentResult}"
        }
        failure {
            echo 'ALERT: Pipeline failed! Check monitoring dashboard.'
        }
    }
}

Here is the complete Jenkinsfile-failing:

pipeline {
    agent any
    options {
        timeout(time: 10, unit: 'MINUTES')
        buildDiscarder(logRotator(numToKeepStr: '10'))
        timestamps()
    }
    stages {
        stage('Build') {
            steps {
                echo 'Building application...'
                sleep(time: 2, unit: 'SECONDS')
            }
        }
        stage('Test') {
            steps {
                echo 'Running tests...'
                error('Simulated test failure for monitoring exercise')
            }
        }
    }
    post {
        always {
            echo "Build duration: ${currentBuild.durationString.replace(' and counting', '')}"
            echo "Build result: ${currentBuild.currentResult}"
        }
        failure {
            echo 'ALERT: Pipeline failed! This is expected for the monitoring exercise.'
        }
    }
}

You now have real build data flowing through your monitoring stack. Next up, you will build a Grafana dashboard that visualises these metrics into the four critical SRE signals.

Build the SRE Dashboard in Grafana

Prometheus is collecting Jenkins metrics, but scrolling through raw metric names in the Prometheus UI is not how SREs detect problems in production. Dashboards give you a single screen where abnormal values jump out visually.

In this step, you'll connect Grafana to your Prometheus instance and build a dashboard with four panels covering the signals that predict the majority of Jenkins outages.

In this step, get ready to:
  • Add Prometheus as a data source in Grafana.
  • Create a dashboard with four SRE signal panels.
  • Write PromQL queries and set alerting thresholds for each panel.
Add Prometheus as a data source

Before Grafana can display any metrics, it needs to know where to fetch them from. A data source tells Grafana the address of your Prometheus server.

  • Open your browser and navigate to http://localhost:3000.
  • Log in with username admin and password admin.
  • If prompted to change the password, click Skip for now.
  • Click Connections in the left-side menu.
  • Under Connections, click Add new connection.
  • Enter Prometheus in the search bar.
  • Click the Prometheus data source result.
  • Click Add new data source in the upper right.

Grafana opens the Settings tab for your new Prometheus data source.

  • In the Prometheus server URL field, enter:
http://prometheus:9090

Why not localhost?

Grafana runs inside a Docker container on the monitoring network. From inside that network, the Prometheus container is reachable by its service name prometheus, not localhost. If you used localhost, Grafana would try to reach a Prometheus instance running inside its own container, which doesn't exist.

  • Scroll to the bottom and click Save & test.

You should see a green banner confirming the data source is working.

Seeing a red error banner?

  • Double-check the URL is exactly http://prometheus:9090 with no trailing slash.
  • Confirm your Prometheus container is running by opening http://localhost:9090 in your browser.
  • If the container stopped, run docker compose up -d from your project folder to restart it.

Help me debug my Grafana data source connection

Create dashboard panels for the four SRE signals

SREs focus on a small set of signals that predict the vast majority of outages. For Jenkins, those signals are build failure rate, queue depth, executor utilisation, and JVM heap pressure.

Why these four signals?

JVM heap exhaustion is the number one cause of Jenkins outages. A deep build queue means agents are overwhelmed. Executor saturation means no capacity remains for new builds. A high failure rate means something is broken in your pipelines.

Watching all four together lets you distinguish between 'one bad pipeline' and 'the whole system is overloaded.'

Start by creating a new dashboard and your first panel.

  • Click Dashboards in the left-side menu.
  • Click New and select New Dashboard.
  • On the empty dashboard, click + Add visualization.
  • In the dialog box that opens, select your Prometheus data source.

The panel editor opens. You'll build your first panel here: Build Success Rate.

Write PromQL queries and set thresholds

Each panel needs a PromQL query that fetches the right metric and a threshold that marks when values become dangerous. You'll configure all four panels in sequence.

Panel 1: Build Success Rate

  • In the query editor at the bottom of the panel editor, switch to Code mode (if not already active).
  • Paste the following PromQL query:
(rate(default_jenkins_builds_failed_build_count[1h]) / (rate(default_jenkins_builds_failed_build_count[1h]) + rate(default_jenkins_builds_success_build_count[1h]))) * 100

What does this query do?

This calculates the percentage of builds that failed over the last hour. It divides the rate of failed builds by the total rate of all builds (failed + successful) and multiplies by 100 to get a percentage.

  • Click Refresh to query the data source.
  • In the visualization list on the right, select Stat (for a single large number display).
  • Under Panel options in the right sidebar, set the title to Build Failure Rate.
  • Scroll down to Standard options and set Unit to Percent (0-100).
  • Scroll to Thresholds and set the red threshold to 20 (meaning values above 20% turn red).
  • Click Back to dashboard in the top-left.

Your first panel is now on the dashboard. Next, add Panel 2.

Panel 2: Queue Depth

  • Click Add in the dashboard header and select Visualization.
  • Select your Prometheus data source.
  • Switch to Code mode and paste:
default_jenkins_queue_size_value
  • Click Refresh.
  • Select the Stat visualization.
  • Set the panel title to Queue Depth.
  • Under Thresholds, set the red threshold to 5.
  • Click Back to dashboard.

Panel 3: Executor Utilisation

  • Click Add in the dashboard header and select Visualization.
  • Select your Prometheus data source.
  • Switch to Code mode and paste:
(default_jenkins_executor_in_use_value / default_jenkins_executor_count_value) * 100

What does this query do?

This divides the number of executors currently running builds by the total number of available executors. The result is a percentage showing how saturated your build capacity is.

  • Click Refresh.
  • Select the Gauge visualization.
  • Set the panel title to Executor Utilisation.
  • Under Standard options, set Unit to Percent (0-100) and Max to 100.
  • Under Thresholds, set the red threshold to 80.
  • Click Back to dashboard.

Panel 4: JVM Heap Usage

  • Click Add in the dashboard header and select Visualization.
  • Select your Prometheus data source.
  • Switch to Code mode and paste:
(default_jenkins_jvm_memory_bytes_used{area="heap"} / default_jenkins_jvm_memory_bytes_max{area="heap"}) * 100

Why is JVM heap the top priority?

JVM heap exhaustion is the number one cause of Jenkins outages. When Jenkins runs out of heap memory, it throws an OutOfMemoryError and either becomes unresponsive or crashes entirely. Catching heap usage climbing above 80% gives you time to act before the system goes down.

  • Click Refresh.
  • Select the Gauge visualization.
  • Set the panel title to JVM Heap Usage.
  • Under Standard options, set Unit to Percent (0-100) and Max to 100.
  • Under Thresholds, set the red threshold to 80.
  • Click Back to dashboard.

All four panels are now on your dashboard.

  • Click Save dashboard in the top-right.
  • Enter Jenkins SRE Dashboard as the title.
  • Click Save.

You should see four panels on your dashboard displaying live Jenkins metrics from Prometheus.

Panels showing No data?

  • Make sure you ran both pipelines in Jenkins at least a few times in the previous step. Prometheus needs scraped data to return results.
  • Check that your Prometheus data source shows a green Save & test result.
  • Try widening the time range in the top-right corner of the dashboard to Last 1 hour or Last 3 hours.
  • Confirm Prometheus is scraping Jenkins by visiting http://localhost:9090/targets and checking the jenkins job shows UP.

Help me troubleshoot my Grafana panels showing no data

Your SRE dashboard is live. Next up, you'll configure Prometheus alerting rules that automatically fire when these metrics cross dangerous thresholds.

Configure Alerting Rules

Your Grafana dashboard gives you visibility into Jenkins health, but dashboards only help when someone is watching them.

In production, SREs rely on automated alerts that fire when metrics cross dangerous thresholds. Prometheus has a built-in alerting engine that evaluates rules on every scrape cycle and transitions alerts through inactive, pending, and firing states.

In this step, get ready to:
  • Write Prometheus alerting rules for queue depth, JVM heap pressure, and build failure rate.
  • Reload Prometheus to pick up the new rules without restarting the container.
  • Trigger an alert by running the failing pipeline and observe it in the Prometheus Alerts UI.
Write the alerting rules

Prometheus alerting rules live in YAML files that define a condition (a PromQL expression), a duration the condition must hold, and metadata like severity labels and human-readable annotations.

Your prometheus.yml already references alert_rules.yml in its rule_files section. You just need to fill in the rules.

  • Open alert_rules.yml in your text editor and replace its contents with the following alerting rules:
groups:
  - name: jenkins_alerts
    rules:
      # Alert when the build queue backs up
      - alert: JenkinsQueueTooDeep
        expr: default_jenkins_queue_size_value > 5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Jenkins build queue is backing up"
          description: "Queue depth is {{ $value }} (threshold: 5). Agents may be overwhelmed."

      # Alert when JVM heap approaches capacity
      - alert: JenkinsHighHeapUsage
        expr: (default_jenkins_jvm_memory_bytes_used{area="heap"} / default_jenkins_jvm_memory_bytes_max{area="heap"}) * 100 > 80
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Jenkins JVM heap usage is above 80%"
          description: "Current heap usage is {{ $value }}%. Risk of OutOfMemoryError."

      # Alert when too many builds are failing
      - alert: JenkinsHighFailureRate
        expr: (rate(default_jenkins_builds_failed_build_count[1h]) / (rate(default_jenkins_builds_failed_build_count[1h]) + rate(default_jenkins_builds_success_build_count[1h]))) * 100 > 20
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Jenkins build failure rate exceeds 20%"
          description: "Failure rate is {{ $value }}% over the last hour."

What do these rules do?

  • JenkinsQueueTooDeep fires when more than 5 jobs are waiting in the queue for over 5 minutes. This means agents are overwhelmed and builds are stalling.
  • JenkinsHighHeapUsage fires when the JVM heap exceeds 80% capacity for 5 minutes. This is the number one cause of Jenkins outages.
  • JenkinsHighFailureRate fires when more than 20% of builds fail over a rolling 1-hour window, sustained for 10 minutes. This catches systemic issues like broken shared libraries or misconfigured agents.

What does the for field mean?

The for duration prevents noisy alerts from brief spikes. Prometheus must see the condition continuously for that duration before the alert transitions from "pending" to "firing".

  • Save alert_rules.yml.

YAML indentation issues?

YAML is sensitive to spaces. Make sure you use exactly 2 spaces per indentation level with no tabs. The groups: key should be at the very start of the file with no leading spaces.

help me fix YAML indentation errors in my alert_rules.yml

Reload Prometheus to pick up the rules

Prometheus does not automatically detect file changes. You need to tell it to reload its configuration. The --web.enable-lifecycle flag you configured in docker-compose.yml exposes an HTTP endpoint for exactly this purpose.

  • Reload Prometheus by running this command in your terminal:
curl -X POST http://localhost:9090/-/reload
  • Verify Prometheus loaded the rules by opening http://localhost:9090/alerts in your browser.
  • You should see three alert names listed: JenkinsQueueTooDeep, JenkinsHighHeapUsage, and JenkinsHighFailureRate.
  • All three should show a green "inactive" state (no threshold is currently breached).

Don't see your alerts listed?

If the page shows no rules, Prometheus failed to parse the file. Check the Prometheus container logs for YAML errors by running:

docker logs prometheus --tail 20

Look for lines mentioning "error loading rule group" or "bad rule file". Fix any indentation issues in alert_rules.yml, save, and run the curl reload command again.

help me debug why Prometheus is not loading my alerting rules

Trigger an alert by running the failing pipeline

Alerting rules are only useful if they actually fire when things go wrong. You will now deliberately break things to see the JenkinsHighFailureRate alert activate.

  • Switch back to Jenkins at http://localhost:8080.
  • Click on your failing pipeline job (the one using Jenkinsfile-failing).
  • Click Build Now at least 4-5 times to generate a burst of failures.
  • Wait for all builds to complete (each takes a few seconds).

Why multiple runs?

The JenkinsHighFailureRate rule uses rate() over a 1-hour window. A single failure won't push the rate above 20%. Multiple rapid failures combined with the builds you ran earlier create enough signal for the alert to trigger.

  • Return to the Prometheus Alerts UI at http://localhost:9090/alerts.
  • Wait 1-2 minutes for Prometheus to scrape the new metrics (your scrape interval is 60 seconds).
  • Refresh the page. You should see JenkinsHighFailureRate in a yellow "pending" or red "firing" state.

What is the difference between pending and firing?

When Prometheus first detects a threshold breach, the alert enters "pending" state. It stays pending for the for duration (10 minutes for this rule). If the condition is still true after that window, it transitions to "firing".

Seeing "pending" confirms everything is working correctly. In a real production environment, the "firing" state would trigger notifications to your on-call team.

Alert still showing inactive?

Make sure you ran the failing pipeline at least 4-5 times and all builds completed. Then wait at least 60 seconds for Prometheus to scrape.

You can confirm Prometheus is receiving failure data by querying default_jenkins_builds_failed_build_count in the Prometheus expression browser at http://localhost:9090/graph. The value should be greater than 0.

help me understand why my Prometheus alert is not firing after running failing Jenkins builds

✔️ Awesome, I've got everything!

Great. Double check you've saved alert_rules.yml and that all three alerts appear on the Prometheus Alerts page.

ⓧ I'd like to double check the full code

Here is the complete alert_rules.yml file:

groups:
  - name: jenkins_alerts
    rules:
      - alert: JenkinsQueueTooDeep
        expr: default_jenkins_queue_size_value > 5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Jenkins build queue is backing up"
          description: "Queue depth is {{ $value }} (threshold: 5). Agents may be overwhelmed."

      - alert: JenkinsHighHeapUsage
        expr: (default_jenkins_jvm_memory_bytes_used{area="heap"} / default_jenkins_jvm_memory_bytes_max{area="heap"}) * 100 > 80
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Jenkins JVM heap usage is above 80%"
          description: "Current heap usage is {{ $value }}%. Risk of OutOfMemoryError."

      - alert: JenkinsHighFailureRate
        expr: (rate(default_jenkins_builds_failed_build_count[1h]) / (rate(default_jenkins_builds_failed_build_count[1h]) + rate(default_jenkins_builds_success_build_count[1h]))) * 100 > 20
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Jenkins build failure rate exceeds 20%"
          description: "Failure rate is {{ $value }}% over the last hour."

Your monitoring stack is now fully operational. You have dashboards for visibility and alerts for automated detection. This is the same two-layer approach SREs use in production to keep CI/CD pipelines healthy at scale.

Secret mission

Track P95 Build Duration and Create an SLO Alert

Your dashboard catches failures after they happen. But what if you could detect slow pipelines before they breach your team's SLO? In this secret mission, you will add a P95 build duration panel, define a 5-minute SLO, and create a Prometheus alert that fires the moment your pipeline trends toward a breach.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally with Docker, so there are no ongoing cloud costs.

Resources you used:

  • Docker containers: jenkins, prometheus, grafana (consuming local CPU and memory).
  • Docker volumes: jenkins_home, prometheus_data, grafana_data (storing Jenkins jobs, Prometheus time-series data, and Grafana dashboards).
  • Local project files: docker-compose.yml, prometheus.yml, alert_rules.yml, Jenkinsfile, Jenkinsfile-failing.

Keep everything running

No action needed. Choose this if you are still actively experimenting with your monitoring stack. The three containers use approximately 2GB of RAM while running.

  • Your Jenkins pipelines, Grafana dashboards, and Prometheus data all persist in Docker volumes.
  • Access Jenkins at http://localhost:8080, Prometheus at http://localhost:9090, and Grafana at http://localhost:3000 any time.

Pause - I'll come back to this later

Shut down running containers to free up memory. Your data stays intact in Docker volumes so you can resume later.

  • Stop all three containers by running this command from your project folder:
docker compose stop
  • When you are ready to resume, start the stack again by running:
docker compose start
  • All Jenkins jobs, Grafana dashboards, and Prometheus metrics will still be there.

Delete - I don't want to use this again

Remove all containers and volumes permanently. All Jenkins jobs, Grafana dashboards, and Prometheus data will be lost.

  • Remove all containers and their associated volumes by running this command from your project folder:
docker compose down -v
  • Confirm the containers and volumes are gone by running:
docker ps -a
docker volume ls
  • You should not see jenkins, prometheus, or grafana in either output.
  • Delete the project folder from your Desktop by running:

macOS

cd ~/Desktop
rm -rf jenkins-monitoring

Windows

cd ~\Desktop
rmdir /s /q jenkins-monitoring

Nice Work!

Nice Work!

You just built the same monitoring stack that SREs at companies running hundreds of Jenkins pipelines rely on every day. You can now detect pipeline failures, queue saturation, and JVM memory pressure before they cause outages.

What you learned:

  • Deployed a three-service observability stack (Jenkins + Prometheus + Grafana) using Docker Compose and configured Prometheus to scrape Jenkins metrics via the /prometheus/ endpoint.
  • Built a Grafana dashboard surfacing the four critical SRE signals: build success rate, queue depth, executor utilisation, and JVM heap usage.
  • Wrote PromQL alerting rules that catch failures before they become outages, and practised real incident simulation by deliberately failing pipelines and observing the monitoring response.
  • Secret Mission: Tracked P95 build duration against an SLO using histogram_quantile and created automated breach alerts in both Prometheus and your Jenkinsfile.

Ready to quiz yourself?