Monitor Kubernetes with Prometheus and Grafana

Deploy a full observability stack on EKS and get Gmail alerts when things break.

Introduction

30 Second Summary

Imagine your application crashes at 2 AM. Without monitoring, you would not find out until the next morning when users start complaining. The difference between a minor hiccup and a major outage often comes down to how fast you know something went wrong.

In this project, you will deploy a complete monitoring and alerting stack on your existing EKS cluster. You'll set up Prometheus to collect metrics, Grafana to visualize them, and Alertmanager to send you Gmail notifications the moment something breaks.

What You'll Build

When you finish this project, you'll open Grafana, see live dashboards tracking your entire delivery chain from Jenkins build to running application, intentionally kill a pod, and watch an alert land in your Gmail inbox minutes later.

By the end of this project, you'll have:

  • A Grafana dashboard showing real-time CPU, memory, and pod health across your 2-node EKS cluster, updating as you watch.
  • Your Jenkins pipeline metrics (build count, duration, success/failure) flowing from your external EC2 instance into Grafana so you can spot slow or broken builds at a glance.
  • Automatic Gmail alerts firing within minutes of a pod crash, high memory usage, or a Jenkins build failure, so you never have to check manually.
  • Secret Mission: Instrument your Python attendance tracker with custom business metrics and build a dedicated Grafana dashboard for application-level observability.

Are there any prerequisites?

This project builds on top of an existing CI/CD pipeline. You need a running EKS cluster with 2 nodes, your simple-attendance-tracker app deployed via ArgoCD, and Jenkins running on a separate EC2 instance with your CI pipeline configured.

You also need kubectl configured for your cluster and a Google account with 2-Step Verification enabled (for Gmail alerts).

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

You have a running EKS cluster, a deployed application, and a Jenkins CI pipeline. Right now, if something breaks, you would only find out by manually checking each piece. This project gives you a monitoring and alerting stack that watches everything and tells you the moment something goes wrong.

Install Helm and Deploy the Monitoring Stack

Your EKS cluster is running, your attendance-tracker app is deployed, and Jenkins is building your code. But right now, you have zero visibility into what's happening inside the cluster.

In this step, you'll install Helm (a package manager for Kubernetes) and use it to deploy the kube-prometheus-stack. This single Helm chart bundles Prometheus, Grafana, Alertmanager, node-exporter, and kube-state-metrics into one release.

In this step, get ready to:
  • Install Helm on macOS using Homebrew.
  • Deploy kube-prometheus-stack into a monitoring namespace with custom configuration.
  • Verify all monitoring pods are running.
Install Helm
  • Check if Helm is already installed by running this command:
helm version

✔️ I see a version number

Helm is ready to go. Skip ahead to the next substep.

ⓧ Command not found

  • Install Helm using Homebrew by running:
brew install helm
  • Once the install finishes, confirm it worked by running:
helm version

You should see version information printed to your terminal (something like v4.2.2 or similar).

Still seeing command not found?

Make sure Homebrew itself is installed and up to date. Run brew --version to confirm. If Homebrew is missing, install it from https://brew.sh first, then retry.

Help me troubleshoot Helm not being found after install

Add the Helm repo and deploy the monitoring stack

Helm charts live in repositories, similar to how packages live in registries like npm or pip. You need to add the prometheus-community repository so Helm knows where to find the chart.

  • Add the prometheus-community Helm repo and refresh the chart index by running these commands:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

You should see "prometheus-community" has been added to your repositories followed by Update Complete.

Seeing a network error?

Helm needs internet access to reach the chart repository. Check your network connection and try again. If you're behind a corporate proxy, ensure your terminal's proxy settings are configured.

Help me fix my Helm repo add network error

Before you install the chart, you'll create a configuration file that customizes the deployment. This tells Helm to enable persistent storage for Grafana, turn on Alertmanager, and keep metrics data for 15 days.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create a folder for your monitoring configuration and move into it by running:
mkdir monitoring-config
cd monitoring-config
  • Create a new file called values.yaml in the monitoring-config folder and paste the following content into it:
# kube-prometheus-stack custom values
grafana:
  persistence:
    enabled: true
    size: 5Gi
alertmanager:
  enabled: true
prometheus:
  prometheusSpec:
    retention: 15d

What do these values configure?

  • grafana.persistence.enabled: true tells Helm to create a persistent volume for Grafana. Without this, your dashboards and settings vanish every time the pod restarts.
  • alertmanager.enabled: true deploys Alertmanager alongside Prometheus so you can route notifications later.
  • retention: 15d keeps 15 days of metrics history. This gives you enough data to spot trends without consuming excessive disk space.
  • Save values.yaml and confirm it exists by running:
ls

You should see values.yaml listed in the output.

  • Deploy the full monitoring stack into a new monitoring namespace by running:
helm install monitoring prometheus-community/kube-prometheus-stack -n monitoring --create-namespace -f values.yaml

What does this command do?

  • helm install monitoring creates a Helm release named "monitoring" so you can manage the deployment later.
  • prometheus-community/kube-prometheus-stack is the chart name from the repo you just added.
  • -n monitoring --create-namespace puts everything into a dedicated namespace (and creates it if it doesn't exist).
  • -f values.yaml applies your custom configuration on top of the chart's defaults.

This command takes 1-2 minutes. Helm downloads the chart, creates all the Kubernetes resources, and prints a summary when finished.

Helm install failed?

  • If you see a YAML parsing error, check that your values.yaml uses spaces for indentation (not tabs) and matches the structure shown above exactly.
  • If you see a "namespace already exists" error, remove --create-namespace from the command and run it again.
  • If you see insufficient CPU or memory errors, your EKS nodes may be too small. Ensure your node group uses instances with at least 2 vCPUs and 4 GiB memory.

Help me fix my failed Helm install

Verify all monitoring pods are running

Before you run the next command, predict how many pods you expect to see in the monitoring namespace. Think about what the chart said it would install.

  • Check the status of all pods in the monitoring namespace by running:
kubectl get pods -n monitoring

You should see 6-8 pods, all showing a Running status. This includes the Prometheus server, Grafana, Alertmanager, a node-exporter pod on each of your 2 EKS nodes, and kube-state-metrics.

Pods stuck in Pending or CrashLoopBackOff?

  • Pending pods usually mean insufficient resources. Run kubectl describe pod <pod-name> -n monitoring to see the scheduling failure reason.
  • CrashLoopBackOff often means a configuration issue. Check logs with kubectl logs <pod-name> -n monitoring to identify the problem.
  • If PersistentVolumeClaims are stuck in Pending, your EKS cluster may need the EBS CSI driver installed. Run kubectl get pvc -n monitoring to check.

Help me diagnose my monitoring pods that aren't running

Why kube-prometheus-stack instead of separate installs?

You could install Prometheus, Grafana, and Alertmanager individually, but the kube-prometheus-stack chart bundles them together with pre-configured connections. Prometheus already knows where to find Alertmanager, Grafana already has Prometheus as a data source, and dozens of dashboards come pre-built.

This means you skip the manual wiring and go straight to using the stack.

✔️ Awesome, I've got everything!

Great. Double check you've saved values.yaml and that all pods show Running when you run kubectl get pods -n monitoring.

ⓧ I'd like to double check the full code

Here is the complete values.yaml file you should have in your monitoring-config folder:

# kube-prometheus-stack custom values
grafana:
  persistence:
    enabled: true
    size: 5Gi
alertmanager:
  enabled: true
prometheus:
  prometheusSpec:
    retention: 15d

Your monitoring stack is live. Next up, you'll open Grafana and explore the dashboards that are already tracking your cluster.

Explore Grafana and Cluster Dashboards

Your monitoring stack is deployed and all pods are running in the monitoring namespace.

The stack is collecting metrics, but you have not seen it in action yet. Grafana comes pre-configured with dashboards that already track your 2-node EKS cluster without any extra setup. In this step, you will port-forward Grafana and explore those dashboards to see real-time CPU, memory, and pod health.

In this step, get ready to:
  • Port-forward Grafana and log in with admin credentials.
  • Explore pre-built Kubernetes dashboards for CPU, memory, and pod metrics.
  • Locate your attendance-tracker pods and observe their resource usage.
Port-forward Grafana and log in

Grafana is running inside your cluster, but it is not exposed to the internet. Port-forwarding creates a tunnel from your local machine to the Grafana service so you can access it in your browser.

  • First, retrieve the Grafana admin password from the Kubernetes secret by running this command:
kubectl get secret -n monitoring monitoring-grafana -o jsonpath="{.data.admin-password}" | base64 --decode
  • Copy the password that is printed to your terminal. You will need it in a moment.
  • Start the port-forward to Grafana by running this command:
kubectl port-forward svc/monitoring-grafana 3000:80 -n monitoring

What does this command do?

This forwards traffic from port 3000 on your local machine to port 80 on the monitoring-grafana service inside the cluster. As long as this command is running, you can access Grafana at http://localhost:3000.

The terminal will appear to hang. This is normal. It stays open to keep the tunnel active.

  • Open your browser and navigate to http://localhost:3000.
  • In the login screen, enter admin as the username.
  • Paste the password you copied from the secret retrieval command.
  • Click Log in.

You should see the Grafana home page with a welcome banner and a left-hand sidebar for navigation.

Can't connect to localhost:3000?

  • Make sure the port-forward command is still running in your terminal. If you accidentally closed it, run the kubectl port-forward command again.
  • Check that nothing else is using port 3000. If another process is on that port, stop it or change the local port in the command (e.g. 3001:80).
  • If the password was rejected, re-run the kubectl get secret command and make sure you copy the full output with no trailing newline.

Still stuck? Help me troubleshoot the Grafana port-forward or login.

Explore the pre-built cluster dashboards

The kube-prometheus-stack ships with dozens of pre-built dashboards that visualize your cluster metrics out of the box. You do not need to create any queries or panels yourself.

  • In the left sidebar, click the Dashboards icon (four squares).
  • You will see a list of folders and dashboards pre-loaded by the Helm chart.
  • Click on the dashboard named Kubernetes / Compute Resources / Cluster.

This dashboard shows aggregate resource usage across your entire cluster. You should see panels for CPU usage, memory usage, and network bandwidth for both of your EKS nodes.

What am I looking at?

Each panel pulls data from Prometheus via PromQL queries running behind the scenes. The node-exporter pods on each node report hardware-level metrics (CPU, memory, disk). The kube-state-metrics pod reports Kubernetes object states (pod counts, deployment status).

Together, these two data sources power the entire cluster view you see on this dashboard.

  • Use the browser back button or click Dashboards in the sidebar again.
  • Now open the dashboard named Kubernetes / Compute Resources / Namespace (Pods).
  • In the namespace dropdown at the top of the dashboard, select the namespace where your simple-attendance-tracker app is deployed.

You should now see per-pod CPU and memory graphs that show exactly how much each pod in that namespace is consuming.

Locate your attendance-tracker pods

The namespace dashboard lets you drill down to individual pods. This is where you confirm that your application is visible to the monitoring stack.

  • On the Kubernetes / Compute Resources / Namespace (Pods) dashboard, look at the CPU Usage panel.
  • Identify your simple-attendance-tracker pod(s) in the legend below the graph. Each pod name appears as a colored line.
  • Hover over the graph line for your attendance-tracker pod to see its current CPU consumption in the tooltip.
  • Scroll down to the Memory Usage panel and confirm your attendance-tracker pod appears there as well.

Before you move on, predict what you think will happen to these graphs if your pod suddenly restarts. Keep that thought in mind for Step 5.

Why does this matter?

Seeing your application pods in Grafana confirms that Prometheus is scraping them via kube-state-metrics and node-exporter. If your pod did not appear here, you would know something is misconfigured before relying on alerts.

Keep the port-forward running

Leave the kubectl port-forward terminal open. You will use Grafana again in later steps when you add Jenkins dashboards and verify alert rules.

You now have eyes on your entire cluster. You can see CPU and memory consumption for every node and every pod, including your attendance-tracker application. Next up, you will close a major blind spot by pulling Jenkins pipeline metrics into this same monitoring stack.

Scrape Jenkins Metrics from Your External EC2 Instance

Your cluster monitoring looks great. You can see CPU, memory, and pod health for both EKS nodes in Grafana. But your Jenkins CI server running on a separate EC2 instance is completely invisible to Prometheus.

A build could be failing right now and you would not know until you manually open the Jenkins UI. In this step, you'll install the Prometheus Metrics plugin on Jenkins, open the network path from your EKS cluster, and add Jenkins as a scrape target so Prometheus pulls its metrics automatically.

In this step, get ready to:
  • Install the Prometheus Metrics plugin on your Jenkins instance.
  • Update your Helm values to scrape Jenkins metrics from the external EC2 instance.
  • Verify the Jenkins target appears as UP in the Prometheus Targets UI.
Install the Prometheus Metrics plugin on Jenkins

The Jenkins Prometheus Metrics plugin exposes a /prometheus/ endpoint that Prometheus can scrape. Without this plugin, Jenkins has no metrics endpoint at all.

  • Open your Jenkins instance in a browser by navigating to your Jenkins EC2 instance's public IP on port 8080.
  • Click Manage Jenkins in the left sidebar.
  • Click Manage Plugins (or Plugins depending on your Jenkins version).
  • Select the Available tab.
  • In the search box, type Prometheus metrics.
  • Check the box next to Prometheus metrics and click Install.
  • Wait for the installation to complete, then restart Jenkins if prompted.

After the plugin installs, verify the metrics endpoint is active by visiting your Jenkins URL followed by /prometheus/ in your browser.

  • Navigate to http://<your-jenkins-ip>:8080/prometheus/ in your browser.
  • Confirm you see a wall of text with metric lines starting with default_jenkins.

What are these metrics?

The plugin exposes build duration, success/failure counts, queue length, and more. All metrics are prefixed with default_jenkins by default. You'll use these in Grafana dashboards later.

Trailing slash matters

The endpoint must end with a trailing slash (/prometheus/). Without it, Jenkins returns a 302 redirect that Prometheus cannot follow.

Not seeing metrics at /prometheus/?

Make sure the plugin installation completed and Jenkins was restarted. If the page returns a 404, go back to Manage Jenkins > Installed plugins and confirm Prometheus metrics appears in the list.

help me troubleshoot why my Jenkins /prometheus/ endpoint isn't working

Open the network path and add the scrape configuration

Prometheus runs inside your EKS cluster. For it to reach Jenkins on a separate EC2 instance, the Jenkins security group must allow inbound traffic on port 8080 from your EKS cluster's VPC CIDR range.

  • Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
  • In the left navigation pane, click Security Groups.
  • Select the security group attached to your Jenkins EC2 instance.
  • Click the Inbound rules tab, then click Edit inbound rules.
  • Click Add rule.
  • Set Type to Custom TCP.
  • Set Port range to 8080.
  • Set Source to your EKS cluster's VPC CIDR (for example, 10.0.0.0/16). You can find this in the VPC console under your cluster's VPC details.
  • Click Save rules.

How do I find my EKS VPC CIDR?

Open the VPC console, find the VPC your EKS cluster is running in, and look at the IPv4 CIDR column. This is the range you enter as the source for the inbound rule.

Now update your values.yaml to tell Prometheus where to find Jenkins.

  • Open your values.yaml file and find the prometheusSpec section.
  • Find this line:
    retention: 15d
  • Add the following additionalScrapeConfigs block directly below retention: 15d, at the same indentation level:
    additionalScrapeConfigs:
      - job_name: 'jenkins'
        metrics_path: '/prometheus/'
        static_configs:
          - targets: ['[[JENKINS_EC2_IP="your Jenkins EC2 public IP"]]:8080']

What does this configuration do?

  • job_name: 'jenkins' gives this scrape target a label you can filter by in Grafana.
  • metrics_path: '/prometheus/' tells Prometheus where the metrics endpoint lives on Jenkins. The trailing slash is required.
  • static_configs lists the target host and port. Since Jenkins is outside the cluster, you use its EC2 public IP directly.
  • Save values.yaml.

✔️ Awesome, I've got everything!

Great. Make sure you saved the file before running the Helm upgrade next.

ⓧ I'd like to double check the full code

# kube-prometheus-stack custom values
grafana:
  persistence:
    enabled: true
    size: 5Gi
alertmanager:
  enabled: true
prometheus:
  prometheusSpec:
    retention: 15d
    additionalScrapeConfigs:
      - job_name: 'jenkins'
        metrics_path: '/prometheus/'
        static_configs:
          - targets: ['<JENKINS_EC2_IP>:8080']
Run Helm upgrade and verify the Jenkins target

With the security group open and the scrape config saved, apply the changes to your Helm release.

  • Apply the updated values by running this command:
helm upgrade monitoring prometheus-community/kube-prometheus-stack -n monitoring -f values.yaml

What does helm upgrade do?

This command takes your updated values.yaml and re-renders the Helm templates. Prometheus picks up the new additionalScrapeConfigs entry and begins scraping the Jenkins endpoint on its next cycle.

You should see output confirming the release was upgraded successfully.

Seeing a connection refused or timeout error?

Double check that your security group rule saved correctly. Go back to the EC2 console, select the Jenkins security group, and confirm the inbound rule shows TCP port 8080 with your VPC CIDR as the source.

Also confirm you used the correct public IP for your Jenkins EC2 instance in values.yaml.

help me troubleshoot why Prometheus cannot reach my Jenkins EC2 instance

Now verify that Prometheus is successfully scraping Jenkins. Before you run the port-forward command below, what do you expect to see in the Prometheus Targets page for the Jenkins job?

  • Port-forward the Prometheus UI by running:
kubectl port-forward svc/monitoring-kube-prometheus-prometheus 9090:9090 -n monitoring
  • Open http://localhost:9090 in your browser.
  • Click Status in the top navigation bar.
  • Click Targets.
  • Find the jenkins job in the list.
  • Confirm it shows a state of UP with a recent Last Scrape timestamp.

The jenkins target shows UP. Prometheus is now pulling metrics from your CI server every scrape interval.

What if the target shows DOWN?

A DOWN state means Prometheus attempted to reach Jenkins but failed. Click on the target's error column to see the specific error message. Common causes are a wrong IP address in values.yaml, a missing trailing slash on the metrics path, or the security group rule not yet applied.

Your Jenkins CI server is no longer a blind spot. Prometheus is scraping build metrics in real time. Next up, you'll configure Alertmanager to send you email alerts when something goes wrong.

Configure Alertmanager with Gmail Notifications

Your Grafana dashboards give you visibility into the cluster and Jenkins pipeline. But dashboards only help when someone is actively watching them.

When a pod crashes at 2 AM, you need a notification in your inbox. Alertmanager is the component that routes alerts from Prometheus to external notification channels like email. In this step, you will connect it to your Gmail account so alerts arrive automatically.

In this step, get ready to:
  • Generate a Gmail App Password for SMTP authentication.
  • Create a Kubernetes Secret containing the Alertmanager Gmail receiver configuration.
  • Verify the Gmail receiver appears in the Alertmanager Status UI.
Generate a Gmail App Password

Alertmanager needs to authenticate with Gmail's SMTP server to send alert emails. Gmail blocks basic username/password logins from third-party apps. Instead, you generate a special 16-digit App Password that gives Alertmanager permission to send emails on your behalf.

  • Open your browser and go to https://myaccount.google.com.
  • Click Security in the left sidebar.
  • Under the How you sign in to Google section, click 2-Step Verification.
  • Scroll to the bottom of the page and click App passwords.
  • In the App name field, type Alertmanager and click Create.
  • Copy the 16-digit password that appears. Save it somewhere safe because you will not be able to see it again.

What is an App Password?

An App Password is a 16-digit passcode that gives a third-party app permission to access your Google Account. It requires 2-Step Verification to be enabled first.

Alertmanager cannot use Google's OAuth flow, so it authenticates via SMTP with this App Password instead of your real account password.

Don't see the App passwords option?

Make sure 2-Step Verification is turned on for your account. The App passwords option only appears after 2-Step Verification is enabled.

If you are using a work, school, or organization account, App passwords may be disabled by your admin. Use a personal Gmail account instead.

help me find the App passwords option in my Google Account settings

Create the Alertmanager configuration secret

The Prometheus Operator reads Alertmanager's configuration from a Kubernetes Secret. You will create an alertmanager.yaml file with the Gmail receiver config, then apply it as a Secret that the operator picks up automatically.

  • Create a file called alertmanager.yaml with the following Alertmanager configuration:
# Alertmanager configuration with Gmail receiver
global:
  resolve_timeout: 5m
route:
  receiver: 'gmail'
  group_by: ['alertname', 'namespace']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
receivers:
  - name: 'gmail'
    email_configs:
      - to: 'your-email@gmail.com'
        from: 'your-email@gmail.com'
        smarthost: 'smtp.gmail.com:587'
        auth_username: 'your-email@gmail.com'
        auth_identity: 'your-email@gmail.com'
        auth_password: 'your-app-password-here'

What does this configuration do?

  • global.resolve_timeout sets how long Alertmanager waits before marking an alert as resolved if it stops firing.
  • route defines how incoming alerts are grouped and which receiver handles them. All alerts go to the gmail receiver.
  • group_by groups alerts by name and namespace so you get one email per logical issue instead of dozens of individual alerts.
  • smarthost points to Gmail's SMTP server at smtp.gmail.com:587. The auth_password is your 16-digit App Password.
  • Replace every instance of your-email@gmail.com with your actual Gmail address.
  • Replace your-app-password-here with the 16-digit App Password you generated in the previous substep (remove any spaces).
  • Save the file.

Keep your App Password safe

Your App Password is stored in plain text inside this file. Do not commit it to a public Git repository. In production, you would store credentials in a sealed secret or external secrets manager.

Now you need to apply this configuration as a Kubernetes Secret with the specific name that the Prometheus Operator expects.

  • Create the Alertmanager configuration secret by running this command:
kubectl create secret generic alertmanager-monitoring-kube-prometheus-alertmanager --from-file=alertmanager.yaml -n monitoring --dry-run=client -o yaml | kubectl apply -f -

Why this specific secret name?

The Prometheus Operator requires the configuration secret to be named alertmanager-{ALERTMANAGER_NAME}. For the kube-prometheus-stack chart with release name "monitoring", the Alertmanager resource is named monitoring-kube-prometheus-alertmanager.

The secret must contain a key called alertmanager.yaml with the configuration data. The --from-file flag sets the key name to the filename automatically.

You should see output confirming the secret was created or configured.

Verify the Gmail receiver in the Alertmanager UI

The Prometheus Operator detects the updated secret and automatically reloads Alertmanager. Let's confirm the new configuration is active.

  • Wait about 30 seconds for the operator to reload, then confirm the alertmanager pod is running:
kubectl get pods -n monitoring -l app.kubernetes.io/name=alertmanager

You should see the alertmanager pod in Running status with all containers ready.

  • Port-forward the Alertmanager service by running:
kubectl port-forward svc/monitoring-kube-prometheus-alertmanager 9093:9093 -n monitoring

Before you open the UI, what do you expect to see on the Status page? Think about what receiver name should appear in the route configuration.

  • Open http://localhost:9093 in your browser.
  • Click Status in the top navigation bar.

You should see your gmail receiver listed under the route tree configuration. The route section shows receiver: gmail with your email address in the email_configs section.

Don't see the gmail receiver on the Status page?

  • Verify the secret was created with the correct name by running kubectl get secret -n monitoring | grep alertmanager. You should see alertmanager-monitoring-kube-prometheus-alertmanager in the list.
  • Check the secret contains valid YAML by running kubectl get secret alertmanager-monitoring-kube-prometheus-alertmanager -n monitoring -o jsonpath="{.data.alertmanager\.yaml}" | base64 --decode. Confirm the decoded output matches your configuration.
  • If the configuration still shows the old default, try restarting the alertmanager pod by running kubectl delete pod -n monitoring -l app.kubernetes.io/name=alertmanager. The StatefulSet recreates it with the updated secret.

help me troubleshoot why my Alertmanager gmail receiver is not showing up after creating the secret

✔️ Awesome, I've got everything!

Great. Double check the Alertmanager Status page shows the gmail receiver and that your email address appears in the configuration.

ⓧ I'd like to double check the full code

Here is the complete alertmanager.yaml file you created. Confirm yours matches this structure (with your own email and App Password substituted in):

# Alertmanager configuration with Gmail receiver
global:
  resolve_timeout: 5m
route:
  receiver: 'gmail'
  group_by: ['alertname', 'namespace']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
receivers:
  - name: 'gmail'
    email_configs:
      - to: 'your-email@gmail.com'
        from: 'your-email@gmail.com'
        smarthost: 'smtp.gmail.com:587'
        auth_username: 'your-email@gmail.com'
        auth_identity: 'your-email@gmail.com'
        auth_password: 'your-app-password-here'

Your values.yaml remains unchanged from the previous step. The Alertmanager config lives in its own Kubernetes Secret, separate from the Helm values.

Alertmanager is now wired up to your Gmail. Next up, you will create custom alert rules that tell Prometheus when to fire alerts, then trigger a real one to prove the full chain works end-to-end.

Create Alert Rules and Trigger a Real Alert

Your Alertmanager is configured and ready to send Gmail notifications. Now you need to tell it what to actually watch for.

The kube-prometheus-stack ships with default alerting rules that cover extreme infrastructure cases, but those rules know nothing about your specific application or Jenkins pipeline. Custom PrometheusRule resources let you define alerts tailored to your delivery chain. In this step, you will create those rules and prove the full alert chain works end-to-end.

In this step, get ready to:
  • Create a PrometheusRule YAML defining alerts for pod crash loops, high memory usage, and Jenkins build failures.
  • Apply the rule and verify it appears in the Prometheus Alerts UI.
  • Trigger a real alert by killing a pod and confirm the email arrives in your Gmail inbox.
Create the PrometheusRule YAML file

A PrometheusRule is a Custom Resource Definition (CRD) installed by the Prometheus Operator. It lets you declare alerting rules as Kubernetes resources. The Prometheus Operator automatically picks them up and loads them into Prometheus without any manual reload.

  • Create a new file called prometheus-rules.yaml and add the file header plus your first alert rule by pasting the following code:
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: attendance-tracker-alerts
  namespace: monitoring
spec:
  groups:
    - name: attendance-tracker.rules
      rules:
        - alert: PodCrashLooping
          expr: rate(kube_pod_container_status_restarts_total{namespace="attendance-tracker"}[5m]) > 0
          for: 2m
          labels:
            severity: critical
          annotations:
            summary: "Pod crash looping in attendance-tracker namespace"
            description: "Pod {{ $labels.pod }} is crash looping."

What does this code do?

The metadata places this rule in the monitoring namespace, which is where the Prometheus Operator looks for rules by default.

The PodCrashLooping alert uses the kube_pod_container_status_restarts_total metric (provided by kube-state-metrics) and fires if any pod in the attendance-tracker namespace restarts within a 5-minute window. The for: 2m clause means the condition must hold for 2 minutes before the alert transitions from pending to firing.

  • Add the remaining two alert rules below the PodCrashLooping block in the same file:
        - alert: HighMemoryUsage
          expr: container_memory_usage_bytes / container_spec_memory_limit_bytes > 0.8
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "High memory usage detected"
            description: "Container {{ $labels.container }} memory usage exceeds 80%."
        - alert: JenkinsBuildFailure
          expr: default_jenkins_builds_last_build_result{result!="SUCCESS"} == 1
          for: 1m
          labels:
            severity: warning
          annotations:
            summary: "Jenkins build failure detected"
            description: "Jenkins job {{ $labels.jenkins_job }} last build failed."

What do these alerts watch for?

HighMemoryUsage fires when any container uses more than 80% of its memory limit for 5 minutes. This gives you early warning before out-of-memory kills happen.

JenkinsBuildFailure uses the default_jenkins_builds_last_build_result metric from the Jenkins Prometheus plugin. It fires when the last build result is not a success, so you get notified about pipeline failures without checking the Jenkins UI.

  • Save prometheus-rules.yaml.

✔️ Awesome, I've got everything!

Great. Double check you have saved the file.

ⓧ I'd like to double check the full code

Your prometheus-rules.yaml should have the following sections in order. Scroll up and confirm each block is present:

  • The file header (apiVersion, kind, metadata, and spec.groups opening) plus the PodCrashLooping alert.
  • The HighMemoryUsage and JenkinsBuildFailure alerts immediately following, at the same indentation level as PodCrashLooping.
Apply the PrometheusRule and verify in Prometheus
  • Apply the rule to your cluster by running:
kubectl apply -f prometheus-rules.yaml

You should see output confirming the resource was created.

  • Port-forward to the Prometheus UI so you can verify the rule loaded by running:
kubectl port-forward svc/monitoring-kube-prometheus-prometheus 9090:9090 -n monitoring
  • Open http://localhost:9090 in your browser.
  • Click the Alerts tab in the top navigation.
  • Look for the attendance-tracker.rules group. You should see your three alerts listed: PodCrashLooping, HighMemoryUsage, and JenkinsBuildFailure. All three will show as inactive (green) because nothing is currently triggering them.

Don't see your alerts in the list?

  • Make sure the namespace in your YAML metadata is set to monitoring. The Prometheus Operator only discovers rules in namespaces that match its ruleNamespaceSelector. The monitoring namespace is selected by default.
  • Verify the YAML indentation is correct. YAML is sensitive to spaces. Run kubectl get prometheusrules -n monitoring and confirm attendance-tracker-alerts appears.
  • It may take 30-60 seconds for the Prometheus Operator to reload the rule. Refresh the Alerts page.

Why aren't my PrometheusRule alerts showing in the Prometheus UI?

Trigger a real alert and receive it in Gmail

Your alerts are loaded, but so far they are sitting quietly because nothing is wrong. Time to intentionally break something and prove the full notification chain works end-to-end.

  • Open a new terminal tab (leave the Prometheus port-forward running).
  • List the pods in your attendance-tracker namespace by running:
kubectl get pods -n attendance-tracker
  • Copy the pod name from the output. It will look something like attendance-tracker-xxxxxxxxxx-xxxxx.

Before you run the next command, think about what you expect to happen. You are about to delete the pod. Do you think Kubernetes will restart it? How long do you think it will take for the alert to fire and an email to land in your inbox?

  • Delete the pod to simulate a crash by running:
kubectl delete pod [[POD_NAME="your attendance-tracker pod name from above"]] -n attendance-tracker

What happens next?

Kubernetes will automatically restart the pod (because it is managed by a Deployment). The restart increments the kube_pod_container_status_restarts_total counter. After the for: 2m duration passes, the PodCrashLooping alert transitions from pending to firing.

Once firing, Prometheus sends the alert to Alertmanager. Alertmanager routes it through your Gmail receiver and delivers the email. The entire chain typically takes 2-5 minutes.

  • Wait 2-5 minutes. While waiting, you can watch the alert state change in the Prometheus Alerts tab. Refresh the page periodically. You will first see PodCrashLooping turn yellow (pending), then red (firing).
  • Check your Gmail inbox. Look for an email from Alertmanager with a subject containing PodCrashLooping and details about your attendance-tracker pod.

That email proves the full observability chain is working. A metric was collected, evaluated against your rule, transitioned through pending to firing, routed through Alertmanager, and delivered to your inbox.

No email after 5 minutes?

  • Check the Alertmanager UI at http://localhost:9093 (port-forward if needed). Look under the Alerts tab to see if the alert arrived there.
  • If the alert is in Alertmanager but no email arrived, check the Gmail App Password is correct. Re-check the Alertmanager config secret you created in the previous step.
  • If the alert is not in Alertmanager, verify it shows as firing in the Prometheus Alerts tab. If it is still pending, wait longer. The for: 2m clause means 2 full minutes must pass before firing.
  • Check your Gmail spam folder. Alertmanager emails sometimes land there on first delivery.

My PodCrashLooping alert is not firing or I am not receiving the email notification. Help me debug the alert chain.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. The monitoring stack runs as pods on your existing EKS nodes, so it does not add new EC2 instances. The incremental cost from this project is the EBS volumes provisioned for Prometheus and Grafana persistence.

Cost warning

Your EKS cluster control plane costs approximately $0.10/hr, and EC2 node costs continue based on instance type. The additional EBS volumes for Prometheus and Grafana persistence add a few cents per GB per month.

If you are not actively using the monitoring stack, consider pausing (scaling nodes to 0) or deleting the Helm release to stop accruing storage costs.

Resources you used:

  • kube-prometheus-stack Helm release in the monitoring namespace (Prometheus, Grafana, Alertmanager, node-exporter, kube-state-metrics pods).
  • PersistentVolumeClaims for Prometheus data and Grafana storage (EBS volumes in AWS).
  • Kubernetes Secret for Alertmanager Gmail configuration (alertmanager-config).
  • PrometheusRule custom resources (attendance-tracker-alerts and app-error-rate-alert).
  • ServiceMonitor for attendance-tracker (attendance-tracker-monitor).
  • Jenkins EC2 Security Group inbound rule (port 8080 from EKS VPC CIDR).
  • Jenkins Prometheus Metrics plugin installed on your Jenkins instance.
  • Gmail App Password generated for Alertmanager SMTP.

Keep everything running

No action needed. Choose this if you are still actively building on your CI/CD pipeline and want continuous observability. Your dashboards and alerts remain functional.

  • Grafana dashboards remain accessible via port-forward whenever you need them.
  • Alertmanager continues routing alerts to your Gmail inbox.
  • The Jenkins Prometheus Metrics plugin stays active on your EC2 instance.
  • Your Gmail App Password remains valid until you revoke it.

Pause - I'll come back to this later

Scale down your EKS node group to 0 nodes. The monitoring pods will stop running, but PersistentVolumeClaims retain your Prometheus data and Grafana dashboards. Scale back up when you are ready to resume.

  • In the AWS Console, navigate to EKS > your cluster > Compute > your node group > Edit.
  • Set the Minimum size, Desired size, and Maximum size all to 0.
  • Click Save changes.
  • Your EBS volumes persist and you only pay storage costs (a few cents per GB per month) while nodes are at 0.
  • When you are ready to resume, scale the node group back up and your monitoring pods will restart with their data intact.

Delete - I don't want to use this again

Remove all monitoring resources from your cluster and revert the changes made during this project. Your Jenkins EC2 instance and EKS cluster remain as they were before this project.

  • Uninstall the kube-prometheus-stack Helm release by running:
  • This removes all monitoring pods, services, and Helm-managed resources.
helm uninstall monitoring -n monitoring
  • Delete the entire monitoring namespace (removes PVCs, secrets, PrometheusRules, and the ServiceMonitor) by running:
kubectl delete namespace monitoring
  • The associated EBS volumes will be automatically deleted if your storage class uses the default Delete reclaim policy.

Remove the Jenkins Security Group rule:

  • In the AWS Console, navigate to EC2 > Security Groups.
  • Select the Security Group attached to your Jenkins EC2 instance.
  • Click Edit inbound rules.
  • Delete the rule that allows TCP port 8080 from the EKS cluster VPC CIDR.
  • Click Save rules.

Uninstall the Jenkins Prometheus Metrics plugin:

  • In Jenkins, go to Manage Jenkins > Plugins > Installed tab.
  • Find Prometheus metrics and click Uninstall.
  • Restart Jenkins when prompted for the change to take effect.

Revoke your Gmail App Password:

  • Go to myaccount.google.com/apppasswords.
  • Find the app password you created for Alertmanager.
  • Click the trash icon next to it to revoke it immediately.

Nice Work!

Nice Work!

You just built a production-style observability layer on top of your existing CI/CD pipeline. Your EKS cluster, Jenkins server, and attendance-tracker app are all monitored, dashboarded, and wired to alert you the moment something breaks.

You've learned how to:

  • Deploy a full kube-prometheus-stack (Prometheus, Grafana, Alertmanager, node-exporter, kube-state-metrics) to your EKS cluster using Helm and explore pre-built dashboards showing real-time cluster health.
  • Configure Prometheus to scrape Jenkins pipeline metrics from an external EC2 instance using the Prometheus Metrics plugin and additionalScrapeConfigs, then visualize build success, failure, and duration in Grafana.
  • Set up Alertmanager with Gmail SMTP to route alerts to your inbox, create custom PrometheusRule definitions, and prove the full chain works end-to-end by triggering a real alert and receiving it via email.
  • Secret Mission: Instrument your Python app with prometheus_client to expose custom metrics, create a ServiceMonitor for auto-discovery, and build a dedicated Grafana dashboard for application-level observability.

Ready to quiz yourself?