Monitor a Kubernetes App with Prometheus

Deploy a full Prometheus and Grafana monitoring stack on Kubernetes.

Introduction

30 Second Summary

When an app goes down in production, the first question is always "what happened?" Without monitoring, you're flying blind. Observability tools give you real-time visibility into how your applications are performing so you can catch problems before users do.

In this project, you will deploy a full monitoring stack on your local Kubernetes cluster, instrument a Python app with custom metrics, and build a live dashboard to visualize your app's performance.

What You'll Build

A complete Kubernetes observability pipeline using Prometheus for metrics collection and Grafana for visualization, monitoring a custom Flask API you deploy yourself.

By the end of this project, you'll have:

  • A running kube-prometheus-stack (Prometheus, Grafana, Alertmanager) deployed to your cluster using Helm.
  • A containerized Flask API exposing a custom /metrics endpoint with request count and latency histogram metrics.
  • A Kubernetes ServiceMonitor that auto-discovers your app and a custom Grafana dashboard visualizing request rate and latency percentiles using PromQL.
  • Secret Mission: Create a PrometheusRule alert that fires when your app's error rate exceeds a threshold.

Are there any prerequisites?

This project assumes you're familiar with Docker and Kubernetes basics (deployments, services, pods, namespaces). You should have prior experience containerizing a Python app.

Before We Start

Before you can monitor a Kubernetes app, you need the right toolkit in place.

In this step, you'll install Helm (a package manager for Kubernetes), confirm your cluster is healthy, and create the project directory for your Flask app files.

In this step, get ready to:
  • Install Helm, the package manager for Kubernetes.
  • Verify your Docker Desktop Kubernetes cluster is running.
  • Create a project directory for your Flask app files.
Install Helm

Helm lets you install complex Kubernetes applications with a single command. You'll use it in the next step to deploy an entire monitoring stack.

  • Open your terminal by pressing Cmd+Space (macOS) or the Windows key (Windows), typing Terminal or PowerShell, and pressing Enter.
  • Check if Helm is already on your machine by running:
helm version

✔️ I see a version number

Helm is ready to go. Skip ahead to the next substep (Verify your Kubernetes cluster).

ⓧ Command not found

Helm is not installed yet. Follow the instructions for your operating system below.

macOS

  • Install Helm using Homebrew by running this command:
brew install helm
  • Verify Helm installed successfully by running:
helm version

You should see version information including v4.2.2 or similar.

Windows

  • Install Helm by running this command:
winget install Helm.Helm
  • Close your terminal and reopen it for the installation to take effect.
  • Verify Helm installed successfully by running:
helm version

You should see version information including v4.2.2 or similar.

Helm install not working?

On Windows, make sure you run your terminal as Administrator if winget is not recognized. If helm version still fails after installing, confirm you closed and reopened your terminal.

On macOS, if brew is not found, install Homebrew first from https://brew.sh.

Help me troubleshoot my Helm installation

Verify your Kubernetes cluster

Docker Desktop includes a built-in single-node Kubernetes cluster. You need to confirm it's running before deploying anything to it.

  • Confirm your cluster is reachable by running:
kubectl cluster-info

You should see output showing the Kubernetes control plane is running at a local address.

  • Check your node status by running:
kubectl get nodes

You should see one node with status Ready.

Node not showing Ready?

Open Docker Desktop and go to Settings then Kubernetes. Make sure the Enable Kubernetes checkbox is ticked and the status indicator shows a green dot.

If you just enabled Kubernetes, it may take a minute or two to start. Wait and run kubectl get nodes again.

Help me get my Kubernetes node to Ready status

Create your project directory

You need a dedicated folder to hold your Flask application code, Dockerfile, and Kubernetes manifests.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir flask-metrics-app
cd flask-metrics-app
  • Confirm the folder was created by running:
pwd

You should see a path ending in flask-metrics-app, confirming you're in the right place.

Why this directory?

Keeping all your project files in one folder makes it easy to build a Docker image and apply Kubernetes manifests later. Everything you create in the next steps lives inside flask-metrics-app.

Your environment is ready. Next up, you'll deploy a full Prometheus and Grafana monitoring stack to your cluster with a single Helm command.

Deploy the Monitoring Stack with Helm

Your tools are ready and your cluster is online. Now it's time to deploy the actual monitoring infrastructure that will collect and visualize your app's metrics.

The industry-standard observability stack combines Prometheus (for scraping and storing metrics) with Grafana (for visualizing them in dashboards). Instead of installing each piece manually, you'll use a single Helm chart that deploys everything together.

In this step, get ready to:
  • Add the Prometheus Community Helm repository to your machine.
  • Install the kube-prometheus-stack (Prometheus, Grafana, Alertmanager, and more) in one command.
  • Verify all monitoring pods are running in your cluster.
Add the Prometheus Community Helm repository

Helm charts are hosted in remote repositories. Before you can install the kube-prometheus-stack chart, you need to tell Helm where to find it.

  • Add the Prometheus Community chart repository and refresh your local index by running these commands:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

What do these commands do?

  • helm repo add registers the Prometheus Community repository under the name prometheus-community on your machine. Think of it like bookmarking a package source.
  • helm repo update downloads the latest list of available charts from all your registered repos. This ensures you get the newest chart version when you install.
  • You should see "prometheus-community" has been added to your repositories followed by a successful update message.

Seeing a connection error?

Make sure you have an active internet connection. Helm needs to reach the remote repository URL to download the chart index.

If you see a message that the repo already exists, that's fine. Run helm repo update to refresh it.

Help me troubleshoot my Helm repo add command

Install the kube-prometheus-stack

The kube-prometheus-stack chart bundles Prometheus, Grafana, Alertmanager, kube-state-metrics, and node-exporter into a single deployment. One command gives you a complete monitoring platform.

  • Install the full monitoring stack into a dedicated namespace by running this command:
helm install monitoring prometheus-community/kube-prometheus-stack --namespace monitoring --create-namespace --set grafana.adminPassword=admin --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false

What does this command do?

  • helm install monitoring creates a Helm release named monitoring. This name is used to reference all the resources Helm creates.
  • --namespace monitoring --create-namespace deploys everything into a dedicated monitoring namespace, creating it if it doesn't exist. This keeps monitoring resources separate from your application.
  • --set grafana.adminPassword=admin sets the Grafana login password to admin. You'll use this to log in later.
  • --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false tells Prometheus to discover ServiceMonitors across all namespaces, not just ones with the chart's default labels. This makes it easier to connect your app later.

This command takes a moment to run. You should see a summary of the installed release with status deployed when it completes.

Why use a Helm chart instead of manual YAML?

The kube-prometheus-stack chart installs dozens of Kubernetes resources (Deployments, Services, ConfigMaps, CRDs, and more). Writing all that YAML by hand would take hours and be error-prone.

Helm charts are the industry-standard way to deploy complex applications to Kubernetes. DevOps and SRE job postings regularly list Helm as a required skill.

Seeing a timeout or connection error?

Make sure Docker Desktop is running and Kubernetes is enabled. Check with kubectl cluster-info to confirm your cluster is accessible.

If you see a "cannot re-use a name" error, it means a release called monitoring already exists. Run helm uninstall monitoring -n monitoring first, then try the install again.

Help me fix my Helm install error

Verify the monitoring pods are running

The Helm chart creates multiple pods that need time to pull their container images and start up. On Docker Desktop, this typically takes 2-3 minutes.

  • Check the status of all pods in the monitoring namespace by running:
kubectl get pods -n monitoring

What should I see?

You should see pods for each component of the monitoring stack. Look for names containing: prometheus, grafana, alertmanager, kube-state-metrics, and node-exporter.

Every pod's STATUS column should show Running or Completed. If some show ContainerCreating or Pending, wait another minute and run the command again.

  • Wait until all pods show Running or Completed status. Re-run kubectl get pods -n monitoring every 30 seconds until they're all up.
  • Confirm you see at least 5-6 pods total, covering prometheus, grafana, alertmanager, kube-state-metrics, and node-exporter.

Pods stuck in Pending or CrashLoopBackOff?

If pods remain in Pending for more than 3 minutes, your Docker Desktop might not have enough memory allocated. Open Docker Desktop settings and increase the memory to at least 4 GB.

If a pod shows CrashLoopBackOff, check its logs with kubectl logs <pod-name> -n monitoring (replace <pod-name> with the actual pod name from your list).

Help me debug my monitoring pods that aren't starting

Your monitoring infrastructure is live. Prometheus, Grafana, and Alertmanager are all running in your cluster, ready to collect metrics. Next, you'll build a Flask application that exposes custom metrics for Prometheus to scrape.

Deploy an Instrumented Flask App

Your monitoring stack is live. Prometheus, Grafana, and Alertmanager are all running and ready to collect metrics.

Now you need something to monitor. In this step, you will build a Flask API that exposes custom Prometheus metrics, containerize it with Docker, and deploy it to your Kubernetes cluster.

In this step, get ready to:
  • Create a Flask app instrumented with Prometheus Counter and Histogram metrics.
  • Containerize the app with a Dockerfile and build a Docker image.
  • Deploy the app to Kubernetes and confirm it is running.
Create the Flask app with Prometheus metrics

For Prometheus to monitor your application, the app needs to expose its own metrics in a format Prometheus understands. You will instrument a Flask API with two metric types that capture how your app is performing.

  • Create a new file called app.py in your flask-metrics-app folder. Paste in the following code to set up the app and define two Prometheus metrics:
import time
import random
from flask import Flask, Response
from prometheus_client import Counter, Histogram, generate_latest, CONTENT_TYPE_LATEST

# Create the Flask app
app = Flask(__name__)

# Counter metric: tracks total requests by method, endpoint, and status code
REQUEST_COUNT = Counter(
    "flask_request_count_total",
    "Total number of requests",
    ["method", "endpoint", "status"]
)

# Histogram metric: tracks how long each request takes to process
REQUEST_DURATION = Histogram(
    "flask_request_duration_seconds",
    "Request duration in seconds",
    ["method", "endpoint"]
)

What does this code do?

REQUEST_COUNT is a Counter. It only goes up, tracking the total number of requests your app receives. The labels (method, endpoint, status) let you slice the data later.

REQUEST_DURATION is a Histogram. It records how long each request takes, automatically bucketing response times so you can calculate percentiles like p95 latency.

What is a Counter vs a Histogram?

A Counter is a cumulative metric that only increases. It answers "how many times has this happened?" A Histogram samples observations (like request durations) and counts them in configurable buckets. It answers "how long did things take?"

  • Below the metric definitions in the same file, add the route handlers that use those metrics:

@app.route("/")
def home():
    start = time.time()
    response_text = "Hello from Flask metrics app!"
    duration = time.time() - start
    # Increment the request counter with labels for this endpoint
    REQUEST_COUNT.labels(method="GET", endpoint="/", status="200").inc()
    # Record how long this request took
    REQUEST_DURATION.labels(method="GET", endpoint="/").observe(duration)
    return response_text, 200


@app.route("/items")
def items():
    start = time.time()
    item_list = [{"id": i, "name": f"Item {i}"} for i in range(1, 6)]
    duration = time.time() - start
    REQUEST_COUNT.labels(method="GET", endpoint="/items", status="200").inc()
    REQUEST_DURATION.labels(method="GET", endpoint="/items").observe(duration)
    return {"items": item_list}, 200


@app.route("/slow")
def slow():
    start = time.time()
    # Simulate a slow operation with random delay
    delay = random.uniform(0.5, 2.0)
    time.sleep(delay)
    duration = time.time() - start
    REQUEST_COUNT.labels(method="GET", endpoint="/slow", status="200").inc()
    REQUEST_DURATION.labels(method="GET", endpoint="/slow").observe(duration)
    return {"message": "slow response", "delay": round(delay, 2)}, 200


# Expose all collected metrics at /metrics for Prometheus to scrape
@app.route("/metrics")
def metrics():
    return Response(generate_latest(), mimetype=CONTENT_TYPE_LATEST)


if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

What do these routes do?

Each route follows the same instrumentation pattern: record the start time, do work, calculate duration, then increment the counter and observe the duration in the histogram.

The /slow endpoint adds a random delay between 0.5 and 2 seconds. This gives you interesting latency data to visualize later in Grafana.

The /metrics endpoint is what Prometheus will scrape. It returns all your metrics in Prometheus exposition format using generate_latest().

  • Save app.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved app.py before moving on.

ⓧ I'd like to double check the full code

import time
import random
from flask import Flask, Response
from prometheus_client import Counter, Histogram, generate_latest, CONTENT_TYPE_LATEST

app = Flask(__name__)

REQUEST_COUNT = Counter(
    "flask_request_count_total",
    "Total number of requests",
    ["method", "endpoint", "status"]
)

REQUEST_DURATION = Histogram(
    "flask_request_duration_seconds",
    "Request duration in seconds",
    ["method", "endpoint"]
)


@app.route("/")
def home():
    start = time.time()
    response_text = "Hello from Flask metrics app!"
    duration = time.time() - start
    REQUEST_COUNT.labels(method="GET", endpoint="/", status="200").inc()
    REQUEST_DURATION.labels(method="GET", endpoint="/").observe(duration)
    return response_text, 200


@app.route("/items")
def items():
    start = time.time()
    item_list = [{"id": i, "name": f"Item {i}"} for i in range(1, 6)]
    duration = time.time() - start
    REQUEST_COUNT.labels(method="GET", endpoint="/items", status="200").inc()
    REQUEST_DURATION.labels(method="GET", endpoint="/items").observe(duration)
    return {"items": item_list}, 200


@app.route("/slow")
def slow():
    start = time.time()
    delay = random.uniform(0.5, 2.0)
    time.sleep(delay)
    duration = time.time() - start
    REQUEST_COUNT.labels(method="GET", endpoint="/slow", status="200").inc()
    REQUEST_DURATION.labels(method="GET", endpoint="/slow").observe(duration)
    return {"message": "slow response", "delay": round(delay, 2)}, 200


@app.route("/metrics")
def metrics():
    return Response(generate_latest(), mimetype=CONTENT_TYPE_LATEST)


if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)
Containerize the app with Docker

Kubernetes runs containers, not raw Python scripts. You need to package your Flask app into a Docker image so Kubernetes can deploy it as a pod.

  • Create a file called requirements.txt in the same flask-metrics-app folder with your Python dependencies:
flask==3.1.3
prometheus-client==0.25.0
  • Create a file called Dockerfile (no extension) in the same folder:
# Use a lightweight Python base image
FROM python:3.12-slim

WORKDIR /app

# Install dependencies first for better Docker layer caching
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy the application code
COPY app.py .

EXPOSE 5000

CMD ["python", "app.py"]

What does this Dockerfile do?

It starts from a slim Python 3.12 image, copies and installs your dependencies first (so Docker can cache that layer), then copies your app code. The container listens on port 5000.

  • Save both requirements.txt and Dockerfile.
  • Build the Docker image by running this command:
docker build -t flask-metrics-app:1.0 .
  • Wait for the build to complete. You should see a success message ending with a line that includes flask-metrics-app:1.0.

Build failing?

  • Make sure Docker Desktop is running. The Docker daemon must be active for builds to work.
  • Check that requirements.txt and Dockerfile are in the same folder as app.py. The . at the end of the build command refers to the current directory.
  • help me debug my Docker build error
Deploy to Kubernetes

With your image built, you can now tell Kubernetes to run it. You need two manifests: a Deployment (which creates the pod) and a Service (which exposes the pod to the cluster network).

  • Create a file called deployment.yaml in your flask-metrics-app folder:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: flask-metrics-app
  labels:
    app: flask-metrics-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: flask-metrics-app
  template:
    metadata:
      labels:
        app: flask-metrics-app
    spec:
      containers:
        - name: flask-metrics-app
          image: flask-metrics-app:1.0
          # Use the locally-built image instead of pulling from a registry
          imagePullPolicy: Never
          ports:
            - containerPort: 5000
              name: http

Why is imagePullPolicy set to Never?

By default, Kubernetes tries to pull images from a container registry like Docker Hub. Since you built the image locally on the same machine running Docker Desktop Kubernetes, the image already exists. Setting imagePullPolicy: Never tells Kubernetes to use the local image instead of trying to pull it.

  • Create a file called service.yaml in the same folder:
apiVersion: v1
kind: Service
metadata:
  name: flask-metrics-app
  labels:
    # This label lets the ServiceMonitor find this service later
    app: flask-metrics-app
spec:
  selector:
    app: flask-metrics-app
  ports:
    - port: 5000
      targetPort: 5000
      name: http
  type: ClusterIP

What does the Service do?

A ClusterIP Service gives your pod a stable internal address inside the cluster. The app: flask-metrics-app label on the Service metadata is important. In the next step, you will create a ServiceMonitor that matches on this label to tell Prometheus where to scrape.

  • Save both deployment.yaml and service.yaml.
  • Apply both manifests to your cluster by running:
kubectl apply -f deployment.yaml -f service.yaml
  • Verify your pod is running by running:
kubectl get pods

You should see a pod with a name starting with flask-metrics-app showing 1/1 under READY and Running under STATUS.

  • Confirm the Service was created by running:
kubectl get svc

You should see flask-metrics-app listed as a ClusterIP service on port 5000.

Pod not showing Running?

  • If the STATUS column shows ErrImageNeverPull, Kubernetes cannot find your local image. Run docker images | grep flask-metrics-app to confirm the image exists. Make sure imagePullPolicy: Never is in your deployment.yaml.
  • If STATUS shows CrashLoopBackOff, the container is starting and crashing. Run kubectl logs followed by the pod name to see the Python error.
  • help me debug my Flask app pod that is not running

✔️ Awesome, I've got everything!

Great. Make sure both deployment.yaml and service.yaml are saved and your pod shows Running.

ⓧ I'd like to double check the full code

apiVersion: apps/v1
kind: Deployment
metadata:
  name: flask-metrics-app
  labels:
    app: flask-metrics-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: flask-metrics-app
  template:
    metadata:
      labels:
        app: flask-metrics-app
    spec:
      containers:
        - name: flask-metrics-app
          image: flask-metrics-app:1.0
          imagePullPolicy: Never
          ports:
            - containerPort: 5000
              name: http
apiVersion: v1
kind: Service
metadata:
  name: flask-metrics-app
  labels:
    app: flask-metrics-app
spec:
  selector:
    app: flask-metrics-app
  ports:
    - port: 5000
      targetPort: 5000
      name: http
  type: ClusterIP

Your instrumented Flask app is live on the cluster. Next up, you will connect Prometheus to your app so it starts collecting those metrics automatically.

Connect Prometheus to Your App

Your Flask app is exposing metrics on its /metrics endpoint, but Prometheus doesn't know it exists yet.

In Kubernetes, the Prometheus Operator uses a custom resource called a ServiceMonitor to automatically discover and scrape targets. In this step, you'll create a ServiceMonitor that points Prometheus at your app, then verify the connection is live.

In this step, get ready to:
  • Create a ServiceMonitor that tells Prometheus where your app's metrics live.
  • Verify your app appears as an active target in the Prometheus UI.
  • Run your first PromQL query against live application data.
Create the ServiceMonitor manifest

A ServiceMonitor is a custom resource definition (CRD) installed by the kube-prometheus-stack. It tells Prometheus which Services to scrape, which port to use, and how often to collect metrics.

  • Create a new file called servicemonitor.yaml in your flask-metrics-app folder and paste the following content:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: flask-metrics-app
  labels:
    release: monitoring
spec:
  selector:
    matchLabels:
      app: flask-metrics-app
  endpoints:
    - port: http
      path: /metrics
      interval: 15s

What does this manifest do?

The release: monitoring label is the key piece. The Prometheus instance deployed by the Helm chart watches for ServiceMonitors carrying this label. Without it, Prometheus ignores your resource entirely.

Under spec.selector.matchLabels, the ServiceMonitor looks for any Kubernetes Service with the label app: flask-metrics-app. Your service from the previous step already has this label.

The endpoints section tells Prometheus to scrape the /metrics path on the port named http every 15 seconds.

  • Save servicemonitor.yaml.
Apply the ServiceMonitor
  • Apply the ServiceMonitor to your cluster by running this command:
kubectl apply -f servicemonitor.yaml
  • You should see output confirming the resource was created:
servicemonitor.monitoring.coreos.com/flask-metrics-app created

Prometheus checks for new ServiceMonitors on a short interval. Wait about 30 to 60 seconds for it to pick up your new target.

Seeing an error about unknown resource type?

If kubectl says the resource type servicemonitors.monitoring.coreos.com is not found, the kube-prometheus-stack CRDs haven't finished installing. Run kubectl get pods -n monitoring and make sure all pods are in Running or Completed state before retrying.

help me figure out why my ServiceMonitor CRD isn't recognized

Verify in the Prometheus UI

Now you'll open the Prometheus web interface to confirm it's actively scraping your Flask app.

  • Port-forward to the Prometheus service by running this command:
kubectl port-forward svc/monitoring-kube-prometheus-stack-prometheus 9090:9090 -n monitoring

What does this command do?

This forwards traffic from localhost:9090 on your machine to the Prometheus service running inside the monitoring namespace. The service name monitoring-kube-prometheus-stack-prometheus follows the Helm chart's naming convention of {release-name}-kube-prometheus-stack-prometheus.

  • Open your browser and go to http://localhost:9090.
  • Click Status in the top navigation bar.
  • Select Targets from the dropdown.
  • Look for a target group called serviceMonitor/default/flask-metrics-app. Confirm that its state shows UP.

Don't see your target listed?

If your Flask app target isn't showing after 60 seconds, check these common causes:

  • Confirm the ServiceMonitor has the label release: monitoring by running kubectl get servicemonitor flask-metrics-app -o yaml and checking the metadata.labels section.
  • Confirm your Flask app service has the label app: flask-metrics-app by running kubectl get svc flask-metrics-app --show-labels.
  • Make sure the port name in your service (http) matches the port name in your ServiceMonitor (http).

help me debug why my Flask app target isn't appearing in Prometheus

Now run a PromQL query to confirm metrics data is flowing.

  • Click the Graph tab at the top of the Prometheus UI (or navigate to http://localhost:9090/graph).
  • Type this query into the expression input field:
flask_request_count_total
  • Click Execute.
  • You should see one or more time series rows in the results table, each labeled with method, endpoint, and status values from your Flask app.

What if the query returns empty?

If no data appears, your app hasn't received any requests yet. Open a new terminal and port-forward to your Flask app:

kubectl port-forward svc/flask-metrics-app 5000:5000

Then visit http://localhost:5000 in your browser. Go back to the Prometheus query page and click Execute again. You should now see data.

✔️ Awesome, I've got everything!

Great. Double check you've saved servicemonitor.yaml and that your target shows UP in the Prometheus UI.

ⓧ I'd like to double check the full code

Here is the complete servicemonitor.yaml for comparison:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: flask-metrics-app
  labels:
    release: monitoring
spec:
  selector:
    matchLabels:
      app: flask-metrics-app
  endpoints:
    - port: http
      path: /metrics
      interval: 15s

Prometheus is now scraping your Flask app every 15 seconds and storing all those metrics in its time-series database. Next up, you'll visualize this data by building a custom Grafana dashboard with real-time panels for request rate and latency.

Build a Grafana Dashboard

Prometheus is now scraping your Flask app and storing its metrics. However, raw metric data sitting in a time-series database is not very useful on its own.

In this step, you will connect to Grafana and build a custom dashboard that visualizes your app's request rate and latency in real time. This is the final piece of the observability pipeline: turning raw numbers into actionable visual insights.

In this step, get ready to:
  • Access Grafana and explore the pre-built Kubernetes dashboards.
  • Create a custom dashboard with Request Rate and Latency Percentiles panels.
  • Generate live traffic and confirm metrics flowing into your dashboard.
Port-forward to Grafana and log in

Grafana runs inside your Kubernetes cluster, so you need to forward its service port to your local machine. The kube-prometheus-stack exposes Grafana on service port 80.

  • Start the port-forward to Grafana by running this command:
kubectl port-forward svc/monitoring-grafana 3000:80 -n monitoring

What does this command do?

This forwards port 3000 on your machine to port 80 on the monitoring-grafana service inside the monitoring namespace. Grafana's web UI becomes accessible at http://localhost:3000.

Keep this terminal running. If you close it, the port-forward stops and you lose access to Grafana.

  • Open your browser and navigate to http://localhost:3000.
  • Log in with username admin and password admin.

You should see the Grafana home page with a welcome screen and navigation menu on the left.

Can't reach localhost:3000?

  • Check that the port-forward command is still running in your terminal. If it exited, re-run it.
  • Make sure nothing else is using port 3000 on your machine. If something is, stop that process first.
  • Confirm the Grafana pod is running with kubectl get pods -n monitoring. Look for a pod with grafana in its name showing Running status.

Why can't I access Grafana at localhost:3000?

Explore the pre-built Kubernetes dashboards

The kube-prometheus-stack comes with dozens of pre-configured dashboards for cluster-level monitoring. These show CPU usage, memory, network, and pod health across your entire cluster.

  • Click Dashboards in the main menu on the left side.
  • Browse through the list of dashboards. You should see names like Kubernetes / Compute Resources / Cluster, Node Exporter / Nodes, and others.
  • Click on any dashboard that interests you to see real-time cluster metrics already being collected.

Why are there dashboards already?

The kube-prometheus-stack Helm chart bundles community-maintained dashboards for common Kubernetes metrics. These are powered by kube-state-metrics and node-exporter, which were deployed alongside Prometheus and Grafana.

These dashboards monitor your cluster infrastructure. Next, you will build a dashboard that monitors your application specifically.

Create a custom dashboard with two panels

Now you will build your own dashboard with two panels: one showing request rate and one showing latency percentiles. These use PromQL queries against the custom metrics your Flask app exposes.

  • Click Dashboards in the main menu.
  • Click New and select New Dashboard.
  • Click the Add new element icon (blue plus sign) and click a panel onto the dashboard.
  • On the new panel, click Configure visualization.

The panel editor opens with a data source pre-selected. You need to make sure Prometheus is the active data source.

  • In the Queries tab at the bottom, confirm the Data source drop-down shows Prometheus. If not, select it from the drop-down.
  • In the query editor, switch to raw query mode (click the Code button if you see a builder view).
  • Enter the following PromQL query for request rate:
rate(flask_request_count_total[1m])

What does this query do?

The rate() function calculates the per-second rate of increase for a counter metric over a 1-minute window. This tells you how many requests per second your app is receiving across all endpoints.

  • Click Refresh to query the data source.
  • Select the Time series visualization if it is not already selected.
  • Enter Request Rate as the panel title.
  • Click Save to save the panel.
  • Click Back to dashboard.

Now add a second panel for latency percentiles.

  • Click the Add new element icon (blue plus sign) again and click a panel onto the dashboard.
  • On the new panel, click Configure visualization.
  • Confirm Prometheus is the data source and switch to Code mode.
  • Enter the following PromQL query for 95th percentile latency:
histogram_quantile(0.95, rate(flask_request_duration_seconds_bucket[1m]))

What does this query do?

The histogram_quantile() function computes a specific percentile from histogram buckets. Here, 0.95 means the 95th percentile. This tells you the maximum response time that 95% of requests complete within.

The /slow endpoint in your app deliberately adds random delay, so you will see this panel reflect higher latency for that endpoint compared to / and /items.

  • Click Refresh to query the data source.
  • Select the Time series visualization.
  • Enter Latency Percentiles as the panel title.
  • Click Save.
  • Click Back to dashboard.
  • Enter a title for the dashboard (e.g. Flask Metrics) and click Save.
  • Click Exit edit.

You should now see your dashboard with two panels. They may show "No data" if your app has not received any recent traffic.

Why is there no data yet?

Prometheus only stores metrics for requests that have actually happened. If no one has hit your Flask endpoints since deployment, there is nothing to graph. The next substep generates traffic to fix this.

Panels show an error instead of data?

  • Check that the data source is set to Prometheus (not the default Grafana data source).
  • Verify the query is entered in Code mode, not the visual query builder mode. The builder sometimes fails to parse raw PromQL.
  • Confirm your Flask app target is still UP in Prometheus. If the previous port-forward to Prometheus (port 9090) is still active, check http://localhost:9090 under Status > Targets.

Why are my Grafana panels showing an error?

Generate traffic and see live data

To see your dashboard come alive, you need to send requests to your Flask app. You will port-forward the Flask service and hit its endpoints.

  • Open a new terminal window (keep the Grafana port-forward running in the other one).
  • Start a port-forward to your Flask app by running this command:
kubectl port-forward svc/flask-metrics-app 5000:5000
  • Open your browser and visit each of these URLs a few times:
  • http://localhost:5000/ to hit the home endpoint.
  • http://localhost:5000/items to hit the items endpoint.
  • http://localhost:5000/slow to hit the slow endpoint (takes 0.5-2 seconds).
  • Refresh each URL several times to generate enough data points.

Generate more interesting data

Hit the /slow endpoint 5-10 times. Each call has a random delay between 0.5 and 2 seconds, which produces a nice spread on the latency panel and makes the p95 calculation meaningful.

  • Switch back to your Grafana browser tab at http://localhost:3000.
  • Navigate to your custom dashboard (click Dashboards in the main menu and select your Flask Metrics dashboard).

You should now see live data in both panels. The Request Rate panel shows lines for each endpoint, and the Latency Percentiles panel shows the 95th percentile response time with the /slow endpoint clearly higher than the others.

Still seeing no data in the panels?

  • Wait 30-60 seconds after generating traffic. Prometheus scrapes every 15 seconds, and the [1m] window needs at least two data points to calculate a rate.
  • Check the time range in Grafana. Click the time picker in the top right and set it to Last 15 minutes.
  • Confirm the Flask port-forward is running and you can still access http://localhost:5000/ in your browser.

Why isn't my Grafana dashboard showing data after generating traffic?

Your observability pipeline is complete. You have metrics flowing from your Flask app, through Prometheus, and into visual dashboards in Grafana. Next up, you will extend this stack with alerting rules in the Secret Mission.

Secret mission

Create an Alert Rule

Your monitoring stack can collect metrics and visualize them. But what about getting notified when something goes wrong? In this secret mission, you'll create a PrometheusRule that automatically fires an alert when your app's error rate crosses a dangerous threshold.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely on Docker Desktop Kubernetes, so there are no ongoing financial costs. The monitoring stack does consume memory and CPU while running.

Resources you used:

  • Kubernetes namespace monitoring with all kube-prometheus-stack components (Prometheus, Grafana, Alertmanager, kube-state-metrics, node-exporter).
  • Flask app Deployment and Service in the default namespace.
  • ServiceMonitor and PrometheusRule custom resources.
  • Docker image flask-metrics-app:1.0.
  • Local flask-metrics-app project folder on your Desktop.

Keep everything running

No action needed. Choose this if you're still actively experimenting with PromQL queries, building more Grafana dashboards, or exploring alerting rules.

  • Your monitoring stack, Flask app, and all custom resources remain running on your Docker Desktop Kubernetes cluster.
  • Helm stays installed on your machine for future Kubernetes projects.
  • The Docker image remains available locally.

Pause - I'll come back to this later

Shut down Docker Desktop to free up memory and CPU. Everything will resume when you restart it.

  • Right-click the Docker Desktop icon in your system tray.
  • Click Quit Docker Desktop.
  • When you reopen Docker Desktop later, your Kubernetes cluster and all deployed resources will start back up automatically.

Delete - I don't want to use this again

Remove all project resources and start fresh. Run the following commands in your terminal to clean up everything.

  • Uninstall the Helm release and delete the monitoring namespace by running:
helm uninstall monitoring -n monitoring
kubectl delete namespace monitoring
  • Delete the Flask app resources from the default namespace by running:
kubectl delete -f deployment.yaml -f service.yaml -f servicemonitor.yaml
  • Remove the CRDs that the kube-prometheus-stack chart installed by running:
kubectl delete crd alertmanagerconfigs.monitoring.coreos.com
kubectl delete crd alertmanagers.monitoring.coreos.com
kubectl delete crd podmonitors.monitoring.coreos.com
kubectl delete crd probes.monitoring.coreos.com
kubectl delete crd prometheusagents.monitoring.coreos.com
kubectl delete crd prometheuses.monitoring.coreos.com
kubectl delete crd prometheusrules.monitoring.coreos.com
kubectl delete crd scrapeconfigs.monitoring.coreos.com
kubectl delete crd servicemonitors.monitoring.coreos.com
kubectl delete crd thanosrulers.monitoring.coreos.com

Why delete the CRDs separately?

Helm does not remove CRDs when you uninstall a chart. These custom resource definitions stay in your cluster until you explicitly delete them. Removing them keeps your cluster clean.

  • Remove the Docker image by running:
docker rmi flask-metrics-app:1.0
  • Navigate up one directory and delete the project folder. On macOS or Linux, run:
cd ..
rm -rf flask-metrics-app
  • On Windows (PowerShell), run:
cd ..
Remove-Item -Recurse -Force flask-metrics-app
  • Confirm the folder is gone by running:
ls

That's a Wrap!

That's a Wrap!

Nice work! You've just built a production-style Kubernetes observability pipeline from scratch. Your Flask app emits metrics, Prometheus scrapes and stores them, and Grafana visualizes everything in real time.

You've learned how to:

  • Deployed a complete Prometheus + Grafana + Alertmanager monitoring stack to Kubernetes using a single Helm command.
  • Instrumented a Flask application with custom Prometheus metrics (Counter and Histogram) and connected it to Prometheus via a ServiceMonitor CRD.
  • Built a custom Grafana dashboard with PromQL queries visualizing live request rate and latency percentiles.
  • Secret Mission: Created a PrometheusRule alert that fires when your app's error rate exceeds a threshold.

Ready to quiz yourself?