Observability Pipeline with OpenTelemetry

Build a local observability stack with traces, metrics, and dashboards.

Introduction

30 Second Summary

When an app slows down in production, someone asks "what happened?" and the team scrambles through log files hoping to find the answer. The difference between a five-minute fix and a five-hour outage often comes down to whether you can see what your system is actually doing.

In this project, you will instrument a Flask API with OpenTelemetry to produce both traces and custom metrics. An OpenTelemetry Collector routes that telemetry to Jaeger for traces and Prometheus for metrics, while Grafana ties it all together with a provisioned dashboard.

What You'll Build

You'll hit a Flask endpoint, watch the request trace appear in Jaeger with per-span timing, then spot a latency spike on your Grafana dashboard and drill straight into the trace that caused it.

By the end of this project, you'll have:

  • An instrumented Flask API where every request produces a trace in Jaeger and custom metrics in Prometheus, all routed through the OpenTelemetry Collector.
  • A live Grafana dashboard showing request rate, error rate, and p95 latency, with the ability to drill from a metric spike into the exact Jaeger trace that caused it.
  • A dashboard provisioned as code from a JSON file that survives container restarts without any manual UI setup.
  • Secret Mission: Add a second Flask service and trace requests across both services with distributed tracing in a single Jaeger view.

Are there any prerequisites?

This project assumes familiarity with Python, Flask, Docker, and Docker Compose. Prior exposure to OpenTelemetry tracing concepts is helpful.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

Taking a moment to understand the full picture will help each step click into place as you go.

Launch the Observability Stack

Before you can observe anything, you need the systems that receive and display telemetry data. Without backends in place, there is nothing to collect your traces or store your metrics.

In this step, you'll launch four services using Docker Compose: the OpenTelemetry Collector (your telemetry router), Jaeger (for traces), Prometheus (for metrics), and Grafana (for dashboards).

In this step, get ready to:
  • Create the project directory and write a Docker Compose file defining four services.
  • Write configuration files for the OTel Collector, Prometheus, and Grafana datasource provisioning.
  • Start the stack and verify all four services are running.
Create the project structure and Docker Compose file

Every service in your observability pipeline will run inside a Docker container, orchestrated by a single docker-compose.yml file. First, you need a project folder to hold everything.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir otel-pipeline
cd otel-pipeline
  • Confirm the folder was created by running:
pwd

You should see a path ending in otel-pipeline.

The Docker Compose file defines the four backend services your pipeline depends on. You will write it in two parts, starting with the telemetry collector and the trace backend.

  • Create a new file called docker-compose.yml in your otel-pipeline folder and paste the following code:
services:
  # Telemetry router - receives data from apps, exports to backends
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.157.0
    command: ["--config", "/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP
      - "8889:8889"   # Prometheus metrics exporter
    depends_on:
      - jaeger

  # Distributed trace storage and UI
  jaeger:
    image: jaegertracing/all-in-one:1.76.0
    ports:
      - "16686:16686"
    environment:
      COLLECTOR_OTLP_ENABLED: "true"

What does this code do?

  • otel-collector runs the OpenTelemetry Collector contrib image. It listens for telemetry on ports 4317 (gRPC) and 4318 (HTTP), and exposes a Prometheus-scrapable endpoint on port 8889.
  • jaeger runs the Jaeger all-in-one image, which bundles the collector, query service, and UI. Port 16686 exposes the Jaeger UI.
  • COLLECTOR_OTLP_ENABLED: "true" tells Jaeger to accept traces in the OTLP format, which is the modern standard.

Now add the metrics and dashboard services. These go in the same file, directly below the jaeger service you just wrote.

  • In docker-compose.yml, add the following below the jaeger service block:
  # Metrics storage and query engine
  prometheus:
    image: prom/prometheus:v3.13.2
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"
    depends_on:
      - otel-collector

  # Dashboard and visualization
  grafana:
    image: grafana/grafana:13.1.1
    ports:
      - "3000:3000"
    environment:
      GF_SECURITY_ADMIN_PASSWORD: admin
      GF_SECURITY_ADMIN_USER: admin
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning
      - ./grafana/dashboards:/etc/grafana/dashboards
    depends_on:
      - prometheus
      - jaeger

What does this code do?

  • prometheus mounts a local config file and exposes the Prometheus query UI on port 9090. It depends on otel-collector because it scrapes metrics from it.
  • grafana mounts two directories: provisioning/ (for datasource and dashboard configs) and dashboards/ (for dashboard JSON files). This is how you configure Grafana as code instead of clicking through the UI.
  • Save docker-compose.yml.
  • Verify the file is valid by running:
docker compose config --services

You should see the four service names printed, one per line: grafana, jaeger, otel-collector, and prometheus.

Seeing a YAML syntax error?

  • YAML is whitespace-sensitive. Make sure every indentation level uses exactly 2 spaces, not tabs.
  • Check that the prometheus and grafana services are indented at the same level as otel-collector and jaeger (two spaces under services:).
  • help me fix a YAML syntax error in my docker-compose.yml

✔️ Awesome, I've got everything!

Great. Make sure you have saved docker-compose.yml before moving on.

ⓧ I'd like to double check the full code

Scroll up and confirm your docker-compose.yml has these sections in order:

  • The services: key at the top.
  • otel-collector service with image, command, volumes, ports (4317, 4318, 8889), and depends_on jaeger.
  • jaeger service with image, port 16686, and COLLECTOR_OTLP_ENABLED environment variable.
  • prometheus service with image, volumes, port 9090, and depends_on otel-collector.
  • grafana service with image, port 3000, admin credentials, two volume mounts, and depends_on prometheus and jaeger.
Write the OTel Collector, Prometheus, and Grafana provisioning configs

Your Docker Compose file references three config files that do not exist yet. Each one tells a service how to behave.

The most important is the OTel Collector config. It defines what telemetry the Collector accepts, how it processes data, and where it sends it.

  • Create a new file called otel-collector-config.yaml in your otel-pipeline folder and paste the following code:
# Accept telemetry via the OTLP protocol
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

# Group data into batches before exporting
processors:
  batch:

exporters:
  # Forward traces to Jaeger over OTLP
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true

  # Expose metrics on port 8889 for Prometheus to scrape
  prometheus:
    endpoint: 0.0.0.0:8889
    namespace: otel

  # Print telemetry to stdout for debugging
  debug:
    verbosity: basic

What does this config do?

The Collector config follows a receivers-processors-exporters pattern.

  • receivers: accepts incoming telemetry over OTLP on both gRPC and HTTP.
  • exporters: defines three destinations. otlp/jaeger forwards traces to Jaeger. prometheus exposes metrics for Prometheus to scrape. debug prints data to the Collector's logs.
  • namespace: otel prefixes all metrics with otel_ so they are easy to find in Prometheus queries later.

The receivers and exporters are defined, but nothing connects them yet. The service.pipelines section wires them together.

  • In otel-collector-config.yaml, add the following below the debug exporter block:
# Wire receivers to exporters through pipelines
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/jaeger, debug]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus, debug]

What are pipelines?

Pipelines connect receivers to exporters. The traces pipeline sends incoming trace data to Jaeger. The metrics pipeline sends incoming metric data to the Prometheus exporter.

Both pipelines also include the debug exporter so you can see telemetry flowing through the Collector's logs.

  • Save otel-collector-config.yaml.
  • Confirm both config files exist by running:
ls

You should see docker-compose.yml and otel-collector-config.yaml in the output.

✔️ Awesome, I've got everything!

Great. Make sure you have saved the file before moving on.

ⓧ I'd like to double check the full code

Scroll up and confirm your otel-collector-config.yaml has these sections in order:

  • receivers: with an otlp receiver exposing gRPC on 4317 and HTTP on 4318.
  • processors: with batch:.
  • exporters: with three entries: otlp/jaeger, prometheus (namespace otel), and debug.
  • service: with two pipelines: traces and metrics.

Next, Prometheus needs to know where to scrape metrics from.

  • Create a new file called prometheus.yml in your otel-pipeline folder and paste the following code:
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  # Scrape the OTel Collector's Prometheus exporter
  - job_name: 'otel-collector'
    static_configs:
      - targets: ['otel-collector:8889']

What does this config do?

Prometheus pulls metrics by scraping HTTP endpoints on a schedule. This config tells it to scrape the OTel Collector's exporter at otel-collector:8889 every 15 seconds.

  • Save prometheus.yml.

✔️ Awesome, I've got everything!

Great. Make sure you have saved the file before moving on.

ⓧ I'd like to double check the full code

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'otel-collector'
    static_configs:
      - targets: ['otel-collector:8889']

The last config file provisions Grafana's datasources as code. Instead of manually adding Prometheus and Jaeger through the Grafana UI, you declare them in a YAML file that Grafana reads on startup.

  • Create the Grafana provisioning directories by running:
mkdir -p grafana/provisioning/datasources grafana/dashboards
  • Create a new file called datasources.yaml inside grafana/provisioning/datasources/ and paste the following code:
apiVersion: 1

datasources:
  # Prometheus for metrics queries
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: true
    uid: prometheus-ds
    jsonData:
      httpMethod: POST

  # Jaeger for distributed traces
  - name: Jaeger
    type: jaeger
    access: proxy
    url: http://jaeger:16686
    editable: true
    uid: jaeger-ds

Why provision datasources as code?

Clicking through the Grafana UI to add datasources works, but those settings live inside the container. If you recreate the container, the datasources vanish.

Provisioning from a YAML file means Grafana loads the datasources automatically on every startup. You can version-control the file and share it with your team.

  • Save datasources.yaml.
  • Verify your full project structure by running:
find . -type f | sort

You should see these four files listed:

./docker-compose.yml
./grafana/provisioning/datasources/datasources.yaml
./otel-collector-config.yaml
./prometheus.yml

Missing a file?

  • Make sure you ran mkdir -p grafana/provisioning/datasources grafana/dashboards before creating the datasources YAML.
  • Check that you are in the otel-pipeline directory. Run pwd to confirm.
  • help me find a missing config file in my otel-pipeline project

✔️ Awesome, I've got everything!

Great. Make sure all four files are saved before moving on.

ⓧ I'd like to double check the full code

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: true
    uid: prometheus-ds
    jsonData:
      httpMethod: POST

  - name: Jaeger
    type: jaeger
    access: proxy
    url: http://jaeger:16686
    editable: true
    uid: jaeger-ds
Start the stack and verify all four services

All the config files are in place. Time to bring the stack up and make sure every service starts successfully.

  • Start all four containers in the background by running:
docker compose up -d

Docker pulls the images (this may take a minute or two on the first run) and starts each container.

  • Verify all four containers are running by running:
docker compose ps

You should see four containers listed, all with a status of Up.

A container is not running?

  • Check the logs for the failing container. For example, run docker compose logs otel-collector to see what went wrong.
  • If you see a bind mount error mentioning a source path that does not exist, make sure all four config files from the previous substep are saved in the correct locations.
  • If a port is already in use, stop whatever process is using it (for example, another Prometheus instance on port 9090) and run docker compose up -d again.
  • help me debug a failing Docker Compose container in my observability stack

Now verify that each service's UI is accessible. Before you open each URL, take a moment to predict what you expect to see.

  • Open http://localhost:16686 in your browser to confirm the Jaeger UI loads.
  • Open http://localhost:9090 to confirm the Prometheus UI loads.
  • Open http://localhost:3000 to open Grafana.
  • Log into Grafana with username admin and password admin.

Grafana asks you to change the password?

Grafana prompts you to set a new password on first login. For this local project, click Skip to keep the default.

  • In Grafana, click the hamburger menu (three horizontal lines) in the top left.
  • Click Connections, then Data sources.

You should see both Prometheus and Jaeger listed as datasources. These were provisioned automatically from your datasources.yaml file.

Your observability stack is live. You have four services running, datasources provisioned as code, and three UIs ready to receive data. Next up, you'll build a Flask API and watch your first traces appear in Jaeger.

Instrument a Flask API and See Your First Traces

Your observability stack is up and running. Prometheus, Jaeger, Grafana, and the OTel Collector are all waiting for data.

The problem is they have nothing to observe. Without an instrumented application sending telemetry, those four services are just empty dashboards. In this step, you will build a Flask API and wire it up with OpenTelemetry auto-instrumentation so every request automatically produces a trace.

In this step, get ready to:
  • Create a Flask API with three endpoints, including one that randomly introduces latency and errors.
  • Install OpenTelemetry auto-instrumentation to generate traces with zero code changes to your app.
  • Find your first traces in the Jaeger UI.
Create the Flask app with a deliberately flaky endpoint

You need a service that behaves a little unpredictably. A perfectly healthy API would produce boring, uniform telemetry. A flaky one gives you interesting spikes and errors to investigate later in Grafana and Jaeger.

  • Create the app/ directory inside otel-pipeline/ by running this command:
mkdir app
  • Create the requirements.txt file inside app/ and paste the following dependencies:
flask==3.1.3
opentelemetry-distro==0.65b0
opentelemetry-exporter-otlp==1.44.0

What are these packages?

  • flask==3.1.3 is the web framework your API runs on.
  • opentelemetry-distro==0.65b0 bundles the OpenTelemetry SDK and the auto-instrumentation tooling.
  • opentelemetry-exporter-otlp==1.44.0 lets the SDK send telemetry data over the OTLP protocol to the Collector.
  • Confirm the file was created by running:
ls app/

You should see requirements.txt listed in the output.

  • Move into the app/ directory, create a Python virtual environment, and install the dependencies by running these commands:
cd app
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Why a virtual environment?

A virtual environment keeps this project's Python packages separate from your system Python. That way, the specific versions of Flask and OpenTelemetry you install here do not interfere with other projects on your machine.

You will see pip downloading and installing several packages. Once it finishes, your prompt may show (venv) at the start, confirming the virtual environment is active.

Seeing a pip error?

  • Make sure you activated the virtual environment with source venv/bin/activate before running pip install.
  • Confirm your Python version is 3.10 or higher by running python3 --version.
  • help me debug a pip install error for opentelemetry packages
  • Install the Flask-specific instrumentation library by running:
opentelemetry-bootstrap -a install

What does this command do?

The opentelemetry-bootstrap command scans your installed packages, detects Flask, and automatically installs the matching instrumentation library. You do not need to figure out which instrumentation package to install yourself.

Now it is time to write the Flask app. The API will have three endpoints, and one of them will be deliberately unreliable.

  • Create a file called main.py inside the app/ directory and paste the following code:
import time
import random
from flask import Flask, jsonify

app = Flask(__name__)

# Sample data the API serves
ITEMS = [
    {"id": 1, "name": "Widget", "price": 9.99},
    {"id": 2, "name": "Gadget", "price": 24.99},
    {"id": 3, "name": "Doohickey", "price": 4.99},
]


@app.route("/health")
def health():
    return jsonify({"status": "ok"})


@app.route("/items")
def get_items():
    return jsonify(ITEMS)

What does this code do?

  • ITEMS is a hardcoded list of products. In a real app this would come from a database.
  • /health returns a simple status check. Load balancers and monitoring tools use endpoints like this to confirm the service is alive.
  • /items returns the full list of items as JSON.
  • Add the flaky endpoint and the main entry point below the existing code in main.py:
@app.route("/items/<int:item_id>")
def get_item(item_id):
    # 30% chance of a random 1-3 second delay
    if random.random() < 0.3:
        delay = random.uniform(1, 3)
        time.sleep(delay)

    # 15% chance of returning a 500 error
    if random.random() < 0.15:
        return jsonify({"error": "Internal server error"}), 500

    item = next((i for i in ITEMS if i["id"] == item_id), None)
    if item is None:
        return jsonify({"error": "Item not found"}), 404

    return jsonify(item)


if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5001)

Why make the endpoint flaky on purpose?

The /items/<id> endpoint randomly sleeps for 1-3 seconds (30% of requests) and randomly returns a 500 error (15% of requests). This simulates a misbehaving service so you have interesting latency spikes and errors to hunt down in your observability tools.

Without these quirks, every trace and metric would look identical. The flakiness is what makes the observability pipeline worth building.

  • Save main.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved main.py before moving on.

ⓧ I'd like to double check the full code

Your main.py should contain two sections. Scroll up and confirm each one is present, in order:

  • First block: imports (time, random, Flask, jsonify), the app instance, the ITEMS list, the /health endpoint, and the /items endpoint.
  • Second block: the /items/<int:item_id> endpoint with its random delay and error logic, followed by the if __name__ == "__main__" block running on port 5001.
Run the Flask app with OTel auto-instrumentation

Instead of running the app with a plain python main.py, you will use the opentelemetry-instrument wrapper. This wrapper injects tracing into Flask automatically. Your application code stays untouched.

  • Start the Flask app with auto-instrumentation by running this command:
OTEL_SERVICE_NAME=flask-api \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 \
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
OTEL_TRACES_EXPORTER=otlp \
OTEL_METRICS_EXPORTER=otlp \
opentelemetry-instrument python main.py

What do these environment variables do?

  • OTEL_SERVICE_NAME=flask-api gives your service a name that shows up in Jaeger and Prometheus.
  • OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 points telemetry at the OTel Collector's HTTP receiver.
  • OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf tells the exporter to use HTTP with protobuf encoding.
  • OTEL_TRACES_EXPORTER=otlp and OTEL_METRICS_EXPORTER=otlp send both traces and metrics over OTLP to the Collector.

You should see Flask's standard startup output showing it is serving on http://0.0.0.0:5001. The server will keep running in this terminal.

Seeing a connection error or module not found?

  • Make sure your virtual environment is still active. You should see (venv) at the start of your terminal prompt. If not, run source venv/bin/activate again.
  • Confirm you ran opentelemetry-bootstrap -a install earlier. Without it, the Flask instrumentation library is missing.
  • Check that the four Docker containers are still running by opening a new terminal and running docker compose ps from the otel-pipeline/ directory.
  • help me troubleshoot my opentelemetry-instrument Flask startup
  • Open a new terminal tab or window (the Flask server is using your current one).
  • Send a request to the /items endpoint by running:
curl http://localhost:5001/items

You should see a JSON array of three items returned.

  • Hit the flaky endpoint a few times by running:
curl http://localhost:5001/items/1
curl http://localhost:5001/items/1
curl http://localhost:5001/items/1

Some responses will return instantly. Others may take a few seconds (the random delay). You might even see a 500 error. That is the flaky behaviour working as designed.

Before you check Jaeger, take a moment to predict. Do you think the traces will show the delay in the slow requests, or will every trace look the same?

  • Open the Jaeger UI at http://localhost:16686 in your browser.
  • In the Service dropdown, select flask-api.
  • Click Find Traces.

You should see traces listed for each request you made. Click on any trace to expand it and see spans with timing information. The slow requests show noticeably longer durations in the span waterfall.

What is auto-instrumentation doing here?

You did not write a single line of tracing code in main.py. The opentelemetry-instrument wrapper injected tracing hooks into Flask at startup. Every incoming HTTP request now automatically creates a span with the method, URL, status code, and duration.

The span data flows from your app to the OTel Collector (via OTLP on port 4318), then the Collector forwards it to Jaeger.

Don't see flask-api in the Jaeger dropdown?

  • Make sure you hit at least one endpoint with curl before checking Jaeger. Traces only appear after a request has been made.
  • Click the refresh icon next to the Service dropdown. Jaeger caches the service list and may not show new services immediately.
  • Check the terminal where the Flask app is running. If it shows connection errors mentioning port 4318, the OTel Collector might not be running. Open a new terminal and run docker compose ps from otel-pipeline/ to confirm.
  • help me find my flask-api traces in Jaeger

You have a running API that produces traces automatically. Next up, you will add custom metrics so you can answer questions like "how many requests per minute are failing?" and "what is my p95 latency?"

Add Custom Metrics and Query Them in Prometheus

In the last step, you got traces flowing from your Flask API through the OTel Collector into Jaeger. You can click into any individual request and see exactly how long each span took.

But try answering this: how many requests per minute is your /items/<id> endpoint failing? What is the 95th percentile latency across all requests? Traces capture individual events. To answer aggregate questions about your service, you need a different signal.

In this step, get ready to:
  • Add custom OpenTelemetry metrics (counters and histograms) to your Flask app.
  • Generate traffic and query the metrics live in Prometheus.
Try answering an aggregate question with traces

Before writing any new code, test whether the tools you already have can answer a simple operational question.

  • Switch back to Jaeger at http://localhost:16686.
  • Select flask-api from the Service dropdown.
  • Click Find Traces.

Can Jaeger show you how many requests per minute are failing?

It cannot. You can see individual traces tagged with errors, but there is no count, no rate, no percentage. Each trace is a snapshot of one request.

Traces vs metrics

Traces tell you what happened inside a single request. Metrics tell you how your service is behaving overall.

"How many requests are failing?" is an aggregate question. The answer requires counting across many requests over time. That is what counters and histograms are for.

Add custom OTel metrics to the Flask app

The OpenTelemetry metrics API provides instrument types that record aggregate data. Counters count things that only go up, like total requests or total errors. Histograms record distributions, like how long each request takes.

  • In the terminal where your Flask app is running, press Ctrl+C to stop it.
  • In app/main.py, find the line from flask import Flask, jsonify and add this import directly below it:
from opentelemetry import metrics
  • Next, find the line app = Flask(__name__) and add the following meter and instrument definitions directly below it:
meter = metrics.get_meter("flask-api", "1.0.0")

request_counter = meter.create_counter(
    name="http.requests",
    description="Total HTTP requests",
    unit="1",
)

error_counter = meter.create_counter(
    name="http.errors",
    description="Total HTTP 5xx errors",
    unit="1",
)

duration_histogram = meter.create_histogram(
    name="http.request.duration",
    description="HTTP request duration in seconds",
    unit="s",
)

What do these instruments do?

  • get_meter("flask-api", "1.0.0") creates a named meter that groups your instruments together under one logical unit.
  • request_counter tracks the total number of HTTP requests, labeled by method, endpoint, and status code.
  • error_counter tracks only 5xx server errors so you can query error rate independently.
  • duration_histogram records how long each request takes in seconds, so you can later compute percentiles like p95.
  • Still in app/main.py, find the get_items function:
@app.route("/items")
def get_items():
    return jsonify(ITEMS)
  • Replace the entire function with this version that records metrics on every request:
@app.route("/items")
def get_items():
    # Record the start time for duration measurement
    start = time.time()
    response = jsonify(ITEMS)
    duration = time.time() - start
    # Increment the request counter with labels
    request_counter.add(1, {"method": "GET", "endpoint": "/items", "status_code": "200"})
    # Record the request duration in the histogram
    duration_histogram.record(duration, {"method": "GET", "endpoint": "/items"})
    return response

What changed here?

  • start = time.time() captures the moment the request begins.
  • duration = time.time() - start measures how long the handler took in seconds.
  • request_counter.add(1, {...}) increments the request count with labels so you can filter and group in Prometheus later.
  • duration_histogram.record(duration, {...}) drops the duration into histogram buckets for percentile calculations.
  • Save app/main.py.
  • Restart the Flask app with auto-instrumentation by running this command:
OTEL_SERVICE_NAME=flask-api \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 \
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
OTEL_TRACES_EXPORTER=otlp \
OTEL_METRICS_EXPORTER=otlp \
opentelemetry-instrument python app/main.py
  • Open a new terminal tab and hit the /items endpoint a few times by running:
curl http://localhost:5001/items
  • Run the curl command three or four times.
  • Switch back to Prometheus at http://localhost:9090.
  • Type otel_http_requests_total in the query field and click Execute.

You should see results showing your request counter with labels for method, endpoint, and status_code. Your custom metrics are flowing through the OTel Collector into Prometheus.

Why the otel_ prefix?

The OTel Collector's Prometheus exporter is configured with namespace: otel in otel-collector-config.yaml. This prefix prevents naming collisions with metrics from other sources.

No results in Prometheus?

  • Make sure you restarted the Flask app after saving your changes. The old process does not pick up code changes automatically.
  • Verify the app started without errors in the terminal. Look for import errors or syntax issues in the output.
  • Prometheus scrapes every 15 seconds. Wait at least 20 seconds after hitting the endpoint before querying.

Help me troubleshoot why my custom OTel metrics are not showing up in Prometheus.

  • Press Ctrl+C in the terminal where your Flask app is running to stop it.
  • Back in app/main.py, find the get_item function:
@app.route("/items/<int:item_id>")
def get_item(item_id):
    if random.random() < 0.3:
        delay = random.uniform(1, 3)
        time.sleep(delay)

    if random.random() < 0.15:
        return jsonify({"error": "Internal server error"}), 500

    item = next((i for i in ITEMS if i["id"] == item_id), None)
    if item is None:
        return jsonify({"error": "Item not found"}), 404

    return jsonify(item)
  • Replace the entire function with this version that records metrics at every exit point:
@app.route("/items/<int:item_id>")
def get_item(item_id):
    # Start timing and set common labels
    start = time.time()
    method = "GET"
    endpoint = "/items/<id>"

    # 30% chance of random delay (simulates a slow dependency)
    if random.random() < 0.3:
        delay = random.uniform(1, 3)
        time.sleep(delay)

    # 15% chance of a 500 error (simulates a flaky service)
    if random.random() < 0.15:
        duration = time.time() - start
        request_counter.add(1, {"method": method, "endpoint": endpoint, "status_code": "500"})
        error_counter.add(1, {"method": method, "endpoint": endpoint})
        duration_histogram.record(duration, {"method": method, "endpoint": endpoint})
        return jsonify({"error": "Internal server error"}), 500

    item = next((i for i in ITEMS if i["id"] == item_id), None)
    if item is None:
        duration = time.time() - start
        request_counter.add(1, {"method": method, "endpoint": endpoint, "status_code": "404"})
        duration_histogram.record(duration, {"method": method, "endpoint": endpoint})
        return jsonify({"error": "Item not found"}), 404

    duration = time.time() - start
    request_counter.add(1, {"method": method, "endpoint": endpoint, "status_code": "200"})
    duration_histogram.record(duration, {"method": method, "endpoint": endpoint})
    return jsonify(item)

Why record metrics at every exit point?

The get_item handler has three possible outcomes: a 500 error, a 404 not found, or a 200 success. Each path records the duration and increments the request counter with the appropriate status code.

The error_counter only fires on the 500 path. This separation lets you query error rate independently from total request rate later in Prometheus.

  • Save app/main.py.
  • Restart the Flask app by running the same command:
OTEL_SERVICE_NAME=flask-api \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 \
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
OTEL_TRACES_EXPORTER=otlp \
OTEL_METRICS_EXPORTER=otlp \
opentelemetry-instrument python app/main.py

Seeing an import or name error?

  • Check that from opentelemetry import metrics is on its own line after from flask import Flask, jsonify.
  • Verify the meter and instrument definitions appear between app = Flask(__name__) and the ITEMS list.
  • Make sure every reference to request_counter, error_counter, and duration_histogram matches the variable names exactly.

Help me fix an error in my Flask app after adding OTel metrics.

✔️ Awesome, I've got everything!

Great. Make sure you have saved app/main.py before continuing.

ⓧ I'd like to double check the full code

Scroll up and confirm each of these sections is present in your app/main.py, in order:

  • The import block at the top, including from opentelemetry import metrics.
  • The meter and three instrument definitions (request_counter, error_counter, duration_histogram) between app = Flask(__name__) and the ITEMS list.
  • The updated get_items function with timing and metric recording.
  • The updated get_item function with metrics recorded at every exit point (500, 404, 200).
  • The unchanged health function and if __name__ == "__main__" block at the bottom.
Generate traffic and query metrics in Prometheus

Your three custom metrics are now being recorded on every request. Time to generate enough traffic to see meaningful data in Prometheus.

  • Open a new terminal tab and generate 100 requests to the flaky endpoint by running this command:
for i in $(seq 1 100); do curl -s http://localhost:5001/items/1; done

What happens during these 100 requests?

The /items/1 endpoint has a 30% chance of adding a 1-3 second delay and a 15% chance of returning a 500 error. Out of 100 requests, you should see roughly 15 errors and 30 delayed responses.

  • Wait for the loop to finish. Some requests have random delays, so this may take a minute or two.

Before you query Prometheus, what do you think you will see when you search for your custom duration metric? Will it be a single number, or something more complex?

  • Switch back to Prometheus at http://localhost:9090.
  • Type otel_http_request_duration_seconds_bucket in the query field and click Execute.

You should see multiple time series. Each row represents a histogram bucket, showing how many requests fell into each duration range.

  • Clear the query field and type rate(otel_http_requests_total[1m]), then click Execute.

You should see the per-second request rate over the last minute. This is the aggregate signal that Jaeger could not give you.

What is PromQL doing here?

rate(otel_http_requests_total[1m]) calculates the per-second rate of counter increases over a one-minute window. It turns a raw "total requests" counter into a meaningful "requests per second" rate.

histogram_quantile works similarly for histograms. You will use it in the next step to calculate p95 latency on a Grafana dashboard.

Seeing empty results or 'no data'?

  • Make sure the curl loop finished before querying. The metrics need time to be scraped by Prometheus.
  • Check that the Flask app is still running in its terminal. If it crashed, restart it and re-run the curl loop.
  • Try a broader query first: type otel_ in the query field and check the autocomplete suggestions to see which metrics exist.

Help me figure out why my Prometheus queries return empty results after generating traffic.

You now have both traces and metrics flowing through your observability pipeline. Jaeger shows you individual request details. Prometheus shows you aggregate trends. Next up, you will build a Grafana dashboard that visualizes these metrics and lets you jump from a latency spike straight to the trace that caused it.

Build and Provision a Grafana Dashboard

Your custom metrics are flowing into Prometheus, and you can query them with raw PromQL. But running a manual query every time you want to check your API's health is not sustainable.

In this step, you will build a visual dashboard in Grafana that shows request rate, error rate, and latency at a glance. Then you will export it as a JSON file and provision it as code, so the dashboard loads automatically and survives container restarts.

In this step, get ready to:
  • Create a Grafana dashboard with three panels for request rate, error rate, and p95 latency.
  • Export the dashboard as JSON and provision it as code so it survives container restarts.
Create the dashboard in the Grafana UI

Before building panels, you need recent traffic data for the charts to display. The traffic you generated in the previous step may have stopped by now.

  • Generate fresh traffic by running this command in a terminal:
for i in $(seq 1 100); do curl -s http://localhost:5001/items/1 > /dev/null; done

What does this command do?

This loop sends 100 requests to the /items/1 endpoint. Because that endpoint has a 30% chance of adding latency and a 15% chance of returning a 500 error, the traffic produces a realistic mix of fast, slow, and failed requests for your dashboard to visualize.

Now create the dashboard and add the first panel.

  • Return to Grafana at http://localhost:3000 in your browser.
  • Click Dashboards in the left menu.
  • Click New and select New Dashboard.
  • Click the Add new element icon (blue plus sign) and drag a panel onto the dashboard.
  • On the new panel, click Configure visualization. The panel editor opens with your default datasource pre-selected.
  • In the Queries tab, confirm the Data source is set to Prometheus. If not, select it from the drop-down.
  • The query editor defaults to Builder mode. Click the Code tab to switch to the text editor so you can paste PromQL directly.
  • Enter this PromQL query:
rate(otel_http_requests_total[1m])
  • Click Refresh to query the data source.
  • Select the Time series visualization if it is not already selected.
  • Enter Request Rate as the panel title.
  • Click Save in the panel header.

You should see a time series line showing the request rate from your traffic burst.

Seeing a flat line or no data?

  • Make sure you ran the traffic generation loop before creating the panel. Without recent requests, there is no data to display.
  • Confirm the datasource is set to Prometheus, not the default Grafana datasource.
  • Check that you switched to Code mode before pasting the query. The Builder mode may not parse raw PromQL correctly.

Help me debug why my Grafana panel shows no data

Now add the second panel for error rate.

  • Click Back to dashboard in the panel header to return to the dashboard view.
  • Click the Add new element icon and drag another panel onto the dashboard.
  • Click Configure visualization on the new panel.
  • Confirm the datasource is Prometheus and switch to Code mode.
  • Enter this PromQL query:
rate(otel_http_errors_total[1m])
  • Click Refresh, select Time series as the visualization, and set the panel title to Error Rate.
  • Click Save.

You should see a line showing the rate of 5xx errors. It will be lower than the request rate since only 15% of requests produce errors.

Add the third and final panel for p95 latency.

  • Click Back to dashboard, then add a third panel using the Add new element icon.
  • Click Configure visualization, confirm Prometheus is selected, and switch to Code mode.
  • Enter this PromQL query:
histogram_quantile(0.95, rate(otel_http_request_duration_seconds_bucket[1m]))

What does this query do?

This calculates the 95th percentile latency. It reads from the histogram buckets your duration_histogram instrument records, applies a per-second rate, then uses histogram_quantile to find the value below which 95% of requests complete.

In practice, this tells you "95% of requests finish within X seconds." Spikes here mean your deliberately flaky endpoint is introducing delays.

  • Click Refresh, select Time series, and set the panel title to p95 Latency.
  • Click Save.

Now save the entire dashboard with a meaningful name.

  • Click Back to dashboard to see all three panels together.
  • Click Save (the disk icon in the top toolbar).
  • Enter Flask API Observability as the dashboard title.
  • Click Save, then click Exit edit.

Your dashboard now shows three panels side by side with live data from your Flask API.

Export the dashboard and provision it as code

The dashboard you just built lives in Grafana's internal database. If you rebuild the Grafana container, that database is gone and so is your dashboard. Provisioning from a file makes the dashboard permanent and version-controllable.

  • From your saved dashboard, click the Export icon in the toolbar and select Export as code.
  • Click Advanced options to expand the section.
  • Under Model, select Classic.
  • Click Copy to clipboard.

Why the Classic model?

Grafana offers Classic and V2 Resource export formats. The Classic model is the widely supported JSON format that works with file-based provisioning. It contains everything needed to recreate the dashboard: panels, queries, layout, and settings.

Now save the exported JSON to a file that Grafana can load on startup.

  • Create the grafana/dashboards/ directory and the JSON file by running these commands from your otel-pipeline/ folder:
mkdir -p grafana/dashboards
touch grafana/dashboards/flask-api.json
  • Confirm the file was created by running:
ls grafana/dashboards/

You should see flask-api.json listed in the output.

  • Open grafana/dashboards/flask-api.json in your text editor and paste the JSON you copied from Grafana.
  • Save the file.

Next, create the provisioning config that tells Grafana where to find dashboard JSON files.

  • Create the provisioning directory by running:
mkdir -p grafana/provisioning/dashboards
  • Create grafana/provisioning/dashboards/dashboards.yaml with the following content:
apiVersion: 1

providers:
  - name: Default
    folder: ''
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    options:
      path: /etc/grafana/dashboards

What does this config do?

This YAML tells Grafana to look for dashboard JSON files in the /etc/grafana/dashboards directory inside the container. Your docker-compose.yml already mounts ./grafana/dashboards to that path, so the flask-api.json file you just created will be picked up automatically.

The updateIntervalSeconds: 30 setting means Grafana checks for changes to the JSON files every 30 seconds. If you edit the file, the dashboard updates without a full restart.

  • Save dashboards.yaml.

Before you restart, think about what will happen. Do you think the dashboard will still be there after Grafana restarts, or will it disappear?

  • Restart the Grafana container to pick up the new provisioning config by running:
docker compose restart grafana
  • Wait a few seconds, then open http://localhost:3000 in your browser.
  • Click Dashboards in the left menu.

The "Flask API Observability" dashboard appears in the list. Open it and confirm all three panels still show data. The dashboard survived the restart because Grafana loaded it from your provisioned JSON file, not from its internal database.

Dashboard missing after restart?

  • Verify that grafana/provisioning/dashboards/dashboards.yaml exists and the path value is /etc/grafana/dashboards (the container path, not your local path).
  • Confirm grafana/dashboards/flask-api.json is not empty. Open it and check that it contains the JSON you pasted.
  • Check the Grafana container logs for provisioning errors by running docker compose logs grafana.

Help me figure out why my provisioned Grafana dashboard is not loading after restart

✔️ Awesome, I've got everything!

Great. Double check you have saved grafana/provisioning/dashboards/dashboards.yaml and grafana/dashboards/flask-api.json.

ⓧ I'd like to double check the full code

Your grafana/provisioning/dashboards/dashboards.yaml should match this exactly:

apiVersion: 1

providers:
  - name: Default
    folder: ''
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    options:
      path: /etc/grafana/dashboards

The flask-api.json file contains the JSON you exported from the Grafana UI. Its content is unique to your export. Confirm it is not empty and starts with a { character.

All other files (docker-compose.yml, app/main.py, otel-collector-config.yaml, prometheus.yml, grafana/provisioning/datasources/datasources.yaml) are unchanged from the previous step.

Your dashboard is now permanent. It loads from a JSON file, survives restarts, and can be committed to version control alongside your infrastructure code. Next up, you will connect those metric panels to individual traces so you can drill down from a latency spike to the exact request that caused it.

Connect Metrics to Traces

Your Grafana dashboard is provisioned as code and showing live request rate, error rate, and latency data. You can also dig into individual request traces in Jaeger. But right now, those two views live in completely separate worlds.

You can see a latency spike on the dashboard, but there is no way to click that spike and jump straight to the Jaeger trace that caused it. In this step, you will wire the two datasources together so Grafana can link a metric data point to the exact trace behind it.

In this step, get ready to:
  • Configure Grafana's Prometheus datasource to link exemplar trace IDs to Jaeger.
  • Configure Grafana's Jaeger datasource to link traces back to Prometheus metrics.
  • Trace a slow request from a dashboard spike all the way to the trace waterfall.
Configure trace-to-metrics correlation in Grafana

Grafana supports cross-linking between datasources through its provisioning config. You will add two connections: one telling Prometheus where to send trace IDs, and one telling Jaeger where to look up related metrics.

  • Open grafana/provisioning/datasources/datasources.yaml in your editor.
  • In the Prometheus datasource section, find the line that says httpMethod: POST under jsonData.
  • Add the following lines directly below httpMethod: POST, at the same indentation level:
      exemplarTraceIdDestinations:
        - datasourceUid: jaeger-ds
          name: traceID

What does this config do?

When Prometheus receives a metric data point that has an exemplar (a trace ID attached to it), this config tells Grafana which datasource can resolve that trace ID. Here, you point it to jaeger-ds, the UID of your Jaeger datasource.

The name: traceID field tells Grafana which label on the exemplar contains the trace identifier.

Keep the file open. The Jaeger datasource needs matching config before the link works in both directions.

  • Scroll down to the Jaeger datasource section in the same file.
  • Find the line that says uid: jaeger-ds.
  • Add the following block directly below uid: jaeger-ds:
    jsonData:
      tracesToMetrics:
        datasourceUid: prometheus-ds
        tags:
          - key: 'service.name'
            value: 'service'
      nodeGraph:
        enabled: true
      traceIdTimeParams:
        enabled: true
      spanBar:
        type: 'Duration'

What does each setting do?

  • tracesToMetrics tells Grafana that when you are viewing a Jaeger trace, it can look up related metrics in the Prometheus datasource. The service.name tag maps the trace's service name to a Prometheus label so the query filters correctly.
  • nodeGraph renders a visual graph of service dependencies when viewing a trace.
  • spanBar set to Duration shows each span's execution time as a bar in the trace waterfall.
  • Save datasources.yaml.

✔️ Awesome, I've got everything!

Make sure you have saved datasources.yaml before moving on.

ⓧ I'd like to double check the full code

Your datasources.yaml has grown past 30 lines, so scroll through it and confirm the following sections are present in order:

  • The Prometheus datasource block with uid: prometheus-ds, httpMethod: POST, and the new exemplarTraceIdDestinations block pointing to jaeger-ds.
  • The Jaeger datasource block with uid: jaeger-ds and the new jsonData block containing tracesToMetrics, nodeGraph, traceIdTimeParams, and spanBar.

Grafana reads provisioning files at startup. You need to restart Grafana for the new datasource config to take effect.

  • Restart the Grafana container by running this command:
docker compose restart grafana
  • Open http://localhost:3000 in your browser.
  • Log in with admin / admin if prompted.
  • Click the hamburger menu (three horizontal lines) in the top left.
  • Click Connections, then Data sources.
  • Click the Jaeger datasource.
  • Scroll down. You should see a Trace to metrics section with Prometheus selected as the data source. This confirms your provisioning config was picked up.

Don't see the Trace to metrics section?

  • Check your YAML indentation carefully. jsonData must be at the same level as uid (4 spaces in).
  • Make sure you saved datasources.yaml before restarting Grafana.
  • Verify the UID values match exactly: prometheus-ds in the Jaeger section and jaeger-ds in the Prometheus section.

Why isn't my Grafana datasource provisioning config being applied after a restart?

Trace a slow request end-to-end

With the datasources linked, you can now trace a slow request from start to finish. First, you need fresh traffic so there are recent traces to find.

  • Generate traffic against the flaky endpoint by running this command:
for i in $(seq 1 50); do curl -s http://localhost:5001/items/1 > /dev/null; done

Why /items/1?

The /items/<id> endpoint is the one with the deliberate flakiness. It has a 30% chance of adding 1-3 seconds of delay and a 15% chance of returning a 500 error. Hitting it 50 times guarantees you will have a mix of fast, slow, and failed requests to investigate.

Now you will use Grafana Explore to search for traces from the Jaeger datasource and find the slowest ones.

  • Return to Grafana at http://localhost:3000.
  • Click the hamburger menu in the top left.
  • Click Explore.
  • In the datasource dropdown at the top left of the Explore page, select Jaeger.
  • In the Service Name dropdown, select flask-api.
  • Click Run query in the top right.

You should see a list of recent traces from the flask-api service. Each row shows a trace ID, its duration, and the number of spans.

Before you open a trace, think about what you expect to see inside. Will a slow trace show one long span, or multiple spans? What part of the code causes the delay?

  • Sort the traces by duration by clicking the Duration column header.
  • Click on the slowest trace (the one with the longest duration, likely 1-3 seconds).

The trace waterfall opens. You should see spans representing the HTTP request flowing through Flask. The longest span corresponds to the time.sleep(delay) call in your get_item function. The span bar shows exactly how much time each piece of the request consumed.

What is a trace waterfall?

A trace waterfall is a timeline visualization of all the spans in a single request. Each span is a horizontal bar whose length represents its duration. Parent spans contain child spans, so you can see exactly where time was spent.

In your slow trace, the top-level span covers the full HTTP request. Inside it, you can see the Flask handler span where the random delay occurred. This is the power of distributed tracing: instead of guessing why a request was slow, you can see it.

  • Click the hamburger menu in the top left again.
  • Click Dashboards.
  • Open the Flask API Observability dashboard.
  • Look at the p95 Latency panel. You should see spikes corresponding to the slow requests you just generated.

You can now see the full picture. The dashboard shows aggregate trends, and Grafana Explore lets you drill into the specific trace behind any spike. Your metrics and traces are connected.

Not seeing traces in Grafana Explore?

  • Make sure you selected Jaeger (not Prometheus) in the datasource dropdown at the top of the Explore page.
  • Check that your Flask app is still running. If it stopped, restart it with opentelemetry-instrument and generate traffic again.
  • Adjust the time range in the top right corner of Grafana to Last 15 minutes to make sure recent traces are in scope.

Why are my Jaeger traces not appearing in Grafana Explore?

Secret mission

Distributed Tracing Across Two Services

Your observability pipeline traces a single service. In production, requests cross multiple services, and a trace that stops at the boundary leaves you blind to downstream failures. Can you add a second Flask service, instrument it with OpenTelemetry, and prove that the trace context propagates across the HTTP call so both services appear in a single Jaeger trace?

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally with Docker, so there are no ongoing financial costs.

Resources you used:

  • Four Docker containers: otel-collector, jaeger, prometheus, grafana.
  • Two running Flask processes on ports 5001 and 5002.
  • Project directory otel-pipeline/ on your Desktop, including config files, application code, and Python virtual environments.

Keep everything running

No action needed. Choose this if you are still actively experimenting with the observability stack or generating traffic to explore your Grafana dashboard.

  • Your four Docker containers will continue running and consuming roughly 1-2 GB of RAM.
  • Your Flask apps will continue running in their terminal windows on ports 5001 and 5002.
  • All project files remain in otel-pipeline/ on your Desktop.

Pause - I'll come back to this later

Stop running processes to free up memory. Your project files stay on disk so you can pick up where you left off.

  • Press Ctrl+C in each terminal window running a Flask app to stop the processes.
  • Stop the Docker containers without removing data by running this command from inside your otel-pipeline/ directory:
docker compose stop
  • When you are ready to resume, navigate back to otel-pipeline/ and run:
docker compose start
  • Then restart your Flask apps from their virtual environments using opentelemetry-instrument as you did during the project.

Delete - I don't want to use this again

Remove all project resources and start fresh. This deletes your containers, volumes, and all project files.

  • Press Ctrl+C in each terminal window running a Flask app to stop the processes.
  • Stop and remove all Docker containers and volumes by running this command from inside your otel-pipeline/ directory:
docker compose down -v
  • Delete the entire project directory by running:
cd ~/Desktop
rm -rf otel-pipeline
  • Confirm the directory is removed by running:
ls ~/Desktop

You should no longer see otel-pipeline in the output.

That's a Wrap!

That's a Wrap!

Nice work! You just built a full observability pipeline from scratch, turning a single Flask request into traces, metrics, and dashboards that all talk to each other.

Here's what you learned:

  • Deployed a four-service observability stack (OTel Collector, Prometheus, Jaeger, Grafana) with a single docker compose up and configured the Collector as a central telemetry router that decouples your application from specific backends.
  • Instrumented a Flask API with OpenTelemetry auto-instrumentation for traces and custom metrics (counters and histograms) to answer aggregate questions like request rate, error rate, and p95 latency.
  • Provisioned Grafana datasources and dashboards as code using YAML and JSON files, then connected metrics to traces so you can drill down from a latency spike to the exact Jaeger trace that caused it.
  • Secret Mission: Added a second service and traced requests across service boundaries using OTel context propagation.

Ready to quiz yourself?