Deploy an Observability Stack with Terraform

Deploy a full LGTM stack with metrics, logs, and traces via one command.

Introduction

30 Second Summary

When a production system goes down, the first question is always "what happened?" Observability platforms answer that question by collecting metrics, logs, and traces from every service in real time.

In this project, you will deploy a complete LGTM observability stack (Loki, Grafana, Tempo, Prometheus) using Terraform and Docker Compose on your local machine.

What You'll Build

A fully functional observability platform provisioned as Infrastructure as Code, where one command brings up 9 interconnected services and one command tears them down.

By the end of this project, you'll have:

  • A Terraform configuration that provisions Prometheus, Loki, Tempo, Grafana, and supporting services with a single terraform apply command.
  • A sample Python application instrumented with OpenTelemetry, emitting traces and logs through an OpenTelemetry Collector into the correct backends.
  • PromQL expressions defining the Four Golden Signals (Latency, Traffic, Errors, Saturation) as measurable SLIs for your service.
  • Secret Mission: Build a Grafana dashboard provisioned as code that visualizes all Four Golden Signals in real-time panels.

Are there any prerequisites?

No prior experience with observability tools is required. This project includes setup instructions for all tools you need, including Docker Desktop, Terraform, and VS Code.

Before We Start

You are about to build a production-grade observability platform that collects metrics, aggregates logs, and traces requests across your system. Before you write any configuration, you need to confirm that your tools are ready and create a clean project structure.

In this step, you will verify that Docker Desktop and Terraform are working correctly on your machine. Then you will scaffold the folder structure that holds all of your configuration files.

In this step, get ready to:
  • Verify Docker Desktop is running with Linux containers and Docker Compose is available.
  • Verify Terraform is installed and accessible from PowerShell.
  • Create the full project directory structure for configs, Terraform, and the sample app.
Verify Docker Desktop and Docker Compose are available

Docker Desktop runs the containers that make up your observability stack. You need to confirm it is running and set to Linux containers mode.

  • Press the Windows key to open your search bar.
  • Type PowerShell and press Enter to open it.
  • Check that Docker is running by entering this command:
docker --version

✔️ I see a version number

Docker is running and ready to go.

ⓧ I see an error or 'command not found'

  • Make sure Docker Desktop is open and running. Look for the Docker whale icon in your system tray at the bottom-right of your screen.
  • If you do not see the icon, press the Windows key, type Docker Desktop, and press Enter to launch it.
  • Wait about 30 seconds for Docker to finish starting, then try the command again.
  • Confirm Docker is using Linux containers by running:
docker info --format '{{.OSType}}'

You should see linux in the output.

Seeing 'windows' instead of 'linux'?

Right-click the Docker whale icon in your system tray. If you see Switch to Linux containers... in the menu, click it and wait for Docker to restart.

Then run the docker info command again to confirm it now shows linux.

Help me switch Docker Desktop to Linux containers.

  • Verify Docker Compose is available by running:
docker compose version

✔️ I see a version number

Docker Compose is ready. This is included with Docker Desktop by default.

ⓧ I see an error

Docker Compose is bundled with Docker Desktop. If this command fails, your Docker Desktop installation may be incomplete.

  • Open Docker Desktop, go to Settings (gear icon), then select General.
  • Confirm Use Docker Compose V2 is checked.
  • Restart Docker Desktop and try again.

Help me fix Docker Compose not being found.

Verify Terraform is installed

Terraform manages your entire stack as Infrastructure as Code. A single terraform apply command will bring all nine services online.

  • Check that Terraform is accessible by running:
terraform --version

✔️ I see a version number

Terraform is installed and ready.

ⓧ Command not found

  • Download Terraform from terraform.io/downloads.
  • Extract the terraform.exe file to a folder on your machine (for example, C:\terraform).
  • Add that folder to your system PATH: open Settings, search for Environment Variables, edit the Path variable, and add your Terraform folder.
  • Close and reopen PowerShell, then run terraform --version again.

Help me install Terraform on Windows.

Create the project directory structure

Your observability stack uses many configuration files organized into dedicated folders. Creating this structure now keeps everything clean as you build out each service.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir observability-stack
cd observability-stack
  • Create all the subdirectories your stack needs by running:
mkdir terraform
mkdir configs/prometheus
mkdir configs/loki
mkdir configs/tempo
mkdir configs/otel
mkdir configs/alertmanager
mkdir configs/blackbox
mkdir configs/grafana/provisioning/datasources
mkdir configs/grafana/provisioning/dashboards
mkdir configs/grafana/dashboards
mkdir sample-app
mkdir docs

What does this structure do?

  • terraform/ holds the Terraform configuration that brings the entire stack up with one command.
  • configs/ contains configuration files for each observability tool (Prometheus, Loki, Tempo, Grafana, and more).
  • sample-app/ is where you will build a Python Flask app instrumented with OpenTelemetry.
  • docs/ stores your SLI definitions for the Four Golden Signals.
  • Confirm the directories exist by running:
ls

You should see configs, docs, sample-app, and terraform listed in the output.

Don't see all four folders?

Make sure you ran each mkdir command from inside the observability-stack folder. Run pwd to check your current location. It should end with observability-stack.

Help me fix my project directory structure.

  • Open the project in VS Code by running:
code .

VS Code opens with your observability-stack folder loaded in the Explorer sidebar.

'code' is not recognized?

Open VS Code manually (press the Windows key, type Visual Studio Code, press Enter). Then press Ctrl+Shift+P, type Shell Command: Install 'code' command in PATH, and select it. Restart PowerShell and try code . again.

Help me get the 'code' command working in PowerShell.

Your environment is locked and loaded. Next up, you will write the Docker Compose file that defines all nine services in your observability stack.

Write the Docker Compose Stack

Your environment is ready and your project folders are in place. Now it's time to define the actual infrastructure that will run your observability platform.

In this step, you will write a single docker-compose.yaml file that defines 9 interconnected services. When Terraform runs this file later, it brings your entire observability stack to life with one command.

In this step, get ready to:
  • Define all 9 services (Prometheus, Loki, Tempo, Grafana, Alertmanager, Node Exporter, Blackbox Exporter, OTel Collector, and the sample app).
  • Configure a shared Docker network and persistent named volumes.
  • Create the environment file for Docker Compose.
Create the service definitions

A Docker Compose file declares your entire multi-container application in one place. Each service block specifies the container image, port mappings, volume mounts, health checks, and restart policies.

  • In VS Code, create a new file called docker-compose.yaml at the root of your project folder (the same level as your terraform/ and configs/ folders).
  • Paste the following service definitions into the file:
services:
  prometheus:
    image: prom/prometheus:v3.11.3
    container_name: prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=15d'
      - '--web.enable-lifecycle'
    volumes:
      - ./configs/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./configs/prometheus/alert-rules.yml:/etc/prometheus/alert-rules.yml:ro
      - prometheus-data:/prometheus
    ports:
      - "9090:9090"
    networks:
      - observability
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost:9090/-/healthy"]
      interval: 10s
      timeout: 5s
      retries: 3

  loki:
    image: grafana/loki:3.7.2
    container_name: loki
    command: -config.file=/etc/loki/loki-config.yaml
    volumes:
      - ./configs/loki/loki-config.yaml:/etc/loki/loki-config.yaml:ro
      - loki-data:/loki
    ports:
      - "3100:3100"
    networks:
      - observability
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost:3100/ready"]
      interval: 10s
      timeout: 5s
      retries: 3

  tempo:
    image: grafana/tempo:2.8.3
    container_name: tempo
    command: -config.file=/etc/tempo/tempo-config.yaml
    volumes:
      - ./configs/tempo/tempo-config.yaml:/etc/tempo/tempo-config.yaml:ro
      - tempo-data:/var/tempo
    ports:
      - "3200:3200"
    networks:
      - observability
    restart: unless-stopped

  grafana:
    image: grafana/grafana:12.3.6
    container_name: grafana
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
      - ./configs/grafana/provisioning:/etc/grafana/provisioning:ro
      - ./configs/grafana/dashboards:/var/lib/grafana/dashboards:ro
      - grafana-data:/var/lib/grafana
    ports:
      - "3000:3000"
    networks:
      - observability
    restart: unless-stopped
    depends_on:
      - prometheus
      - loki
      - tempo
    healthcheck:
      test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost:3000/api/health"]
      interval: 10s
      timeout: 5s
      retries: 3

  alertmanager:
    image: prom/alertmanager:v0.28.1
    container_name: alertmanager
    command:
      - '--config.file=/etc/alertmanager/alertmanager.yml'
      - '--storage.path=/alertmanager'
    volumes:
      - ./configs/alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
      - alertmanager-data:/alertmanager
    ports:
      - "9093:9093"
    networks:
      - observability
    restart: unless-stopped

  node-exporter:
    image: prom/node-exporter:v1.11.1
    container_name: node-exporter
    command:
      - '--path.rootfs=/host'
    volumes:
      - /:/host:ro
    ports:
      - "9100:9100"
    networks:
      - observability
    restart: unless-stopped

  blackbox-exporter:
    image: prom/blackbox-exporter:v0.28.0
    container_name: blackbox-exporter
    command:
      - '--config.file=/etc/blackbox/blackbox.yml'
    volumes:
      - ./configs/blackbox/blackbox.yml:/etc/blackbox/blackbox.yml:ro
    ports:
      - "9115:9115"
    networks:
      - observability
    restart: unless-stopped

  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.127.0
    container_name: otel-collector
    command: ["--config", "/etc/otelcol-contrib/config.yaml"]
    volumes:
      - ./configs/otel/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
    ports:
      - "4318:4318"
      - "8888:8888"
    networks:
      - observability
    restart: unless-stopped
    depends_on:
      - loki
      - tempo

  sample-app:
    build: ./sample-app
    container_name: sample-app
    environment:
      - OTEL_SERVICE_NAME=sample-app
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
      - OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
      - OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true
    ports:
      - "8080:8080"
    networks:
      - observability
    restart: unless-stopped
    depends_on:
      - otel-collector

What does this configuration do?

Each service block defines one container in your observability stack. Here is how they break down by role:

  • Data stores: Prometheus (metrics), Loki (logs), and Tempo (traces) are the three backends that store your telemetry data.
  • Frontend and alerting: Grafana provides the unified dashboard. Alertmanager routes notifications when things go wrong.
  • Exporters: Node Exporter collects system-level metrics (CPU, memory, disk). Blackbox Exporter probes HTTP endpoints to confirm they respond.
  • Telemetry pipeline: The OpenTelemetry Collector receives logs and traces from the sample app, then routes them to Loki and Tempo respectively.
  • Sample app: A Flask application you will build later that emits traces and logs through the OTel Collector.

Every service uses restart: unless-stopped so containers automatically recover from crashes. The depends_on fields ensure services start in the correct order.

  • You won't see a change yet. The next section adds the networking that connects everything together.
Define the network and volumes

Services need a shared network to communicate by container name. Named volumes persist data across container restarts so you don't lose metrics or logs when you rebuild.

  • Add the following at the bottom of your docker-compose.yaml file, directly after the sample-app service definition:
networks:
  observability:
    driver: bridge

volumes:
  prometheus-data:
  loki-data:
  tempo-data:
  grafana-data:
  alertmanager-data:

What does this section do?

The observability network uses the bridge driver, which creates an isolated virtual network. Every service joined to this network can reach other services by their container name (for example, http://prometheus:9090).

The five named volumes give each data store a persistent location on disk. Without named volumes, all stored metrics, logs, and traces would disappear every time you stop the containers.

  • Save docker-compose.yaml.
  • Confirm the file exists by running:
ls docker-compose.yaml
  • You should see docker-compose.yaml listed in the output.

Don't see the file?

Make sure your terminal is in the project root folder (the same folder that contains terraform/ and configs/). Also confirm you saved the file as docker-compose.yaml (not docker-compose.yml.txt or similar).

Help me find my docker-compose.yaml file

Configure port mappings and the project environment

Each service exposes a specific port so you can access it from your browser or from other tools on your machine. Docker Compose also reads an .env file to set project-wide variables like the project name.

What ports does each service use?

  • Grafana: port 3000 (your main observability dashboard).
  • Prometheus: port 9090 (metrics storage and query UI).
  • Loki: port 3100 (log aggregation backend).
  • Tempo: port 3200 (distributed tracing backend).
  • Alertmanager: port 9093 (alert routing and notifications).
  • OTel Collector: port 4318 (OTLP HTTP receiver) and port 8888 (collector metrics).
  • Sample App: port 8080 (the Flask application you will instrument).
  • Node Exporter: port 9100 (system-level metrics).
  • Blackbox Exporter: port 9115 (HTTP probe results).
  • Create a new file called .env at the project root (same level as docker-compose.yaml) and paste the following:
COMPOSE_PROJECT_NAME=observability

What does this file do?

Docker Compose reads .env automatically from the project directory. The COMPOSE_PROJECT_NAME variable sets the prefix for all container names, networks, and volumes. This keeps your observability stack cleanly separated from other Docker projects on your machine.

  • Save .env.
  • Confirm both files exist by running:
ls -la .env docker-compose.yaml
  • You should see both .env and docker-compose.yaml listed in the output.

Don't see the .env file?

Files starting with a dot are hidden by default. The -la flag in the ls command shows hidden files. If the file is missing, create it again in VS Code and make sure the filename is exactly .env with no extra extension.

Help me create a hidden .env file

✔️ Awesome, I've got everything!

Great. Double check you have saved both files before moving on.

ⓧ I'd like to double check the full code

Here is the complete docker-compose.yaml for comparison:

services:
  prometheus:
    image: prom/prometheus:v3.11.3
    container_name: prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=15d'
      - '--web.enable-lifecycle'
    volumes:
      - ./configs/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./configs/prometheus/alert-rules.yml:/etc/prometheus/alert-rules.yml:ro
      - prometheus-data:/prometheus
    ports:
      - "9090:9090"
    networks:
      - observability
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost:9090/-/healthy"]
      interval: 10s
      timeout: 5s
      retries: 3

  loki:
    image: grafana/loki:3.7.2
    container_name: loki
    command: -config.file=/etc/loki/loki-config.yaml
    volumes:
      - ./configs/loki/loki-config.yaml:/etc/loki/loki-config.yaml:ro
      - loki-data:/loki
    ports:
      - "3100:3100"
    networks:
      - observability
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost:3100/ready"]
      interval: 10s
      timeout: 5s
      retries: 3

  tempo:
    image: grafana/tempo:2.8.3
    container_name: tempo
    command: -config.file=/etc/tempo/tempo-config.yaml
    volumes:
      - ./configs/tempo/tempo-config.yaml:/etc/tempo/tempo-config.yaml:ro
      - tempo-data:/var/tempo
    ports:
      - "3200:3200"
    networks:
      - observability
    restart: unless-stopped

  grafana:
    image: grafana/grafana:12.3.6
    container_name: grafana
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
      - ./configs/grafana/provisioning:/etc/grafana/provisioning:ro
      - ./configs/grafana/dashboards:/var/lib/grafana/dashboards:ro
      - grafana-data:/var/lib/grafana
    ports:
      - "3000:3000"
    networks:
      - observability
    restart: unless-stopped
    depends_on:
      - prometheus
      - loki
      - tempo
    healthcheck:
      test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost:3000/api/health"]
      interval: 10s
      timeout: 5s
      retries: 3

  alertmanager:
    image: prom/alertmanager:v0.28.1
    container_name: alertmanager
    command:
      - '--config.file=/etc/alertmanager/alertmanager.yml'
      - '--storage.path=/alertmanager'
    volumes:
      - ./configs/alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
      - alertmanager-data:/alertmanager
    ports:
      - "9093:9093"
    networks:
      - observability
    restart: unless-stopped

  node-exporter:
    image: prom/node-exporter:v1.11.1
    container_name: node-exporter
    command:
      - '--path.rootfs=/host'
    volumes:
      - /:/host:ro
    ports:
      - "9100:9100"
    networks:
      - observability
    restart: unless-stopped

  blackbox-exporter:
    image: prom/blackbox-exporter:v0.28.0
    container_name: blackbox-exporter
    command:
      - '--config.file=/etc/blackbox/blackbox.yml'
    volumes:
      - ./configs/blackbox/blackbox.yml:/etc/blackbox/blackbox.yml:ro
    ports:
      - "9115:9115"
    networks:
      - observability
    restart: unless-stopped

  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.127.0
    container_name: otel-collector
    command: ["--config", "/etc/otelcol-contrib/config.yaml"]
    volumes:
      - ./configs/otel/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
    ports:
      - "4318:4318"
      - "8888:8888"
    networks:
      - observability
    restart: unless-stopped
    depends_on:
      - loki
      - tempo

  sample-app:
    build: ./sample-app
    container_name: sample-app
    environment:
      - OTEL_SERVICE_NAME=sample-app
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
      - OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
      - OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true
    ports:
      - "8080:8080"
    networks:
      - observability
    restart: unless-stopped
    depends_on:
      - otel-collector

networks:
  observability:
    driver: bridge

volumes:
  prometheus-data:
  loki-data:
  tempo-data:
  grafana-data:
  alertmanager-data:

And here is the complete .env file:

COMPOSE_PROJECT_NAME=observability

Your Docker Compose stack is fully defined. Next up, you will write the configuration files that each service needs to start correctly.

Configure Prometheus and Exporters

Your Docker Compose stack defines all nine services, but they need configuration files to know what to do. Prometheus is the heart of metrics collection. It needs to know which services to scrape, how often to pull data, and where to send alerts.

In this step, you will write the scrape configuration that tells Prometheus where to find metrics. You will also configure Alertmanager for routing alert notifications and the Blackbox Exporter for HTTP endpoint probing.

In this step, get ready to:
  • Write Prometheus scrape configs and alert rules.
  • Configure Alertmanager with Slack notification routing.
  • Define the Blackbox Exporter HTTP probe module.
Write the Prometheus scrape configuration

Prometheus uses a YAML configuration file to determine which targets to scrape and how frequently. Each target is defined as a "job" with its own scrape interval and endpoint details.

  • In VS Code, create a new file at configs/prometheus/prometheus.yml and paste the following configuration:
global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093

rule_files:
  - /etc/prometheus/alert-rules.yml

scrape_configs:
  # Prometheus monitors itself
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Node Exporter exposes system-level metrics (CPU, memory, disk)
  - job_name: 'node-exporter'
    scrape_interval: 15s
    static_configs:
      - targets: ['node-exporter:9100']

  # Blackbox Exporter probes HTTP endpoints from the outside
  - job_name: 'blackbox-http'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - http://sample-app:8080
          - http://sample-app:8080/api/orders
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

  # OpenTelemetry Collector exposes its own health metrics
  - job_name: 'otel-collector'
    static_configs:
      - targets: ['otel-collector:8888']

  # The sample Flask app exposes application-level metrics
  - job_name: 'sample-app'
    static_configs:
      - targets: ['sample-app:8080']

What does this configuration do?

The global section sets defaults: scrape every 15 seconds and evaluate alert rules at the same interval.

The alerting section points to the Alertmanager container so Prometheus knows where to send firing alerts.

The blackbox-http job uses a technique called relabeling. Instead of scraping the target URL directly, Prometheus asks the Blackbox Exporter to probe it. The relabel_configs rewrite the request so Prometheus sends the probe to blackbox-exporter:9115 with the real URL as a parameter.

  • Save configs/prometheus/prometheus.yml.
  • Confirm the file appears in the VS Code Explorer sidebar under configs/prometheus/.

File not showing in the sidebar?

Make sure the folder path is exactly configs/prometheus/ with no typos. YAML files are indentation-sensitive, so use spaces (not tabs) for indentation.

help me check my prometheus.yml for syntax errors

Prometheus needs an alert rules file that defines when to fire alerts. The rule_files section in prometheus.yml references /etc/prometheus/alert-rules.yml, so you need to create that file next.

  • Create a new file at configs/prometheus/alert-rules.yml and paste the following:
groups:
  - name: infrastructure
    rules:
      # Fire an alert when any monitored instance is unreachable
      - alert: InstanceDown
        expr: up == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Instance {{ $labels.instance }} down"
          description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 2 minutes."

What does this alert rule do?

The InstanceDown alert fires when the built-in up metric equals 0 for any scrape target. The for: 2m clause means Prometheus waits 2 minutes before firing, avoiding flapping alerts from brief network hiccups.

The annotations section uses Go template syntax to inject the failing instance name and job into the alert message.

  • Save configs/prometheus/alert-rules.yml.
  • You should now see both prometheus.yml and alert-rules.yml inside the configs/prometheus/ folder in the Explorer sidebar.

✔️ Awesome, I've got everything!

Great. Double check you have saved both files.

ⓧ I'd like to double check the full code

Here is the complete configs/prometheus/prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093

rule_files:
  - /etc/prometheus/alert-rules.yml

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node-exporter'
    scrape_interval: 15s
    static_configs:
      - targets: ['node-exporter:9100']

  - job_name: 'blackbox-http'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - http://sample-app:8080
          - http://sample-app:8080/api/orders
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

  - job_name: 'otel-collector'
    static_configs:
      - targets: ['otel-collector:8888']

  - job_name: 'sample-app'
    static_configs:
      - targets: ['sample-app:8080']

And the complete configs/prometheus/alert-rules.yml:

groups:
  - name: infrastructure
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Instance {{ $labels.instance }} down"
          description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 2 minutes."
Write the Alertmanager configuration

When Prometheus fires an alert, it sends it to Alertmanager. Alertmanager groups related alerts together and routes them to the right notification channel.

  • Create a new file at configs/alertmanager/alertmanager.yml and paste the following:
global:
  slack_api_url: 'your-slack-webhook-url-here'
  resolve_timeout: 5m

route:
  receiver: 'slack-devops-alerts'
  group_by: ['alertname', 'severity', 'job']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

receivers:
  - name: 'slack-devops-alerts'
    slack_configs:
      - channel: '#DevOps-Alerts'
        send_resolved: true
        title: '{{ if eq .Status "firing" }}🚨{{ else }}✅{{ end }} [{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}'
        text: >-
          *Severity:* {{ .CommonLabels.severity }}
          *Instance:* {{ .CommonLabels.instance }}
          *Job:* {{ .CommonLabels.job }}
          *Description:* {{ .CommonAnnotations.description }}
          *Dashboard:* http://localhost:3000
          *Runbook:* https://github.com/your-repo/runbooks/{{ .CommonLabels.alertname }}.md

What does this configuration do?

The route section defines how alerts are grouped before being sent. Grouping by alertname, severity, and job means you get one Slack message per unique combination instead of one per firing instance.

The slack_configs section uses Go templating to build structured Slack messages. The send_resolved: true setting sends a follow-up message when an alert resolves, so your team knows the issue is fixed.

The slack_api_url is a placeholder. You do not need a real Slack webhook for this project to work. Alertmanager will still route and group alerts internally.

  • Save configs/alertmanager/alertmanager.yml.
  • Confirm the file appears under configs/alertmanager/ in the Explorer sidebar.

YAML indentation errors?

Alertmanager configuration is strict about indentation. Make sure receivers is at the root level (no leading spaces), and slack_configs is indented under the receiver name.

help me fix my alertmanager.yml indentation

✔️ Awesome, I've got everything!

Great. Double check you have saved the file.

ⓧ I'd like to double check the full code

Here is the complete configs/alertmanager/alertmanager.yml:

global:
  slack_api_url: 'your-slack-webhook-url-here'
  resolve_timeout: 5m

route:
  receiver: 'slack-devops-alerts'
  group_by: ['alertname', 'severity', 'job']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

receivers:
  - name: 'slack-devops-alerts'
    slack_configs:
      - channel: '#DevOps-Alerts'
        send_resolved: true
        title: '{{ if eq .Status "firing" }}🚨{{ else }}✅{{ end }} [{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}'
        text: >-
          *Severity:* {{ .CommonLabels.severity }}
          *Instance:* {{ .CommonLabels.instance }}
          *Job:* {{ .CommonLabels.job }}
          *Description:* {{ .CommonAnnotations.description }}
          *Dashboard:* http://localhost:3000
          *Runbook:* https://github.com/your-repo/runbooks/{{ .CommonLabels.alertname }}.md
Write the Blackbox Exporter configuration

The Blackbox Exporter does not collect metrics from inside a service. It probes endpoints from the outside, the same way a user would. This lets you detect outages that internal metrics would miss.

  • Create a new file at configs/blackbox/blackbox.yml and paste the following:
modules:
  # Defines an HTTP probe that expects a 200 OK response
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: [200]
      method: GET
      follow_redirects: true
      preferred_ip_protocol: ip4

What does this module define?

The http_2xx module sends an HTTP GET request to whatever target Prometheus tells it to probe. If the response is a 200 status code within 5 seconds, the probe succeeds.

The preferred_ip_protocol: ip4 setting avoids IPv6 issues that commonly occur in Docker environments. Docker networking sometimes blocks IPv6, which would cause probes to fail even when the target is healthy.

  • Save configs/blackbox/blackbox.yml.
  • Confirm the file appears under configs/blackbox/ in the Explorer sidebar.

Don't see the configs/blackbox/ folder?

Make sure you created the configs/blackbox/ directory in Step 1. If it does not exist, create the folder first, then add the blackbox.yml file inside it.

help me verify my project directory structure is correct

✔️ Awesome, I've got everything!

Great. Double check you have saved the file.

ⓧ I'd like to double check the full code

Here is the complete configs/blackbox/blackbox.yml:

modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: [200]
      method: GET
      follow_redirects: true
      preferred_ip_protocol: ip4

Your metrics pipeline is now configured. Prometheus knows which endpoints to scrape, Alertmanager knows where to send alerts, and the Blackbox Exporter knows how to probe HTTP endpoints. Next, you will configure the log and trace backends that complete the observability stack.

Configure Loki, Tempo, and OpenTelemetry Collector

In the previous step, you configured Prometheus to scrape metrics from your exporters and the sample app. Metrics are only one pillar of observability.

Now you'll configure the other two pillars: logs and traces. Loki aggregates logs, Tempo stores distributed traces, and the OpenTelemetry Collector acts as the central router that sends telemetry data to the right backend.

In this step, get ready to:
  • Configure Loki for log aggregation with local storage and 72-hour retention.
  • Configure Tempo as the distributed tracing backend with OTLP receivers.
  • Configure the OpenTelemetry Collector to route logs to Loki and traces to Tempo.
Configure Loki for log aggregation

Loki is a log aggregation system designed by Grafana Labs. It indexes metadata (labels) about your logs rather than the full text, making it lightweight and fast to query.

  • In VS Code, create a new file at configs/loki/loki-config.yaml.
  • Paste the following configuration into the file:
auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

limits_config:
  allow_structured_metadata: true
  volume_enabled: true
  retention_period: 72h

schema_config:
  configs:
    - from: "2024-01-01"
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

compactor:
  working_directory: /loki/compactor
  retention_enabled: true

What does this configuration do?

  • auth_enabled: false disables multi-tenant authentication since this is a single-user local stack.
  • The common block configures local filesystem storage for log chunks and rules, using in-memory ring storage since there is only one instance.
  • retention_period: 72h automatically deletes logs older than 3 days to prevent disk from filling up.
  • The schema_config uses TSDB with schema v13, which supports native OTLP log ingestion from the OpenTelemetry Collector.
  • Save configs/loki/loki-config.yaml.

You won't see a change yet. Loki will use this config when the stack launches later.

YAML indentation errors?

YAML is whitespace-sensitive. Make sure you use spaces (not tabs) for indentation, and that nested keys are indented by exactly 2 spaces from their parent.

Check the bottom-left of VS Code to confirm it says "Spaces: 2". If it shows "Tab Size", click it and switch to spaces.

Help me fix YAML indentation errors in my Loki config.

Configure Tempo for distributed tracing

Tempo stores distributed traces. A trace records the full journey of a request as it moves through your services, capturing timing and metadata at each hop.

  • Create a new file at configs/tempo/tempo-config.yaml.
  • Paste the following configuration:
auth_enabled: false

server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

ingester:
  max_block_duration: 5m

compactor:
  compaction:
    block_retention: 24h

storage:
  trace:
    backend: local
    wal:
      path: /var/tempo/wal
    local:
      path: /var/tempo/blocks

metrics_generator:
  registry:
    collection_interval: 15s
  storage:
    path: /var/tempo/generator/wal
  processor:
    local_blocks:
      filter_server_spans: false

overrides:
  defaults:
    metrics_generator:
      processors:
        - local-blocks

What does this configuration do?

  • The distributor.receivers section opens OTLP receivers on both gRPC (port 4317) and HTTP (port 4318) so traces can arrive via either protocol.
  • block_retention: 24h keeps trace data for one day before compaction removes it.
  • The metrics_generator derives RED metrics (Rate, Errors, Duration) from incoming traces. This means Prometheus can query span-based metrics without the app exporting them separately.
  • Local storage with a write-ahead log (wal) ensures trace data survives a container restart.
  • Save configs/tempo/tempo-config.yaml.

You won't see a change yet. Tempo uses this config when the stack launches.

Not sure where this file goes?

In VS Code's Explorer sidebar, look for the configs/tempo/ folder you created in Step 1. Right-click it and select **New File**, then name it tempo-config.yaml.

Help me create the Tempo config file in the correct location.

Configure the OpenTelemetry Collector

The OpenTelemetry Collector is the central telemetry pipeline. It receives data from your applications via OTLP, processes it in batches, and exports it to the correct backend. Logs go to Loki. Traces go to Tempo.

  • Create a new file at configs/otel/otel-collector-config.yaml.
  • Paste the following configuration:
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 512

exporters:
  otlphttp/loki:
    endpoint: http://loki:3100/otlp
    tls:
      insecure: true

  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/loki]

    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo]

What does this configuration do?

The Collector config has four sections that define the data flow:

  • receivers defines how data enters the Collector. Here, OTLP over HTTP on port 4318.
  • processors transforms data in-flight. The batch processor groups telemetry into batches of 512 items (or flushes every 5 seconds) to reduce network calls.
  • exporters defines where processed data goes. otlphttp/loki sends logs to Loki's native OTLP endpoint. otlp/tempo sends traces to Tempo over gRPC.
  • service.pipelines wires everything together. The logs pipeline and traces pipeline each get their own receiver, processor, and exporter chain.

Why separate pipelines for logs and traces?

Logs and traces are fundamentally different data types. Logs are text records with timestamps. Traces are structured spans with parent-child relationships.

Separate pipelines let you route each type to a purpose-built backend. Loki is optimized for log indexing and querying. Tempo is optimized for trace storage and retrieval by trace ID.

  • Save configs/otel/otel-collector-config.yaml.
  • Confirm all three new config files exist by running this in your terminal:
ls configs/loki, configs/tempo, configs/otel

You should see loki-config.yaml, tempo-config.yaml, and otel-collector-config.yaml listed in the output.

Missing one of the config files?

Make sure you created the file in the correct subfolder. The OTel config goes in configs/otel/, not in configs/ directly.

Check that the filenames match exactly. Typos like otel-config.yaml instead of otel-collector-config.yaml will cause the container to fail to start later.

Help me find where my config files ended up.

✔️ Awesome, I've got everything!

Great. Double check you've saved all three files before moving on.

ⓧ I'd like to double check the full code

Here is the complete content for each file. Compare yours against these:

auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

limits_config:
  allow_structured_metadata: true
  volume_enabled: true
  retention_period: 72h

schema_config:
  configs:
    - from: "2024-01-01"
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

compactor:
  working_directory: /loki/compactor
  retention_enabled: true
auth_enabled: false

server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

ingester:
  max_block_duration: 5m

compactor:
  compaction:
    block_retention: 24h

storage:
  trace:
    backend: local
    wal:
      path: /var/tempo/wal
    local:
      path: /var/tempo/blocks

metrics_generator:
  registry:
    collection_interval: 15s
  storage:
    path: /var/tempo/generator/wal
  processor:
    local_blocks:
      filter_server_spans: false

overrides:
  defaults:
    metrics_generator:
      processors:
        - local-blocks
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 512

exporters:
  otlphttp/loki:
    endpoint: http://loki:3100/otlp
    tls:
      insecure: true

  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/loki]

    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo]

Your telemetry pipeline is fully configured. The OpenTelemetry Collector now knows where to route logs and traces. Next up, you'll configure Grafana's data sources and write the Terraform code that brings this entire stack to life with a single command.

Provision Grafana Data Sources and Write Terraform

Your backend services (Prometheus, Loki, Tempo) are configured, but Grafana does not know they exist yet. Without data source provisioning, you would need to manually add each backend through the Grafana UI every time the stack starts.

In this step, you will provision Grafana's data sources as code so they connect automatically on startup. You will also write the Terraform configuration that makes terraform apply the single command that brings the entire stack up.

In this step, get ready to:
  • Provision Grafana data sources with trace-to-log correlation.
  • Write the Terraform configuration using the Docker provider.
  • Create the project README documenting the single deploy command.
Write the Grafana data sources provisioning file

Grafana supports file-based provisioning. A YAML file placed in Grafana's provisioning directory tells it which data sources to create on startup, including cross-service correlation settings.

  • Create the file configs/grafana/provisioning/datasources/datasources.yaml and paste the following configuration:
apiVersion: 1

datasources:
  # Prometheus - metrics backend (default data source)
  - name: Prometheus
    type: prometheus
    uid: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false

  # Loki - log aggregation with trace correlation
  - name: Loki
    type: loki
    uid: loki
    access: proxy
    url: http://loki:3100
    editable: false
    jsonData:
      derivedFields:
        - datasourceUid: tempo
          matcherRegex: "trace_id=(\\w+)"
          name: TraceID
          url: "$${__value.raw}"
          matcherType: label

  # Tempo - distributed tracing with links back to Loki logs
  - name: Tempo
    type: tempo
    uid: tempo
    access: proxy
    url: http://tempo:3200
    editable: false
    jsonData:
      tracesToLogsV2:
        datasourceUid: loki
        spanStartTimeShift: "-1h"
        spanEndTimeShift: "1h"
        filterByTraceID: true
        filterBySpanID: true
      serviceMap:
        datasourceUid: prometheus

What does this configuration do?

This file provisions three data sources that Grafana loads automatically on startup:

  • Prometheus at http://prometheus:9090 is the default data source for metrics queries.
  • Loki at http://loki:3100 has a derivedFields entry that extracts trace IDs from log lines and links them to Tempo. This lets you jump from a log entry directly to its distributed trace.
  • Tempo at http://tempo:3200 has tracesToLogsV2 linking back to Loki, so you can jump from a trace span to its associated logs. The serviceMap field uses Prometheus for topology visualization.
  • Save configs/grafana/provisioning/datasources/datasources.yaml.

Why is correlation between data sources important?

In a production incident, you typically start by seeing a metric spike (Prometheus), then want to read the relevant logs (Loki), then trace the request path (Tempo). These correlation links let you pivot between all three without manually copying IDs between tools.

Write the Terraform configuration

Terraform wraps your entire Docker Compose stack into a single managed resource. Running terraform apply brings everything up. Running terraform destroy tears everything down.

  • Create the file terraform/main.tf and paste the following configuration:
terraform {
  required_providers {
    docker = {
      source  = "kreuzwerker/docker"
      version = "4.4.0"
    }
  }
}

provider "docker" {}

resource "docker_compose" "observability_stack" {
  project_name = "observability"

  config_paths = [
    "${path.module}/../docker-compose.yaml",
  ]

  env_files = [
    "${path.module}/../.env",
  ]

  remove_orphans = true
  wait           = true
  wait_timeout   = "120s"
}

What does this Terraform file do?

The kreuzwerker/docker provider version 4.4.0 includes a docker_compose resource that manages Docker Compose applications directly using Go packages. It does not shell out to the docker compose CLI.

  • config_paths points to your docker-compose.yaml one directory up from the terraform/ folder.
  • env_files loads the .env file you created earlier before parsing the Compose configuration.
  • wait = true makes Terraform wait until all services reach a running or healthy state before returning success.
  • wait_timeout = "120s" gives the stack two minutes to become healthy (pulling images may take time on first run).

Why Terraform instead of plain Docker Compose?

You could run docker compose up -d directly and achieve the same result. Terraform adds state tracking, so it knows exactly what is deployed. It also integrates with broader infrastructure (cloud providers, DNS, secrets managers) if you scale this stack later.

For this project, the key benefit is a consistent interface: terraform apply brings the stack up. terraform destroy tears it down. One command in, one command out.

  • Save terraform/main.tf.

Seeing a provider error later?

If Terraform cannot find the provider when you run terraform init in a later step, confirm the source is exactly kreuzwerker/docker and version is "4.4.0" in your main.tf.

help me troubleshoot a Terraform provider error with kreuzwerker/docker

Create variables, outputs, and the project README

Terraform projects separate concerns into variables (inputs), outputs (useful values after apply), and documentation. You will create all three now.

  • Create the file terraform/variables.tf and paste the following:
variable "slack_webhook_url" {
  description = "Slack Incoming Webhook URL for alert notifications"
  type        = string
  sensitive   = true
  default     = ""
}

What does this variable do?

This declares a sensitive input variable for the Slack webhook URL used by Alertmanager. Marking it sensitive = true prevents Terraform from displaying its value in logs or plan output. The empty default means the stack runs without Slack integration unless you provide a URL.

  • Save terraform/variables.tf.
  • Create the file terraform/outputs.tf and paste the following:
output "grafana_url" {
  description = "Grafana dashboard URL"
  value       = "http://localhost:3000"
}

output "prometheus_url" {
  description = "Prometheus UI URL"
  value       = "http://localhost:9090"
}

output "sample_app_url" {
  description = "Sample application URL"
  value       = "http://localhost:8080"
}

output "stack_command" {
  description = "The single command to bring the stack up"
  value       = "cd terraform && terraform init && terraform apply -auto-approve"
}

What do outputs do?

After terraform apply completes, Terraform prints all output values to your terminal. This gives you quick access to every URL without memorizing port numbers. The stack_command output also serves as documentation for anyone else who clones this repo.

  • Save terraform/outputs.tf.
  • Create the file README.md in the project root and paste the following:
# Production Observability Stack (LGTM)

## One Command to Rule Them All

```powershell
cd terraform; terraform init; terraform apply -auto-approve
```

## Stack Components

| Service | Port | Purpose |
|---------|------|---------|
| Grafana | 3000 | Unified observability frontend |
| Prometheus | 9090 | Metrics collection and storage |
| Loki | 3100 | Log aggregation and querying |
| Tempo | 3200 | Distributed tracing backend |
| Alertmanager | 9093 | Alert routing and notifications |
| Node Exporter | 9100 | System-level metrics |
| Blackbox Exporter | 9115 | HTTP probe monitoring |
| OTel Collector | 4318 | Telemetry pipeline (logs + traces) |
| Sample App | 8080 | Instrumented demo application |

## Tear Down

```powershell
cd terraform; terraform destroy -auto-approve
```

## Data Retention

- Prometheus metrics: 15 days
- Loki logs: 72 hours
- Tempo traces: 24 hours

Why document the single command?

A good Infrastructure as Code project is self-documenting. Anyone who clones this repository should be able to read the README and have the entire stack running in under two minutes. The retention section also sets expectations about how long data persists across restarts.

  • Save README.md.
  • Confirm all new files exist by running:
ls configs/grafana/provisioning/datasources/
ls terraform/
ls README.md

You should see datasources.yaml in the first listing, main.tf, variables.tf, and outputs.tf in the second, and README.md in the third.

Missing files or folders?

If ls configs/grafana/provisioning/datasources/ shows an error, the directory does not exist yet. Go back to Step 1 and confirm you created the full directory structure including the configs/grafana/provisioning/datasources/ path.

help me fix missing directories in my observability project structure

✔️ Awesome, I've got everything!

Great. Double check you have saved all five files before moving on.

ⓧ I'd like to double check the full code

Here is the full content of each file created in this step:

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    uid: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false

  - name: Loki
    type: loki
    uid: loki
    access: proxy
    url: http://loki:3100
    editable: false
    jsonData:
      derivedFields:
        - datasourceUid: tempo
          matcherRegex: "trace_id=(\\w+)"
          name: TraceID
          url: "$${__value.raw}"
          matcherType: label

  - name: Tempo
    type: tempo
    uid: tempo
    access: proxy
    url: http://tempo:3200
    editable: false
    jsonData:
      tracesToLogsV2:
        datasourceUid: loki
        spanStartTimeShift: "-1h"
        spanEndTimeShift: "1h"
        filterByTraceID: true
        filterBySpanID: true
      serviceMap:
        datasourceUid: prometheus
terraform {
  required_providers {
    docker = {
      source  = "kreuzwerker/docker"
      version = "4.4.0"
    }
  }
}

provider "docker" {}

resource "docker_compose" "observability_stack" {
  project_name = "observability"

  config_paths = [
    "${path.module}/../docker-compose.yaml",
  ]

  env_files = [
    "${path.module}/../.env",
  ]

  remove_orphans = true
  wait           = true
  wait_timeout   = "120s"
}
variable "slack_webhook_url" {
  description = "Slack Incoming Webhook URL for alert notifications"
  type        = string
  sensitive   = true
  default     = ""
}
output "grafana_url" {
  description = "Grafana dashboard URL"
  value       = "http://localhost:3000"
}

output "prometheus_url" {
  description = "Prometheus UI URL"
  value       = "http://localhost:9090"
}

output "sample_app_url" {
  description = "Sample application URL"
  value       = "http://localhost:8080"
}

output "stack_command" {
  description = "The single command to bring the stack up"
  value       = "cd terraform && terraform init && terraform apply -auto-approve"
}
# Production Observability Stack (LGTM)

## One Command to Rule Them All

```powershell
cd terraform; terraform init; terraform apply -auto-approve
```

## Stack Components

| Service | Port | Purpose |
|---------|------|---------|
| Grafana | 3000 | Unified observability frontend |
| Prometheus | 9090 | Metrics collection and storage |
| Loki | 3100 | Log aggregation and querying |
| Tempo | 3200 | Distributed tracing backend |
| Alertmanager | 9093 | Alert routing and notifications |
| Node Exporter | 9100 | System-level metrics |
| Blackbox Exporter | 9115 | HTTP probe monitoring |
| OTel Collector | 4318 | Telemetry pipeline (logs + traces) |
| Sample App | 8080 | Instrumented demo application |

## Tear Down

```powershell
cd terraform; terraform destroy -auto-approve
```

## Data Retention

- Prometheus metrics: 15 days
- Loki logs: 72 hours
- Tempo traces: 24 hours

Your observability frontend is now fully wired to its backends, and Terraform is ready to orchestrate the entire stack. Next up, you will build and instrument the sample application that generates the telemetry data flowing through these pipelines.

Build and Instrument the Sample Application

Your observability backends are configured. Prometheus knows what to scrape, Loki is ready to receive logs, and Tempo is waiting for traces. But none of that matters without an application generating real telemetry data.

In this step, you will build a Python Flask application instrumented with OpenTelemetry. The app will emit traces and logs through the OTel Collector into Tempo and Loki, giving you real signals to query once the stack is running.

In this step, get ready to:
  • Write a Flask application with OpenTelemetry instrumentation that emits traces and logs.
  • Create the requirements file and Dockerfile to containerize the app.
  • Review the environment variables that connect the app to the OpenTelemetry Collector.
Write the Flask application with OpenTelemetry

The sample application serves as the source of telemetry for your entire stack. It uses the OpenTelemetry SDK to create traces for every request, and Python's logging module to emit structured log messages that flow through the OTel Collector.

  • In VS Code, create a new file called app.py inside the sample-app folder.
  • Paste in the imports and OpenTelemetry configuration by copying the following code:
import time
import random
import logging

from flask import Flask, jsonify
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.sdk.resources import Resource

# Configure OpenTelemetry
resource = Resource.create({"service.name": "sample-app"})
provider = TracerProvider(resource=resource)
otlp_exporter = OTLPSpanExporter()
provider.add_span_processor(BatchSpanProcessor(otlp_exporter))
trace.set_tracer_provider(provider)

tracer = trace.get_tracer(__name__)

# Configure logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s %(levelname)s %(message)s')
logger = logging.getLogger(__name__)

app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)

What does this code do?

This block sets up the entire telemetry pipeline for your application before any request handlers run.

  • Resource.create({"service.name": "sample-app"}) tags every trace with the service name so Tempo knows which service generated it.
  • TracerProvider with BatchSpanProcessor batches completed spans and exports them via OTLP to the collector.
  • FlaskInstrumentor().instrument_app(app) automatically creates a span for every incoming HTTP request without you writing tracing code in each handler.
  • Add the three route handlers and the main entry point below the code you just pasted:

@app.route('/')
def health():
    return jsonify({"status": "healthy"}), 200


@app.route('/api/orders')
def create_order():
    with tracer.start_as_current_span("process-order") as span:
        # Simulate variable latency
        latency = random.uniform(0.05, 0.5)
        time.sleep(latency)
        order_id = random.randint(1000, 9999)
        span.set_attribute("order.id", order_id)
        span.set_attribute("order.latency_ms", int(latency * 1000))
        logger.info(f"Order {order_id} processed successfully in {int(latency*1000)}ms")
        return jsonify({"order_id": order_id, "status": "completed", "latency_ms": int(latency * 1000)}), 200


@app.route('/api/orders/fail')
def fail_order():
    with tracer.start_as_current_span("process-order-failed") as span:
        latency = random.uniform(0.1, 1.0)
        time.sleep(latency)
        span.set_attribute("error", True)
        span.set_attribute("order.latency_ms", int(latency * 1000))
        logger.error(f"Order processing failed after {int(latency*1000)}ms - database timeout")
        return jsonify({"error": "Database connection timeout", "latency_ms": int(latency * 1000)}), 500


if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8080)

What do these endpoints do?

  • / is a health check that Blackbox Exporter probes to confirm the app is alive.
  • /api/orders simulates a successful order with random latency between 50ms and 500ms. Each call creates a process-order span with custom attributes.
  • /api/orders/fail simulates a failure returning a 500 error. The span is marked with error=True so you can filter for errors in Tempo.
  • Save sample-app/app.py.

Seeing red squiggles in VS Code?

VS Code may show import errors because the OpenTelemetry packages are not installed on your local machine. This is expected. The packages will be installed inside the Docker container when you build it.

Help me understand why my imports show errors in VS Code

✔️ Awesome, I've got everything!

Great. Double check you have saved sample-app/app.py.

ⓧ I'd like to double check the full code

import time
import random
import logging

from flask import Flask, jsonify
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.sdk.resources import Resource

resource = Resource.create({"service.name": "sample-app"})
provider = TracerProvider(resource=resource)
otlp_exporter = OTLPSpanExporter()
provider.add_span_processor(BatchSpanProcessor(otlp_exporter))
trace.set_tracer_provider(provider)

tracer = trace.get_tracer(__name__)

logging.basicConfig(level=logging.INFO, format='%(asctime)s %(levelname)s %(message)s')
logger = logging.getLogger(__name__)

app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)


@app.route('/')
def health():
    return jsonify({"status": "healthy"}), 200


@app.route('/api/orders')
def create_order():
    with tracer.start_as_current_span("process-order") as span:
        latency = random.uniform(0.05, 0.5)
        time.sleep(latency)
        order_id = random.randint(1000, 9999)
        span.set_attribute("order.id", order_id)
        span.set_attribute("order.latency_ms", int(latency * 1000))
        logger.info(f"Order {order_id} processed successfully in {int(latency*1000)}ms")
        return jsonify({"order_id": order_id, "status": "completed", "latency_ms": int(latency * 1000)}), 200


@app.route('/api/orders/fail')
def fail_order():
    with tracer.start_as_current_span("process-order-failed") as span:
        latency = random.uniform(0.1, 1.0)
        time.sleep(latency)
        span.set_attribute("error", True)
        span.set_attribute("order.latency_ms", int(latency * 1000))
        logger.error(f"Order processing failed after {int(latency*1000)}ms - database timeout")
        return jsonify({"error": "Database connection timeout", "latency_ms": int(latency * 1000)}), 500


if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8080)
Create the requirements file and Dockerfile

Your app needs its Python dependencies listed in a requirements file, and a Dockerfile that tells Docker how to package the app into a container image.

  • Create a new file called requirements.txt inside the sample-app folder and paste in the following dependencies:
flask==3.1.1
opentelemetry-api==1.33.0
opentelemetry-sdk==1.33.0
opentelemetry-exporter-otlp-proto-http==1.33.0
opentelemetry-instrumentation-flask==0.54b0

Why does the instrumentation package have a different version?

The opentelemetry-instrumentation-flask package uses version 0.54b0 while the core SDK uses 1.33.0. OpenTelemetry instrumentation packages are still in beta, so they follow a separate versioning scheme. This is standard across the OpenTelemetry Python ecosystem.

  • Save sample-app/requirements.txt.
  • Create another new file called Dockerfile inside the sample-app folder and paste in the following:
FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

EXPOSE 8080

CMD ["python", "app.py"]

What does this Dockerfile do?

  • FROM python:3.12-slim uses a lightweight Python base image to keep the container small.
  • COPY requirements.txt is copied first so Docker caches the dependency install layer. Changing app.py later does not re-download all packages.
  • EXPOSE 8080 documents the port the Flask app listens on. The actual port mapping is handled in docker-compose.yaml.
  • Save sample-app/Dockerfile.
  • Confirm your sample-app folder now contains three files: app.py, requirements.txt, and Dockerfile in the VS Code Explorer sidebar.

Missing one of the files?

Make sure the files are inside the sample-app/ folder, not in the project root. The Dockerfile has no file extension. If you accidentally named it Dockerfile.txt, rename it to just Dockerfile.

Help me fix my sample-app file structure

Review the OpenTelemetry environment variables

Your app knows how to create traces and logs, but it needs to know where to send them. The docker-compose.yaml you created earlier already configures this through environment variables on the sample-app service.

  • Open docker-compose.yaml and scroll to the sample-app service. Find the environment section:
    environment:
      - OTEL_SERVICE_NAME=sample-app
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
      - OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
      - OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true

What do these environment variables control?

  • OTEL_SERVICE_NAME identifies this service in trace views. When you search for traces in Tempo, you will filter by this name.
  • OTEL_EXPORTER_OTLP_ENDPOINT tells the OpenTelemetry SDK where to send telemetry. It points to the OTel Collector on port 4318 (OTLP HTTP).
  • OTEL_EXPORTER_OTLP_PROTOCOL specifies the encoding format. http/protobuf is the recommended default for OTLP over HTTP.
  • OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED connects Python's standard logging module to the OTel pipeline, so log messages are exported alongside traces.

How does telemetry flow from app to backend?

The sample app sends traces and logs to the OTel Collector on port 4318. The Collector batches this data and routes it to the correct backend: traces go to Tempo on port 4317, and logs go to Loki at /otlp. You configured these routing rules in otel-collector-config.yaml in a previous step.

Your sample application is ready to be built and deployed. In the next step, you will bring the entire stack online with a single Terraform command and verify that metrics, logs, and traces are flowing end-to-end.

Deploy the Stack and Define the Four Golden Signals

You have written every configuration file your observability platform needs. Now it is time to bring the entire stack to life with a single command and prove that telemetry is flowing end-to-end.

In this step you will deploy all 9 services, generate realistic traffic, and then formalize the Four Golden Signals as documented PromQL expressions. These signals are the same measurements Google uses to assess the health of any service.

In this step, get ready to:
  • Deploy the full LGTM stack with terraform apply and verify all services are healthy.
  • Generate traffic to produce metrics, logs, and traces across the stack.
  • Define the Four Golden Signals as measurable PromQL SLI expressions.
Deploy the stack with Terraform

Terraform needs to download the kreuzwerker/docker provider before it can manage your Docker Compose application. The terraform init command handles this automatically.

  • Switch back to your PowerShell terminal.
  • Navigate into the terraform/ directory by running this command:
cd terraform
  • Initialize the Terraform working directory by running:
terraform init

What does terraform init do?

This command downloads the kreuzwerker/docker provider plugin specified in your main.tf and creates a .terraform directory to store it.

You only need to run init once per project (or again if you change providers). After initialization, terraform apply and terraform destroy become available.

  • You should see a message ending with "Terraform has been successfully initialized!" in your terminal.

Seeing a provider download error?

Make sure Docker Desktop is running. The kreuzwerker/docker provider requires the Docker daemon to be active during initialization.

Also confirm you are inside the terraform/ directory, not the project root. Terraform needs to find main.tf in the current directory.

Help me troubleshoot why terraform init is failing

  • Deploy the entire observability stack by running:
terraform apply -auto-approve

What does -auto-approve do?

Normally terraform apply asks you to type "yes" to confirm changes. The -auto-approve flag skips that prompt so the stack deploys immediately.

Behind the scenes, Terraform calls docker compose up -d using your docker-compose.yaml. It then waits up to 120 seconds for all health checks to pass before marking the deployment as successful.

This command will take 1-2 minutes as Docker pulls images and builds the sample app container. Wait until you see the "Apply complete!" message with your output URLs.

  • Verify Grafana is running by opening your browser and navigating to http://localhost:3000.
  • Log in with username admin and password admin.
  • Grafana will prompt you to change the password. Click Skip for now.
  • Verify Prometheus by navigating to http://localhost:9090 in a new tab. You should see the Prometheus query interface.
  • Verify the sample app by navigating to http://localhost:8080. You should see a JSON response: {"status": "healthy"}.

Containers not starting?

Run docker ps to see which containers are running. If any show a status of "Restarting", check its logs with docker logs <container-name> to find the configuration error.

Common causes: a YAML indentation error in a config file, or a port conflict if another application is already using one of the mapped ports (3000, 9090, 8080, etc.).

Help me debug why my observability containers are not starting

Generate traffic and verify telemetry

An observability stack with no traffic is like a security camera pointed at an empty room. You need requests flowing through the sample app to produce metrics, logs, and traces.

  • Open a new PowerShell terminal tab (your current terminal is still inside the terraform/ directory).
  • Generate a mix of successful and failing requests by running this loop:
# Send 50 requests to both endpoints to generate metrics, logs, and traces
for ($i = 1; $i -le 50; $i++) {
    Invoke-WebRequest -Uri http://localhost:8080/api/orders -UseBasicParsing | Out-Null
    Invoke-WebRequest -Uri http://localhost:8080/api/orders/fail -UseBasicParsing | Out-Null
    Start-Sleep -Milliseconds 500
}
Write-Host "Traffic generation complete!"

What does this loop do?

This sends 50 requests to /api/orders (returns 200) and 50 to /api/orders/fail (returns 500). Each pair is spaced 500ms apart, so the full loop takes about 25 seconds.

The successful endpoint simulates normal order processing with variable latency. The failure endpoint simulates a database timeout. Together they generate a realistic mix of telemetry signals.

Wait for the loop to finish. You should see "Traffic generation complete!" printed in your terminal.

  • Verify metrics in Prometheus: switch to the Prometheus tab at http://localhost:9090.
  • Type up in the query box and click Execute. You should see all your scrape targets listed with a value of 1 (meaning healthy).
  • Verify logs in Loki: switch to Grafana at http://localhost:3000.
  • Click the hamburger menu icon (three horizontal lines) in the top-left corner.
  • Click Explore.
  • In the data source dropdown at the top, select Loki.
  • Enter {service_name="sample-app"} in the query field and click Run query. You should see log lines from the sample app including order processing messages.
  • Verify traces in Tempo: change the data source dropdown to Tempo.
  • Select the Search tab and click Run query. You should see trace entries for process-order and process-order-failed spans.

No logs or traces appearing?

Telemetry takes a few seconds to flow through the OpenTelemetry Collector's batch processor (configured with a 5-second timeout). Wait 10 seconds after the traffic loop finishes, then try your query again.

If logs still do not appear, check that the otel-collector container is running with docker ps. Then check its logs with docker logs otel-collector for connection errors to Loki or Tempo.

Help me figure out why logs and traces are not appearing in Grafana

Define the Four Golden Signals as PromQL SLIs

Google's Site Reliability Engineering book defines four signals that every service should measure: Latency, Traffic, Errors, and Saturation. You will now formalize these as PromQL queries in a documentation file.

  • Switch back to VS Code.
  • Create a docs/ folder inside your project root (at the same level as terraform/ and configs/).
  • Create a new file called slis.md inside the docs/ folder.
  • Paste the following content into docs/slis.md:
# Service Level Indicators (SLIs) - Four Golden Signals

## 1. Latency - How long does it take to serve a request?

**Successful request latency (p99):**
histogram_quantile(0.99, sum(rate(flask_http_request_duration_seconds_bucket{status!~"5.."}[5m])) by (le))

**Error request latency (p99):**
histogram_quantile(0.99, sum(rate(flask_http_request_duration_seconds_bucket{status=~"5.."}[5m])) by (le))

## 2. Traffic - How much demand is the system handling?

**Requests per second:**
sum(rate(flask_http_request_total[5m]))

## 3. Errors - Rate of failed requests

**Error ratio (5xx responses / total responses):**
sum(rate(flask_http_request_total{status=~"5.."}[5m])) / sum(rate(flask_http_request_total[5m]))

## 4. Saturation - How full is the service?

**CPU utilization:**
1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m]))

**Memory utilization:**
1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)

What are the Four Golden Signals?

Latency measures how long requests take. Splitting by success vs error status reveals if errors are slow (timeout) or fast (immediate rejection).

Traffic measures demand on the system in requests per second. A sudden drop often indicates an upstream failure.

Errors measures the ratio of failed requests (5xx) to total requests. This is your most immediate signal that something is broken.

Saturation measures how full your system's resources are. CPU and memory utilization approaching 100% means the service is at risk of degradation.

  • Save docs/slis.md.
  • Verify each query works by switching to the Prometheus UI at http://localhost:9090.
  • Paste the Traffic query into the expression box:
sum(rate(flask_http_request_total[5m]))
  • Click Execute. You should see a numeric value greater than 0, confirming that Prometheus is scraping request metrics from the sample app.
  • Try the Error ratio query:
sum(rate(flask_http_request_total{status=~"5.."}[5m])) / sum(rate(flask_http_request_total[5m]))
  • Click Execute. You should see a value around 0.5 (since you sent an equal number of successful and failing requests).

Query returning empty or 'no data'?

Prometheus needs at least one full scrape interval (15 seconds) after traffic is generated before rate calculations return data. If you just finished the traffic loop, wait 30 seconds and try again.

If the metric name is not recognized, confirm that the sample-app scrape target is healthy by navigating to Status > Targets in the Prometheus UI. All targets should show a green "UP" state.

Help me debug why my PromQL queries return no data

✔️ Awesome, I've got everything!

Great. Make sure you have saved docs/slis.md and confirmed at least one PromQL query returns data in Prometheus.

ⓧ I'd like to double check the full code

Here is the complete docs/slis.md file:

# Service Level Indicators (SLIs) - Four Golden Signals

## 1. Latency - How long does it take to serve a request?

**Successful request latency (p99):**
histogram_quantile(0.99, sum(rate(flask_http_request_duration_seconds_bucket{status!~"5.."}[5m])) by (le))

**Error request latency (p99):**
histogram_quantile(0.99, sum(rate(flask_http_request_duration_seconds_bucket{status=~"5.."}[5m])) by (le))

## 2. Traffic - How much demand is the system handling?

**Requests per second:**
sum(rate(flask_http_request_total[5m]))

## 3. Errors - Rate of failed requests

**Error ratio (5xx responses / total responses):**
sum(rate(flask_http_request_total{status=~"5.."}[5m])) / sum(rate(flask_http_request_total[5m]))

## 4. Saturation - How full is the service?

**CPU utilization:**
1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m]))

**Memory utilization:**
1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)

Your observability stack is live and collecting real telemetry. Every metric, log, and trace is flowing through the pipeline you built from scratch. Next up, you will take on the Secret Mission and build a Grafana dashboard that visualizes these Four Golden Signals in real-time panels.

Secret mission

Build a Four Golden Signals Dashboard

Your observability stack collects metrics, but nobody wants to write PromQL queries by hand every time they check system health. Build a Grafana dashboard that visualizes all Four Golden Signals in real-time panels, provisioned as code so it appears automatically on every stack deployment.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally using Docker containers, so there are no ongoing cloud costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • 9 Docker containers (Prometheus, Loki, Tempo, Grafana, Alertmanager, Node Exporter, Blackbox Exporter, OTel Collector, Sample App) consuming approximately 2-4GB RAM total.
  • 5 Docker volumes storing persistent data (prometheus-data, loki-data, tempo-data, grafana-data, alertmanager-data).
  • Project directory on your Desktop containing all Terraform, Docker Compose, and configuration files.

Keep everything running

No action needed. Choose this if you are still actively building or plan to continue with Project 2 (SLOs and Alerting) and Project 3 (DORA Metrics and Game Day), which build on this same stack.

  • Your containers will continue running and consuming approximately 2-4GB of RAM.
  • Prometheus retains metrics for 15 days, Loki retains logs for 72 hours, and Tempo retains traces for 24 hours.
  • The stack restarts automatically if Docker Desktop restarts, thanks to the restart: unless-stopped policy.

Pause - I'll come back to this later

Shut down all running containers to free up RAM and CPU. Your configuration files and Terraform state stay intact so you can bring everything back with a single command.

  • Open PowerShell and navigate to your project's terraform/ directory.
  • Run the following command to stop and remove all containers:
cd terraform
terraform destroy -auto-approve
  • All 9 containers stop and are removed. Your Docker volumes (persistent data) remain intact.
  • To bring the stack back up later, run:
terraform init
terraform apply -auto-approve

Your Grafana dashboards, Prometheus metrics, and Loki logs will still be there when you return (within their retention windows).

Delete - I don't want to use this again

Remove all project containers, volumes, and files from your machine. This is a full cleanup that cannot be undone.

  • Open PowerShell and navigate to your project's terraform/ directory.
  • Destroy all containers managed by Terraform:
cd terraform
terraform destroy -auto-approve
  • Remove the Docker volumes that stored persistent data:
docker volume prune -f
  • Navigate up to your Desktop and delete the entire project folder:
cd ~\Desktop
Remove-Item -Recurse -Force .\observability-stack

Adjust the folder name

If you named your project folder something other than observability-stack, replace it with the actual folder name you created in Step 1.

  • Confirm the folder is gone by running:
ls ~\Desktop

Your project folder should no longer appear in the output.

Nice Work!

Nice Work!

You just built a production-grade observability platform from scratch. Your entire LGTM stack is running locally, provisioned as Infrastructure as Code, collecting metrics, aggregating logs, and tracing requests across your system.

You've learned how to:

  • Deployed a complete LGTM stack (Prometheus, Loki, Tempo, Grafana) with a single terraform apply command.
  • Configured an OpenTelemetry Collector pipeline routing logs to Loki and traces to Tempo, with trace-to-log correlation across Grafana data sources.
  • Defined the Four Golden Signals (Latency, Traffic, Errors, Saturation) as measurable PromQL SLI expressions backed by real application telemetry.
  • Built a provisioned-as-code Grafana dashboard that visualizes all Four Golden Signals automatically on stack startup.

Ready to quiz yourself?