Spring Boot Full Observability Stack

Wire OpenTelemetry, Prometheus, Loki, Tempo, and Grafana to a chaos service.

Introduction

30 Second Summary

When something breaks in production, you need to know immediately what went wrong, where, and why. The tools that give you this visibility are called your "observability stack."

In this project, you will build a Spring Boot 4 chaos service that deliberately creates common production failures, then wire it to a full observability pipeline so you can practice finding and diagnosing those failures in real time.

What You'll Build

A Spring Boot 4 "Chaos Service" connected to a local observability stack with OpenTelemetry Collector, Prometheus, Loki, Tempo, and Grafana.

By the end of this project, you'll have:

  • A Spring Boot 4 chaos service with endpoints that simulate slow responses, random exceptions, and memory pressure.
  • A Docker Compose stack with five observability services (OTel Collector, Prometheus, Loki, Tempo, Grafana) wired together through explicit configuration.
  • Hands-on experience writing PromQL and LogQL queries to troubleshoot the chaos scenarios using Grafana Explore.
  • Secret Mission: Build a JVM monitoring dashboard in Grafana with panels for heap memory, request rates, error percentages, and GC pauses.

Are there any prerequisites?

This project assumes you have Java and Spring Boot development experience. You should also be familiar with Docker and Docker Compose basics (running containers, reading compose files).

Create the Spring Boot 4 Chaos Service

Before you can observe anything, you need something that misbehaves. Production systems fail in predictable patterns: endpoints slow down under load, exceptions bubble up from race conditions, and memory usage creeps toward limits.

In this step, you'll create a Spring Boot 4 service that deliberately triggers these failures on demand. You'll use Spring Initializr to scaffold the project and write a controller with three chaos endpoints.

In this step, get ready to:
  • Generate a Spring Boot 4 project from Spring Initializr.
  • Create a chaos controller with slow, error, and memory-pressure endpoints.
  • Verify the service starts and responds to requests.
Generate the Spring Boot 4 project

Spring Initializr is a web tool that generates a ready-to-run Spring Boot project with your chosen dependencies pre-configured. You'll use it to create a Maven project with the exact starter libraries this project needs.

  • Open start.spring.io in your browser.
  • Under **Project**, select Maven.
  • Under Language, select Java.
  • Under Spring Boot, select 4.0.6.
  • In the Project Metadata section, set Artifact to chaos-service.
  • Leave Group as com.example.
  • Set Java to 25.
  • Click ADD DEPENDENCIES and search for Spring Web. Select it.
  • Search for Spring Boot Actuator and add it.
  • Search for OpenTelemetry and add it.
  • Click GENERATE to download the project as a zip file.

What did Spring Initializr just create?

The downloaded zip contains a complete Maven project with a pom.xml that references spring-boot-starter-web, spring-boot-starter-actuator, and spring-boot-starter-opentelemetry. It also includes a Maven Wrapper (mvnw) so you can build without installing Maven globally.

The main application class ChaosServiceApplication.java is already generated for you. You'll find it at src/main/java/com/example/chaosservice/.

  • Extract the downloaded zip file to a location you can find easily (such as your Desktop).
  • Open IntelliJ IDEA.
  • Click Open on the Welcome Screen.
  • Navigate to the extracted chaos-service folder and select the pom.xml file inside it.
  • In the dialog that appears, click Open as Project.
  • If IntelliJ shows a Trust Project dialog, click Trust Project.

IntelliJ will import the Maven project automatically. Wait for the import to finish (you'll see a progress bar in the bottom-right corner).

IntelliJ not resolving dependencies?

If you see red underlines on import statements after the project loads, right-click the pom.xml in the Project tool window and select Maven then Reload project. This forces IntelliJ to re-download all dependencies.

Help me resolve Maven dependency issues in IntelliJ

✔️ Awesome, I've got everything!

Great. Double check your pom.xml includes all three starter dependencies, then move on to writing the controller.

ⓧ I'd like to double check the full code

Here is the full pom.xml that Spring Initializr generated. Confirm yours matches the key sections (parent version, Java version, dependencies):

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>4.0.6</version>
        <relativePath/>
    </parent>
    <groupId>com.example</groupId>
    <artifactId>chaos-service</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>chaos-service</name>
    <description>Demo project for Spring Boot</description>
    <properties>
        <java.version>25</java.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-opentelemetry</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>
</project>
Create the chaos controller

The chaos controller is a REST endpoint class that simulates three common production failures. Each endpoint triggers a different type of problem so you have realistic telemetry to observe later.

  • In IntelliJ, expand src/main/java/com/example/chaosservice/ in the Project tool window.
  • Right-click the chaosservice package and select New then Java Class.
  • Name it ChaosController and press Enter.
  • Replace all the contents of the new file with the following code:
package com.example.chaosservice;

import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.ArrayList;
import java.util.List;
import java.util.Random;

@RestController
@RequestMapping("/chaos")
public class ChaosController {

    private final Random random = new Random();

    // Simulates a slow response with a random delay between 2-5 seconds
    @GetMapping("/slow")
    public String slow() throws InterruptedException {
        int delay = 2000 + random.nextInt(3001);
        Thread.sleep(delay);
        return "Responded after " + delay + "ms delay";
    }
}

What does this code do?

  • `@RestController` marks this class as an HTTP endpoint handler. Spring Boot automatically maps incoming requests to its methods.
  • `@RequestMapping("/chaos")` prefixes all endpoint paths in this class with /chaos.
  • The /slow endpoint generates a random delay between 2000ms and 5000ms using Thread.sleep(). This simulates a database query or external API call that's taking too long.
  • Save the file.
  • Confirm IntelliJ shows no red underlines in the editor. The file should compile cleanly.

Now add the error endpoint. This method randomly throws one of three exception types to simulate unpredictable production crashes.

  • Add the following method inside the ChaosController class, below the slow() method:
    // Randomly throws one of several exception types to simulate production errors
    @GetMapping("/error")
    public String error() {
        int type = random.nextInt(3);
        switch (type) {
            case 0 -> throw new NullPointerException("Simulated null pointer");
            case 1 -> throw new IllegalStateException("Simulated illegal state");
            case 2 -> throw new OutOfMemoryError("Simulated OOM condition");
        }
        return "No error this time";
    }

What does this code do?

  • `random.nextInt(3)` picks a number between 0 and 2, then the switch statement throws a different exception type each time.
  • These are the same exception types you'd see in real production: a NullPointerException from missing data, an IllegalStateException from a race condition, or an OutOfMemoryError from a resource leak.
  • Save the file and confirm no red underlines appear.

Finally, add the memory-pressure endpoint. This method allocates a large amount of memory, holds it briefly, then releases it.

  • Add the following method below the error() method, still inside the class:
    // Allocates a large list to simulate memory pressure, then releases it
    @GetMapping("/memory")
    public String memory() {
        List<byte[]> memoryHog = new ArrayList<>();
        for (int i = 0; i < 50; i++) {
            memoryHog.add(new byte[1024 * 1024]); // 1MB chunks, ~50MB total
        }
        memoryHog.clear();
        return "Allocated and released 50MB";
    }

What does this code do?

The method creates an ArrayList and fills it with 50 byte arrays of 1MB each. This forces the JVM to allocate roughly 50MB of heap memory. After the loop, memoryHog.clear() releases all references so the garbage collector can reclaim the memory.

In a real system, this pattern mimics what happens when a large report query loads too much data into memory before releasing it.

  • Save ChaosController.java.
  • Confirm no red underlines or compilation errors appear anywhere in the file.

Seeing red underlines on List or ArrayList?

Make sure your import section at the top of the file includes import java.util.ArrayList; and import java.util.List;. If IntelliJ shows a red lightbulb, press Alt+Enter (Windows) or Option+Enter (macOS) to auto-import the missing classes.

Help me fix import errors in my ChaosController

✔️ Awesome, I've got everything!

Great. Make sure you've saved ChaosController.java and there are no compilation errors.

ⓧ I'd like to double check the full code

Here is the complete ChaosController.java with all three endpoints:

package com.example.chaosservice;

import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.ArrayList;
import java.util.List;
import java.util.Random;

@RestController
@RequestMapping("/chaos")
public class ChaosController {

    private final Random random = new Random();

    // Simulates a slow response with a random delay between 2-5 seconds
    @GetMapping("/slow")
    public String slow() throws InterruptedException {
        int delay = 2000 + random.nextInt(3001);
        Thread.sleep(delay);
        return "Responded after " + delay + "ms delay";
    }

    // Randomly throws one of several exception types to simulate production errors
    @GetMapping("/error")
    public String error() {
        int type = random.nextInt(3);
        switch (type) {
            case 0 -> throw new NullPointerException("Simulated null pointer");
            case 1 -> throw new IllegalStateException("Simulated illegal state");
            case 2 -> throw new OutOfMemoryError("Simulated OOM condition");
        }
        return "No error this time";
    }

    // Allocates a large list to simulate memory pressure, then releases it
    @GetMapping("/memory")
    public String memory() {
        List<byte[]> memoryHog = new ArrayList<>();
        for (int i = 0; i < 50; i++) {
            memoryHog.add(new byte[1024 * 1024]); // 1MB chunks, ~50MB total
        }
        memoryHog.clear();
        return "Allocated and released 50MB";
    }
}
Verify the app starts and responds

With all three endpoints written, it's time to run the service and confirm each chaos endpoint behaves as expected.

  • Open the terminal in IntelliJ by pressing Alt+F12 (Windows) or Option+F12 (macOS).
  • Start the application by running this command:
./mvnw spring-boot:run

Windows users

If you're using PowerShell, run .\mvnw.cmd spring-boot:run instead. If you're using Git Bash inside IntelliJ's terminal, the ./mvnw command works as written.

Wait for the application to start. You'll see a line in the terminal that says the application started on port 8080.

  • Open your browser and navigate to http://localhost:8080/chaos/slow.
  • Wait 2-5 seconds. You should see a message like Responded after 3241ms delay.
  • Navigate to http://localhost:8080/chaos/error.
  • You should see a 500 error page. Refresh a few times and notice the error type changes randomly.
  • Navigate to http://localhost:8080/chaos/memory.
  • You should see Allocated and released 50MB almost instantly.

App won't start or port 8080 is in use?

If you see an error about port 8080 being in use, another process is occupying that port. Stop it or add server.port=8081 to src/main/resources/application.properties and use that port in your browser instead.

If the build itself fails, check that your JDK version is 25 by running java -version in the terminal. Spring Boot 4.0.6 requires JDK 25 or higher.

Help me troubleshoot my Spring Boot app not starting

Your chaos service is running and ready to misbehave on command. Next up, you'll stand up the full observability stack that will capture the traces, metrics, and logs this service generates.

Stand Up the Observability Stack

Your chaos service is generating realistic failures. Now you need the infrastructure to see them.

In this step, you will deploy five observability services using Docker Compose. Each service handles a specific role: the OpenTelemetry Collector receives telemetry and routes it, Prometheus stores metrics, Loki stores logs, Tempo stores traces, and Grafana visualizes everything.

In this step, get ready to:
  • Create a Docker Compose file with five observability services.
  • Write configuration files for each backend (OTel Collector, Prometheus, Loki, Tempo).
  • Pre-provision Grafana datasources and verify the full stack is healthy.
Create the Docker Compose file

Docker Compose lets you define all five services in a single file. Each service gets its own container, port mapping, and configuration mount.

  • In IntelliJ, create a new file called docker-compose.yml in the root of your chaos-service project folder.
  • Paste in the following Docker Compose configuration:
# Observability stack for the chaos service
services:
  # OpenTelemetry Collector - receives and routes telemetry
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.153.0
    volumes:
      - ./otel-collector-config.yaml:/etc/otelcol/config.yaml
    ports:
      - "4318:4318"

  # Prometheus - stores metrics
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

  # Loki - stores logs
  loki:
    image: grafana/loki:latest
    volumes:
      - ./loki-config.yaml:/etc/loki/local-config.yaml
    command: -config.file=/etc/loki/local-config.yaml
    ports:
      - "3100:3100"

  # Tempo - stores traces
  tempo:
    image: grafana/tempo:latest
    volumes:
      - ./tempo-config.yaml:/etc/tempo.yaml
    command: ["-config.file=/etc/tempo.yaml"]
    ports:
      - "3200:3200"

  # Grafana - visualizes all signals
  grafana:
    image: grafana/grafana:13.0.2
    volumes:
      - ./grafana/datasources.yml:/etc/grafana/provisioning/datasources/datasources.yml
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    ports:
      - "3000:3000"

What does this file do?

Each service block maps a Docker image to a local port and mounts a configuration file from your project into the container.

  • otel-collector listens on port 4318 for OTLP HTTP traffic from your Spring Boot app.
  • prometheus scrapes metrics from the collector and exposes its query UI on port 9090.
  • loki accepts log pushes on port 3100.
  • tempo receives traces on its internal gRPC port and exposes the query API on 3200.
  • grafana provides the visualization UI on port 3000.
  • Save docker-compose.yml.
Create the configuration files

Each backend needs its own YAML configuration to know how to store data and listen for connections. The OTel Collector is the central routing engine that distributes signals to the right backends.

  • Create a new file called otel-collector-config.yaml in the project root and paste in the following:
# OTel Collector config: receive OTLP, route to backends
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  # Send traces to Tempo via OTLP gRPC
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

  # Expose metrics for Prometheus to scrape
  prometheus:
    endpoint: 0.0.0.0:8889

  # Push logs to Loki
  loki:
    endpoint: http://loki:3100/loki/api/v1/push

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      exporters: [loki]

How does the routing work?

The service.pipelines section is the heart of the Collector. Each pipeline binds one receiver to one or more exporters by signal type.

Traces go to Tempo via OTLP gRPC. Metrics are exposed on port 8889 in Prometheus format so Prometheus can scrape them. Logs are pushed directly to Loki's HTTP API.

  • Save otel-collector-config.yaml.
  • Create prometheus.yml in the project root and paste in the following:
# Prometheus config: scrape metrics from the OTel Collector
scrape_configs:
  - job_name: otel-collector
    scrape_interval: 15s
    static_configs:
      - targets: ["otel-collector:8889"]

What does Prometheus scrape?

Prometheus uses a pull model. Every 15 seconds it fetches metrics from the OTel Collector's /metrics endpoint on port 8889. This is how your Spring Boot metrics end up in Prometheus without a direct connection between the two.

  • Save prometheus.yml.
  • Create loki-config.yaml in the project root and paste in the following:
# Loki config: minimal local storage
auth_enabled: false

server:
  http_listen_port: 3100

common:
  instance_addr: 127.0.0.1
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h
  • Save loki-config.yaml.
  • Create tempo-config.yaml in the project root and paste in the following:
# Tempo config: local storage with OTLP receiver
server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

storage:
  trace:
    backend: local
    wal:
      path: /tmp/tempo/wal
    local:
      path: /tmp/tempo/blocks

Why does Tempo also have an OTLP receiver?

Tempo natively accepts OTLP traces on ports 4317 (gRPC) and 4318 (HTTP). The OTel Collector connects to Tempo's gRPC port directly, which is the most efficient transport for trace spans.

  • Save tempo-config.yaml.
Provision Grafana datasources and start the stack

Grafana can auto-configure datasources at startup if you provide a provisioning YAML. This saves you from clicking through the UI to add Prometheus, Loki, and Tempo manually.

  • Create a grafana directory inside your project root.
  • Inside the grafana directory, create a file called datasources.yml and paste in the following:
# Grafana datasource provisioning
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100

  - name: Tempo
    type: tempo
    access: proxy
    url: http://tempo:3200

Why use Docker service names as URLs?

Inside a Docker Compose network, each service can reach another by its service name. Grafana resolves http://prometheus:9090 because both containers share the same default network created by Compose.

  • Save grafana/datasources.yml.
  • Open a terminal in IntelliJ and make sure you are in the chaos-service project root.
  • Start all five containers in detached mode by running:
docker compose up -d
  • Confirm all containers are running by running:
docker compose ps

You should see five containers listed with a status of running.

  • Open your browser and navigate to http://localhost:3000.
  • Log in with username admin and password admin.
  • Click Connections in the left sidebar.
  • Select Data sources.

You should see three datasources listed: Prometheus, Loki, and Tempo.

Containers not starting?

  • Make sure Docker Desktop is running before you execute docker compose up -d.
  • Check the logs for a specific container by running docker compose logs otel-collector (replace with the failing service name).
  • If you see port conflicts, make sure ports 3000, 3100, 3200, 4318, and 9090 are not already in use on your machine.

Help me troubleshoot my Docker Compose observability stack

✔️ Awesome, I've got everything!

Great. Make sure you have saved all your configuration files before moving on.

ⓧ I'd like to double check the full code

Here is the complete docker-compose.yml for comparison:

# Observability stack for the chaos service
services:
  # OpenTelemetry Collector - receives and routes telemetry
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.153.0
    volumes:
      - ./otel-collector-config.yaml:/etc/otelcol/config.yaml
    ports:
      - "4318:4318"

  # Prometheus - stores metrics
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

  # Loki - stores logs
  loki:
    image: grafana/loki:latest
    volumes:
      - ./loki-config.yaml:/etc/loki/local-config.yaml
    command: -config.file=/etc/loki/local-config.yaml
    ports:
      - "3100:3100"

  # Tempo - stores traces
  tempo:
    image: grafana/tempo:latest
    volumes:
      - ./tempo-config.yaml:/etc/tempo.yaml
    command: ["-config.file=/etc/tempo.yaml"]
    ports:
      - "3200:3200"

  # Grafana - visualizes all signals
  grafana:
    image: grafana/grafana:13.0.2
    volumes:
      - ./grafana/datasources.yml:/etc/grafana/provisioning/datasources/datasources.yml
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    ports:
      - "3000:3000"

Here is the complete otel-collector-config.yaml:

# OTel Collector config: receive OTLP, route to backends
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  # Send traces to Tempo via OTLP gRPC
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

  # Expose metrics for Prometheus to scrape
  prometheus:
    endpoint: 0.0.0.0:8889

  # Push logs to Loki
  loki:
    endpoint: http://loki:3100/loki/api/v1/push

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      exporters: [loki]

Here is the complete prometheus.yml:

# Prometheus config: scrape metrics from the OTel Collector
scrape_configs:
  - job_name: otel-collector
    scrape_interval: 15s
    static_configs:
      - targets: ["otel-collector:8889"]

Here is the complete loki-config.yaml:

# Loki config: minimal local storage
auth_enabled: false

server:
  http_listen_port: 3100

common:
  instance_addr: 127.0.0.1
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

Here is the complete tempo-config.yaml:

# Tempo config: local storage with OTLP receiver
server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

storage:
  trace:
    backend: local
    wal:
      path: /tmp/tempo/wal
    local:
      path: /tmp/tempo/blocks

Here is the complete grafana/datasources.yml:

# Grafana datasource provisioning
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100

  - name: Tempo
    type: tempo
    access: proxy
    url: http://tempo:3200

Your observability infrastructure is live. Next up, you will wire Spring Boot to export traces, metrics, and logs through this pipeline.

Configure Spring Boot OTLP Export

Your observability stack is running and ready to receive telemetry data. Now the chaos service needs to actually send its traces, metrics, and logs to the OTel Collector.

Spring Boot 4 ships an OpenTelemetry starter that auto-configures trace and metric export via OTLP. Log export requires extra wiring: a Logback appender that captures log events and an initializer bean that connects the appender to the OpenTelemetry SDK.

In this step, get ready to:
  • Configure application.yml to point all three telemetry signals at the OTel Collector.
  • Create a Logback appender and initializer bean to export logs via OTLP.
  • Verify traces, metrics, and logs are flowing into Grafana.
Configure application.yml for OTLP export

Spring Boot uses properties in application.yml to decide where to send telemetry. You need to tell it the OTel Collector's OTLP HTTP endpoint for traces, metrics, and logs.

  • In IntelliJ IDEA, open src/main/resources/application.yml (create it if only application.properties exists).
  • Replace any existing content with the following configuration:
spring:
  application:
    name: chaos-service

management:
  endpoints:
    web:
      exposure:
        include: "*"
  tracing:
    sampling:
      probability: 1.0
  opentelemetry:
    tracing:
      export:
        otlp:
          endpoint: http://localhost:4318
    logging:
      export:
        otlp:
          endpoint: http://localhost:4318/v1/logs
  otlp:
    metrics:
      export:
        url: http://localhost:4318/v1/metrics

What does this configuration do?

Each property group targets a different telemetry signal:

  • management.opentelemetry.tracing.export.otlp.endpoint sends traces to the OTel Collector over OTLP HTTP on port 4318.
  • management.otlp.metrics.export.url sends metrics from Micrometer's OtlpMeterRegistry to the same collector.
  • management.opentelemetry.logging.export.otlp.endpoint configures the OTLP endpoint for log records.
  • management.tracing.sampling.probability set to 1.0 means every request is traced (the default is 10%).
  • Save application.yml.

Why sample at 100%?

In production, sampling at 100% generates enormous volumes of trace data. For local development, full sampling ensures every chaos endpoint call produces a visible trace in Tempo so you can verify the pipeline works.

Create the Logback appender and initializer bean

Spring Boot does not auto-install a Logback appender for OpenTelemetry. You need two pieces: an XML file that registers the appender with Logback, and a Java bean that connects the appender to the Spring-managed OpenTelemetry instance.

  • Add the OpenTelemetry Logback appender dependency to your pom.xml inside the <dependencies> section:
<dependency>
    <groupId>io.opentelemetry.instrumentation</groupId>
    <artifactId>opentelemetry-logback-appender-1.0</artifactId>
    <version>2.21.0-alpha</version>
</dependency>

What is this dependency?

The opentelemetry-logback-appender-1.0 artifact provides the Logback appender class that captures log events and forwards them to the OpenTelemetry SDK. It is maintained separately from Spring Boot's OpenTelemetry starter and must be added explicitly.

  • Create a new file at src/main/resources/logback-spring.xml with the following content:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>

    <!-- Standard console output -->
    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>
                %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n
            </pattern>
        </encoder>
    </appender>

    <!-- OpenTelemetry appender sends logs to the OTel SDK -->
    <appender name="OpenTelemetry"
              class="io.opentelemetry.instrumentation.logback.appender.v1_0.OpenTelemetryAppender">
        <captureMdcAttributes>*</captureMdcAttributes>
    </appender>

    <root level="INFO">
        <appender-ref ref="CONSOLE"/>
        <appender-ref ref="OpenTelemetry"/>
    </root>

</configuration>

What does this file do?

This Logback configuration registers two appenders:

  • CONSOLE prints logs to your terminal as usual.
  • OpenTelemetry captures every log event and sends it to the OTel SDK, which exports it via OTLP.
  • captureMdcAttributes set to * ensures MDC values like traceId and spanId are included in exported logs.
  • Save logback-spring.xml.
  • Create a new Java class at src/main/java/com/example/chaosservice/OpenTelemetryAppenderInitializer.java with the following code:
package com.example.chaosservice;

import io.opentelemetry.api.OpenTelemetry;
import io.opentelemetry.instrumentation.logback.appender.v1_0.OpenTelemetryAppender;

import org.springframework.beans.factory.InitializingBean;
import org.springframework.stereotype.Component;

// Connects the Logback appender to the Spring-managed OpenTelemetry instance
@Component
class OpenTelemetryAppenderInitializer implements InitializingBean {

    private final OpenTelemetry openTelemetry;

    OpenTelemetryAppenderInitializer(OpenTelemetry openTelemetry) {
        this.openTelemetry = openTelemetry;
    }

    @Override
    public void afterPropertiesSet() {
        // Install the OTel SDK into the Logback appender so logs are exported
        OpenTelemetryAppender.install(this.openTelemetry);
    }
}

Why do you need this bean?

The Logback appender starts before the Spring context is ready, so it has no access to the OpenTelemetry SDK at boot time. This bean injects the auto-configured OpenTelemetry instance and calls OpenTelemetryAppender.install() once the context is fully initialized.

Until install is called, log events are queued internally and replayed once the SDK is connected.

  • Save OpenTelemetryAppenderInitializer.java.

Seeing a compile error on OpenTelemetryAppender?

Make sure you added the opentelemetry-logback-appender-1.0 dependency to your pom.xml and reloaded Maven (click the Maven refresh icon in IntelliJ or run ./mvnw dependency:resolve).

Help me fix this compile error

✔️ Awesome, I've got everything!

Great. Double check you've saved all three files before moving on.

ⓧ I'd like to double check the full code

Here is the full application.yml:

spring:
  application:
    name: chaos-service

management:
  endpoints:
    web:
      exposure:
        include: "*"
  tracing:
    sampling:
      probability: 1.0
  opentelemetry:
    tracing:
      export:
        otlp:
          endpoint: http://localhost:4318
    logging:
      export:
        otlp:
          endpoint: http://localhost:4318/v1/logs
  otlp:
    metrics:
      export:
        url: http://localhost:4318/v1/metrics

Here is the full logback-spring.xml:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>

    <!-- Standard console output -->
    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>
                %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n
            </pattern>
        </encoder>
    </appender>

    <!-- OpenTelemetry appender sends logs to the OTel SDK -->
    <appender name="OpenTelemetry"
              class="io.opentelemetry.instrumentation.logback.appender.v1_0.OpenTelemetryAppender">
        <captureMdcAttributes>*</captureMdcAttributes>
    </appender>

    <root level="INFO">
        <appender-ref ref="CONSOLE"/>
        <appender-ref ref="OpenTelemetry"/>
    </root>

</configuration>

Here is the full OpenTelemetryAppenderInitializer.java:

package com.example.chaosservice;

import io.opentelemetry.api.OpenTelemetry;
import io.opentelemetry.instrumentation.logback.appender.v1_0.OpenTelemetryAppender;

import org.springframework.beans.factory.InitializingBean;
import org.springframework.stereotype.Component;

// Connects the Logback appender to the Spring-managed OpenTelemetry instance
@Component
class OpenTelemetryAppenderInitializer implements InitializingBean {

    private final OpenTelemetry openTelemetry;

    OpenTelemetryAppenderInitializer(OpenTelemetry openTelemetry) {
        this.openTelemetry = openTelemetry;
    }

    @Override
    public void afterPropertiesSet() {
        // Install the OTel SDK into the Logback appender so logs are exported
        OpenTelemetryAppender.install(this.openTelemetry);
    }
}
Restart the chaos service and verify data in Grafana

With all three signals configured, restarting the app and hitting the chaos endpoints should produce traces, metrics, and logs visible in Grafana Explore.

  • Stop the running chaos service if it is still active (press Ctrl+C in the terminal).
  • Restart the chaos service by running:
./mvnw spring-boot:run
  • Once the app starts, hit each chaos endpoint a few times by running this script in a separate terminal:
# Hit each endpoint 3 times to generate telemetry
for i in 1 2 3; do
  curl -s http://localhost:8080/chaos/slow
  curl -s http://localhost:8080/chaos/error
  curl -s http://localhost:8080/chaos/memory
done

What should I see in the terminal?

The /chaos/slow calls will each take 2-5 seconds to return. The /chaos/error calls may return 500 error responses. The /chaos/memory calls return 200 after briefly allocating memory.

  • Open Grafana at http://localhost:3000 and log in with admin / admin.
  • Click the hamburger menu, then click Explore.
  • Select the Tempo datasource from the dropdown at the top.
  • Change the query type to Search and click Run query.

You should see trace results for your chaos service HTTP requests, each with spans showing the request lifecycle.

  • Switch the datasource to Prometheus.
  • In the query editor, type http_server_requests and look for metric names like http_server_requests_seconds_count appearing in the suggestions.
  • Select http_server_requests_seconds_count and click Run query.

You should see metric data points showing the count of HTTP requests to your chaos endpoints.

  • Switch the datasource to Loki.
  • In the query editor, enter this LogQL query and click Run query:
{service_name="chaos-service"}

You should see log lines from the chaos service appearing in the results, with traceId visible in the log metadata.

Not seeing any data in Grafana?

  • Make sure your Docker Compose stack is still running. Run docker compose ps and check all 5 containers are healthy.
  • Check that the chaos service started without errors. Look for OTLP export failures in the Spring Boot console output.
  • Wait 15-30 seconds after hitting the endpoints. Prometheus scrapes every 15 seconds, so metrics may take a moment to appear.

Help me debug why telemetry isn't showing up in Grafana

How does trace-log correlation work?

Micrometer Tracing automatically injects traceId and spanId into Logback's MDC for every traced request. Because the OpenTelemetry Logback appender captures MDC attributes, these IDs travel with every exported log record.

This means you can take any trace ID from Tempo and search Loki for all logs belonging to that specific request.

All three observability signals are now flowing through your pipeline. Next up, you will use PromQL and LogQL queries to troubleshoot the chaos scenarios like a real incident responder.

Troubleshoot with PromQL and LogQL

Your chaos service is now sending traces, metrics, and logs to the observability stack. All three signals are flowing through the OTel Collector to their respective backends.

Now comes the real skill: asking the right questions of your telemetry. In this step, you'll trigger a burst of failures and then use PromQL and LogQL inside Grafana Explore to diagnose exactly what went wrong.

In this step, get ready to:
  • Trigger a burst of chaos traffic by calling each endpoint 10 times.
  • Write PromQL queries to identify latency spikes, error rates, and memory pressure.
  • Write LogQL queries to find error logs and correlate them with traces.
Trigger chaos and diagnose with PromQL

Before you can troubleshoot, you need data to troubleshoot. You'll generate a burst of requests to all three chaos endpoints so that Prometheus, Loki, and Tempo have fresh telemetry to query.

  • Generate chaos traffic by running this script in a new terminal window:

macOS

for i in $(seq 1 10); do curl -s http://localhost:8080/chaos/slow > /dev/null; done
for i in $(seq 1 10); do curl -s http://localhost:8080/chaos/error > /dev/null; done
for i in $(seq 1 10); do curl -s http://localhost:8080/chaos/memory > /dev/null; done

Windows

1..10 | ForEach-Object { Invoke-WebRequest http://localhost:8080/chaos/slow -SkipHttpErrorCheck }
1..10 | ForEach-Object { Invoke-WebRequest http://localhost:8080/chaos/error -SkipHttpErrorCheck }
1..10 | ForEach-Object { Invoke-WebRequest http://localhost:8080/chaos/memory -SkipHttpErrorCheck }

What does this script do?

Each loop calls one chaos endpoint 10 times in sequence. The /chaos/slow calls will take 20-50 seconds total because each request sleeps for 2-5 seconds.

The /chaos/error calls return 500 status codes, which is why the script suppresses error output. The /chaos/memory calls allocate and release memory, creating visible spikes.

The slow endpoint calls take 20-50 seconds to complete. Wait for the script to finish before moving on.

Script hanging or timing out?

Make sure your chaos service is still running in the other terminal with ./mvnw spring-boot:run. If you restarted your machine, you also need to start the Docker Compose stack with docker compose up -d first.

help me troubleshoot why my chaos service isn't responding to curl requests

Now you'll query Prometheus to see what the metrics reveal about those requests.

  • In Grafana, click the Explore icon in the left-hand navigation menu.
  • Select Prometheus from the datasource dropdown in the upper-left corner.
  • Paste this query into the query editor to find the p95 latency of the slow endpoint:
histogram_quantile(0.95, rate(http_server_requests_seconds_bucket{uri="/chaos/slow"}[1m]))
  • Click Run query (or press Shift+Enter).

You should see a value above 2 seconds. This confirms the slow endpoint is taking 2-5 seconds per request, as designed.

  • Clear the query and paste this one to find the error rate:
sum(rate(http_server_requests_seconds_count{status="500"}[1m]))
  • Click Run query.

You should see a value greater than 0, representing the rate of 500-status responses per second from the error endpoint.

  • Clear the query and paste this one to check heap memory usage:
jvm_memory_used_bytes{area="heap"}
  • Click Run query.

You should see a time series showing the JVM heap usage, with visible spikes from the memory pressure endpoint.

What do these PromQL queries reveal?

The first query uses histogram_quantile to calculate the 95th percentile latency. In production, p95 latency tells you how slow requests are for the worst 5% of users.

The second query uses rate() to calculate how many 500 errors are happening per second. A non-zero value means something is actively failing.

The third query shows raw heap memory bytes. Sudden spikes followed by drops indicate the garbage collector reclaiming memory after the /chaos/memory endpoint releases its allocations.

Queries returning no data?

Prometheus scrapes the OTel Collector every 15 seconds. Wait at least 30 seconds after running your chaos script before querying.

If you still see no results, check that the Prometheus container is healthy by running docker compose ps and verifying it shows as running.

help me debug why Prometheus shows no data for my Spring Boot metrics

Find errors with LogQL

Metrics tell you something is wrong, but logs tell you why. Now you'll switch to the Loki datasource and use LogQL to dig into the actual error messages.

  • In Grafana Explore, change the datasource dropdown from Prometheus to Loki.
  • Paste this query to find all exception logs:
{service_name="chaos-service"} |= "Exception"
  • Click Run query.

You should see log lines containing exception stack traces from the /chaos/error endpoint.

  • Clear the query and paste this one to extract just the error messages:
{service_name="chaos-service"} | json | level="ERROR" | line_format "{{.message}}"
  • Click Run query.

You should now see cleaner output with just the error messages, filtered to only ERROR-level logs.

  • Clear the query and paste this one to calculate the error log rate:
sum(rate({service_name="chaos-service"} |= "Exception" [1m]))
  • Click Run query.

This returns a numeric rate similar to the PromQL error rate query, but calculated from log volume instead of metrics.

What do these LogQL queries reveal?

The first query uses |= as a line filter, keeping only log lines that contain the word "Exception". This is the fastest way to find errors in a large stream of logs.

The second query parses each log line as JSON, filters to ERROR level only, and reformats the output to show just the message field. This is how you cut through noisy stack traces to find the root cause.

The third query wraps a log stream in sum(rate(...)) to produce a metric from logs. This is useful when you want to alert on error log volume without pre-defining a metric.

Loki returning empty results?

Make sure the label selector uses service_name (with an underscore, not a dot). This is the OpenTelemetry resource attribute name as it appears in Loki.

If you see timeout errors, try narrowing the time range in the top-right corner of Grafana Explore to the last 15 minutes.

help me debug why LogQL queries in Grafana return no results for my chaos-service logs

Correlate traces with logs using traceId

The real power of a unified observability stack is correlation. Micrometer Tracing automatically injects traceId and spanId into Logback's MDC (Mapped Diagnostic Context) for every request. The OpenTelemetry Logback appender includes these as resource attributes, which means every log line in Loki carries the trace it belongs to.

  • In Grafana Explore, switch the datasource to Tempo.
  • Select the Search tab in the query editor.
  • Set Service Name to chaos-service and click Run query.
  • Click on any trace from the results list to expand it.
  • Copy the Trace ID value shown at the top of the trace detail panel.

Now you'll use that trace ID to find every log line that belongs to that specific request.

  • Switch the datasource back to Loki.
  • Paste this query, replacing the placeholder with your copied trace ID:
{service_name="chaos-service"} |= "[[TRACE_ID="your trace ID from Tempo"]]"
  • Click Run query.

You should see only the log lines that belong to that single HTTP request. If the trace came from the error endpoint, you'll see the exception stack trace for that exact request.

Why is trace-log correlation so powerful?

In production, thousands of requests happen simultaneously. When a user reports an error, you need to find the logs for that specific request out of millions.

The traceId acts as a unique identifier that links a trace span in Tempo to its log lines in Loki. You can start from a slow trace, grab its ID, and immediately see what happened inside the application at that moment.

No logs found for your trace ID?

Make sure you copied the full trace ID with no extra spaces. Trace IDs are 32-character hexadecimal strings.

If Loki returns nothing, the trace may have been generated before the Logback appender was initialized. Try picking a more recent trace from Tempo.

help me understand why my Loki query with a traceId returns no results even though the trace exists in Tempo

You've just completed the core observability workflow: trigger a problem, identify it in metrics, drill into logs, and correlate everything with traces. These are the same techniques you'd use to diagnose a production incident.

Secret mission

Build a JVM Monitoring Dashboard

You have the queries. You have the data. Now build a real-time Grafana dashboard that shows heap memory, request rates, error percentages, and GC pauses at a glance. Add a variable so you can filter every panel by endpoint.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally using Docker containers, so there are no ongoing cloud costs.

Resources you used:

  • Docker containers: otel-collector, prometheus, loki, tempo, grafana.
  • Docker volumes storing data for Prometheus, Loki, Tempo, and Grafana.
  • Local project files: the chaos-service Spring Boot project directory and all configuration files.

Keep everything running

No action needed. Choose this if you're still actively experimenting with the chaos endpoints and observability stack.

  • Leave Docker Compose running to continue triggering chaos endpoints and exploring Grafana.
  • Disk usage will grow slowly as telemetry data accumulates in the Docker volumes for Prometheus, Loki, and Tempo.
  • The Spring Boot app continues running in its terminal. You can stop and restart it any time with ./mvnw spring-boot:run.

Pause - I'll come back to this later

Shut down running processes to free up CPU and memory while keeping your data and project files intact.

  • Stop the Spring Boot app by pressing Ctrl+C in the terminal where it is running.
  • Stop the Docker containers by running:
docker compose stop

This preserves all your telemetry data in the Docker volumes. When you're ready to come back, resume with:

docker compose start
  • Then restart the chaos service with ./mvnw spring-boot:run from the project directory.

Delete - I don't want to use this again

Remove all project resources completely and start fresh. There are no cost implications since everything runs locally.

  • Stop the Spring Boot app by pressing Ctrl+C in the terminal where it is running.
  • Remove all Docker containers and their data volumes by running:
docker compose down -v
  • This removes all 5 containers (otel-collector, prometheus, loki, tempo, grafana) and deletes the associated data volumes.
  • Delete the project directory. On macOS or Linux:
cd ~/Desktop
rm -rf chaos-service
  • On Windows:
cd ~\Desktop
rmdir /s /q chaos-service
  • After deletion, all telemetry data, configuration files, and source code are gone. You can re-create the project from scratch at any time by following the steps again.

That's a Wrap!

That's a Wrap!

Nice work! You just built a full observability pipeline from scratch, starting with a chaos service that deliberately breaks things and ending with a Grafana environment where you can find and diagnose every failure.

You've learned how to:

  • Build a Spring Boot 4 chaos service with endpoints that simulate slow responses, random exceptions, and memory pressure, exporting all three observability signals via OTLP.
  • Deploy 5 individual observability services (OTel Collector, Prometheus, Loki, Tempo, Grafana) with Docker Compose and wire them together through explicit configuration.
  • Write PromQL and LogQL queries to identify latency spikes, error rates, and correlate traces with logs using the traceId field.
  • Secret Mission: Built a JVM monitoring dashboard in Grafana with panels for heap memory, HTTP request rate, error percentage, and GC pause duration.

Ready to quiz yourself?