Full Observability Stack for Java Services

Monitor two Spring Boot microservices with metrics, logs, and traces.

Introduction

30 Second Summary

When a service is slow or broken in a system with multiple communicating services, developers need to answer three questions: what happened, where did it happen, and why. Without the right tools, finding those answers is like searching for a needle in a haystack.

In this project, you will build a complete local observability platform that captures metrics, logs, and distributed traces from two Spring Boot microservices. You will then use Grafana to correlate all three signals and debug a simulated slow request.

What You'll Build

A local observability stack monitoring two Spring Boot microservices that communicate over HTTP, with the entire request lifecycle captured as correlated metrics, structured logs, and distributed traces.

By the end of this project, you'll have:

  • A Docker Compose stack running Grafana, Prometheus, Grafana Loki, Grafana Tempo, and the OpenTelemetry Collector with pre-provisioned datasources.
  • Two Spring Boot microservices auto-instrumented with the OpenTelemetry Java Agent, emitting metrics, logs, and traces with zero code changes.
  • A demonstrated debugging workflow that jumps from a latency metric, to the offending trace, to the exact log line across service boundaries.
  • Secret Mission: Add manual instrumentation with custom spans and business metrics to the Order Service.

Are there any prerequisites?

You should have Docker Desktop, Java 21, and Gradle installed on your machine. Basic familiarity with Spring Boot and Docker is helpful but not required.

Set Up the Observability Backend

Before you can monitor any application, you need an observability backend that can receive, store, and visualize telemetry data. This step builds the entire infrastructure layer: a central telemetry collector, three specialized storage backends (one each for metrics, logs, and traces), and a visualization dashboard that ties them all together.

You will define everything as code in configuration files, then bring the entire stack to life with a single Docker Compose command. By the end of this step, you will have five containers running and ready to accept telemetry from any application.

In this step, get ready to:
  • Create all configuration files for the observability stack.
  • Start the Docker Compose stack and verify all five services are running.
  • Understand how telemetry data flows from applications through the collector to each backend.
Create the directory structure and configuration files

The observability stack needs several configuration files, each telling a different service how to behave. You will create all of them inside a config/ directory within your project folder.

  • Create the directory structure by running these commands:
mkdir -p config/grafana/provisioning/datasources

What does this command do?

The -p flag creates all parent directories in one go. This single command creates the config/ folder and the nested grafana/provisioning/datasources/ path inside it.

  • Confirm the directories exist by running:
ls config/

You should see grafana listed in the output alongside the config files you are about to create.

  • Create docker-compose.yml in your project root with the following content:
services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    container_name: otel-collector
    volumes:
      - ./config/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
    ports:
      - "4317:4317"
      - "4318:4318"
      - "9464:9464"
    depends_on:
      - prometheus
      - loki
      - tempo

  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    volumes:
      - ./config/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus
    ports:
      - "9090:9090"
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.retention.time=1d"

  loki:
    image: grafana/loki:latest
    container_name: loki
    volumes:
      - ./config/loki-config.yaml:/etc/loki/local-config.yaml:ro
      - loki-data:/loki
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml

  tempo:
    image: grafana/tempo:latest
    container_name: tempo
    volumes:
      - ./config/tempo-config.yaml:/etc/tempo/tempo.yaml:ro
      - tempo-data:/tmp/tempo
    ports:
      - "3200:3200"
    command: ["-config.file=/etc/tempo/tempo.yaml"]

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    volumes:
      - ./config/grafana/provisioning:/etc/grafana/provisioning:ro
    ports:
      - "3000:3000"
    depends_on:
      - prometheus
      - loki
      - tempo

volumes:
  prometheus-data:
  loki-data:
  tempo-data:

What does this file do?

This file defines five services that work together as your observability backend:

  • otel-collector receives all telemetry (metrics, logs, traces) from your applications on ports 4317 (gRPC) and 4318 (HTTP), then routes each signal to the correct backend.
  • prometheus stores metrics and exposes them for querying on port 9090.
  • loki stores logs on port 3100.
  • tempo stores distributed traces on port 3200.
  • grafana provides a single UI on port 3000 to query all three backends.

The depends_on fields ensure the storage backends start before the services that need them.

  • Save docker-compose.yml.

✔️ Awesome, I've got everything!

Great. Make sure you have saved docker-compose.yml in your project root directory.

ⓧ I'd like to double check the full code

services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    container_name: otel-collector
    volumes:
      - ./config/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
    ports:
      - "4317:4317"
      - "4318:4318"
      - "9464:9464"
    depends_on:
      - prometheus
      - loki
      - tempo

  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    volumes:
      - ./config/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus
    ports:
      - "9090:9090"
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.retention.time=1d"

  loki:
    image: grafana/loki:latest
    container_name: loki
    volumes:
      - ./config/loki-config.yaml:/etc/loki/local-config.yaml:ro
      - loki-data:/loki
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml

  tempo:
    image: grafana/tempo:latest
    container_name: tempo
    volumes:
      - ./config/tempo-config.yaml:/etc/tempo/tempo.yaml:ro
      - tempo-data:/tmp/tempo
    ports:
      - "3200:3200"
    command: ["-config.file=/etc/tempo/tempo.yaml"]

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    volumes:
      - ./config/grafana/provisioning:/etc/grafana/provisioning:ro
    ports:
      - "3000:3000"
    depends_on:
      - prometheus
      - loki
      - tempo

volumes:
  prometheus-data:
  loki-data:
  tempo-data:
  • Create config/otel-collector-config.yaml with the following content:
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024

exporters:
  prometheus:
    endpoint: "0.0.0.0:9464"
    resource_to_telemetry_conversion:
      enabled: true

  otlphttp/loki:
    endpoint: "http://loki:3100/otlp"
    tls:
      insecure: true

  otlp/tempo:
    endpoint: "tempo:4317"
    tls:
      insecure: true

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/loki]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo]

What does this file do?

This is the heart of your telemetry routing. The OpenTelemetry Collector config defines three pipelines, one for each signal type:

  • Metrics received via OTLP are exported to Prometheus on port 9464. The resource_to_telemetry_conversion setting converts resource attributes (like service name) into Prometheus labels.
  • Logs are forwarded to Loki's native OTLP endpoint at http://loki:3100/otlp.
  • Traces go to Tempo via OTLP gRPC on port 4317.

The batch processor groups telemetry before forwarding, reducing network overhead.

  • Save config/otel-collector-config.yaml.

✔️ Awesome, I've got everything!

Great. Make sure you have saved config/otel-collector-config.yaml in the correct location.

ⓧ I'd like to double check the full code

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024

exporters:
  prometheus:
    endpoint: "0.0.0.0:9464"
    resource_to_telemetry_conversion:
      enabled: true

  otlphttp/loki:
    endpoint: "http://loki:3100/otlp"
    tls:
      insecure: true

  otlp/tempo:
    endpoint: "tempo:4317"
    tls:
      insecure: true

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/loki]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo]
  • Create config/prometheus.yml with the following content:
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "otel-collector"
    static_configs:
      - targets: ["otel-collector:9464"]

What does this file do?

Prometheus pulls (scrapes) metrics from targets at regular intervals. This config tells it to scrape the OTel Collector's Prometheus exporter endpoint (otel-collector:9464) every 15 seconds. That is where all your application metrics end up after the Collector processes them.

  • Save config/prometheus.yml.
  • Create config/loki-config.yaml with the following content:
auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

limits_config:
  allow_structured_metadata: true

schema_config:
  configs:
    - from: "2024-01-01"
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

What does this file do?

This configures Loki to run in single-node mode with filesystem storage. The key setting is allow_structured_metadata: true, which enables OTLP log ingestion to properly store trace IDs and span IDs as queryable metadata. This is what allows you to correlate logs with traces later.

  • Save config/loki-config.yaml.
  • Create config/tempo-config.yaml with the following content:
server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: "0.0.0.0:4317"

storage:
  trace:
    backend: local
    local:
      path: /tmp/tempo/blocks

metrics_generator:
  storage:
    path: /tmp/tempo/metrics
  traces_storage:
    path: /tmp/tempo/traces

What does this file do?

Tempo stores distributed traces and exposes an HTTP API on port 3200 for Grafana to query. It receives traces from the OTel Collector via its own OTLP gRPC receiver on port 4317 (internal to the Docker network).

  • Save config/tempo-config.yaml.
  • Create config/grafana/provisioning/datasources/datasources.yml with the following content:
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: true

  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    editable: true

  - name: Tempo
    type: tempo
    access: proxy
    url: http://tempo:3200
    editable: true

What does this file do?

Grafana provisioning files auto-configure datasources on startup. This means you do not need to manually add Prometheus, Loki, or Tempo in the Grafana UI. They will already be connected and ready to query.

  • Save config/grafana/provisioning/datasources/datasources.yml.
  • Verify all your configuration files are in place by running:
ls config/

You should see grafana, loki-config.yaml, otel-collector-config.yaml, prometheus.yml, and tempo-config.yaml listed.

Missing a file?

Double check that you created each file in the correct directory. All config files (except docker-compose.yml) live inside the config/ folder. The datasources file is nested deeper at config/grafana/provisioning/datasources/datasources.yml.

help me figure out why my config files are not in the right place

Start the Docker Compose stack

With all configuration files in place, you can now bring up the entire observability backend with a single command.

  • Start all five services in detached mode by running:
docker compose up -d

What does this command do?

The -d flag runs all containers in the background (detached mode). Docker Compose reads your docker-compose.yml, pulls the container images if needed, creates the network, and starts services in dependency order.

The first run will take a few minutes as Docker downloads all five images. Subsequent starts are nearly instant.

  • Verify all containers are running by executing:
docker compose ps

You should see five containers listed (otel-collector, prometheus, loki, tempo, grafana), all with a STATUS of running.

  • Open your browser and navigate to http://localhost:3000 to load Grafana.
  • If prompted to log in, enter admin for both the username and password.
  • Click Connections in the left-side menu.
  • Click Data sources.

You should see three datasources listed: Prometheus, Loki, and Tempo. These were automatically provisioned from your datasources.yml file.

  • Open a second browser tab and navigate to http://localhost:9090 to confirm Prometheus is accessible.

You should see the Prometheus query interface. This confirms the metrics backend is ready.

Containers not starting?

If docker compose ps shows containers with a status other than running, check the container logs for errors:

docker compose logs --tail 20

Common causes: a YAML indentation error in one of your config files, Docker Desktop not running, or a port conflict (another service already using port 3000, 9090, 3100, 3200, 4317, or 4318).

help me debug why my Docker Compose stack is not starting

Understand the data flow architecture

Now that the stack is running, take a moment to understand how telemetry flows through the system you just built.

How does telemetry flow through the stack?

Applications send all telemetry (metrics, logs, and traces) via the OpenTelemetry Protocol (OTLP) to the Collector on port 4318 (HTTP). The Collector is the single entry point for all signals.

The Collector acts as a router. It receives mixed telemetry, batches it for efficiency, then forwards each signal type to its dedicated backend: metrics to Prometheus, logs to Loki, traces to Tempo.

Grafana queries each backend independently but can correlate between them using trace IDs. A trace ID appears in all three backends, letting you jump from a metric spike to the offending trace to the exact log line.

Your observability backend is live. Next up, you will build the two Spring Boot microservices that generate the telemetry this stack will capture.

Build the Spring Boot Microservices

Your observability backend is running and ready to receive telemetry. Now you need applications that actually generate telemetry to observe.

In this step, you will create two Spring Boot microservices that talk to each other over HTTP. The Order Service accepts new orders and calls the Product Service to look up product details. This inter-service communication is exactly the pattern that distributed tracing is designed to capture.

In this step, get ready to:
  • Create the Product Service with a GET endpoint that serves product data.
  • Create the Order Service with a POST endpoint that calls the Product Service.
  • Build and verify inter-service communication works.
Create the Product Service

The Product Service is a simple REST API that returns product information. One product (ID 3) has an intentional 2-second delay built in to simulate a slow downstream dependency. This delay will be clearly visible in traces later.

  • Create the Product Service directory structure by running these commands:
mkdir -p product-service/src/main/java/com/example/productservice
mkdir -p product-service/src/main/resources

What do these directories do?

This is the standard Java project layout that Gradle and Spring Boot expect. Source code goes under src/main/java and configuration files go under src/main/resources.

  • Create the build.gradle file for the Product Service by pasting this into product-service/build.gradle:
plugins {
    id 'java'
    id 'org.springframework.boot' version '3.5.14'
    id 'io.spring.dependency-management' version '1.1.7'
}

group = 'com.example'
version = '0.0.1-SNAPSHOT'

java {
    toolchain {
        languageVersion = JavaLanguageVersion.of(21)
    }
}

repositories {
    mavenCentral()
}

dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-web'
}

What does this build file do?

The org.springframework.boot plugin adds the bootJar task that packages your app as a runnable JAR. The spring-boot-starter-web dependency pulls in an embedded Tomcat server and the Spring MVC framework.

The Java toolchain is pinned to version 21 so Gradle uses the correct JDK regardless of what else is installed on your machine.

  • Create the settings.gradle file by pasting this into product-service/settings.gradle:
rootProject.name = 'product-service'
  • Create the application configuration by pasting this into product-service/src/main/resources/application.yml:
server:
  port: 8081

spring:
  application:
    name: product-service

Why set a service name here?

The spring.application.name property identifies this service in logs and, later, in your observability tools. When the OpenTelemetry agent attaches, it uses this name as the default service identifier in all telemetry.

  • Create the application entry point by pasting this into product-service/src/main/java/com/example/productservice/ProductServiceApplication.java:
package com.example.productservice;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class ProductServiceApplication {

    public static void main(String[] args) {
        SpringApplication.run(ProductServiceApplication.class, args);
    }
}
  • Save ProductServiceApplication.java.

Now for the interesting part. The Product Controller defines three products and adds a deliberate 2-second sleep when product ID 3 is requested.

  • Create the REST controller by pasting this into product-service/src/main/java/com/example/productservice/ProductController.java:
package com.example.productservice;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RestController;

import java.util.Map;

@RestController
public class ProductController {

    private static final Logger logger = LoggerFactory.getLogger(ProductController.class);

    private static final Map<Integer, Map<String, Object>> PRODUCTS = Map.of(
        1, Map.of("id", 1, "name", "Mechanical Keyboard", "price", 89.99),
        2, Map.of("id", 2, "name", "USB-C Monitor", "price", 349.99),
        3, Map.of("id", 3, "name", "Standing Desk", "price", 599.99)
    );

    @GetMapping("/products/{id}")
    public Map<String, Object> getProduct(@PathVariable int id) {
        logger.info("Fetching product with id: {}", id);

        if (id == 3) {
            logger.warn("Product {} requires slow database lookup", id);
            try {
                Thread.sleep(2000);
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
            }
        }

        Map<String, Object> product = PRODUCTS.get(id);
        if (product == null) {
            logger.error("Product not found: {}", id);
            throw new RuntimeException("Product not found: " + id);
        }

        logger.info("Returning product: {}", product.get("name"));
        return product;
    }
}

What does this controller do?

The PRODUCTS map is a hardcoded catalog of three items. The @GetMapping("/products/{id}") annotation exposes the endpoint that the Order Service will call.

The Thread.sleep(2000) on product ID 3 simulates a slow database query. When you explore traces later, this 2-second delay will stand out clearly in the waterfall view, making it easy to pinpoint the bottleneck.

  • Save ProductController.java.
  • Confirm all Product Service files exist by running:
ls product-service/src/main/java/com/example/productservice/

You should see ProductController.java and ProductServiceApplication.java listed.

Don't see both files?

Double-check that you created the files in the correct nested path. The package structure com/example/productservice must match exactly. If you mistyped a directory name, remove and recreate it.

help me fix my Product Service directory structure

✔️ Awesome, I've got everything!

Great. Make sure you have saved all five Product Service files before moving on.

ⓧ I'd like to double check the full code

Here are all five Product Service files for comparison:

plugins {
    id 'java'
    id 'org.springframework.boot' version '3.5.14'
    id 'io.spring.dependency-management' version '1.1.7'
}

group = 'com.example'
version = '0.0.1-SNAPSHOT'

java {
    toolchain {
        languageVersion = JavaLanguageVersion.of(21)
    }
}

repositories {
    mavenCentral()
}

dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-web'
}
rootProject.name = 'product-service'
server:
  port: 8081

spring:
  application:
    name: product-service
package com.example.productservice;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class ProductServiceApplication {

    public static void main(String[] args) {
        SpringApplication.run(ProductServiceApplication.class, args);
    }
}
package com.example.productservice;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RestController;

import java.util.Map;

@RestController
public class ProductController {

    private static final Logger logger = LoggerFactory.getLogger(ProductController.class);

    private static final Map<Integer, Map<String, Object>> PRODUCTS = Map.of(
        1, Map.of("id", 1, "name", "Mechanical Keyboard", "price", 89.99),
        2, Map.of("id", 2, "name", "USB-C Monitor", "price", 349.99),
        3, Map.of("id", 3, "name", "Standing Desk", "price", 599.99)
    );

    @GetMapping("/products/{id}")
    public Map<String, Object> getProduct(@PathVariable int id) {
        logger.info("Fetching product with id: {}", id);

        if (id == 3) {
            logger.warn("Product {} requires slow database lookup", id);
            try {
                Thread.sleep(2000);
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
            }
        }

        Map<String, Object> product = PRODUCTS.get(id);
        if (product == null) {
            logger.error("Product not found: {}", id);
            throw new RuntimeException("Product not found: " + id);
        }

        logger.info("Returning product: {}", product.get("name"));
        return product;
    }
}
Create the Order Service

The Order Service accepts order requests and calls the Product Service to fetch product details. It uses RestTemplate for HTTP calls, which is important because the OpenTelemetry agent auto-instruments RestTemplate to propagate trace context across service boundaries.

  • Create the Order Service directory structure by running these commands:
mkdir -p order-service/src/main/java/com/example/orderservice
mkdir -p order-service/src/main/resources
  • Create the Gradle build file by pasting this into order-service/build.gradle:
plugins {
    id 'java'
    id 'org.springframework.boot' version '3.5.14'
    id 'io.spring.dependency-management' version '1.1.7'
}

group = 'com.example'
version = '0.0.1-SNAPSHOT'

java {
    toolchain {
        languageVersion = JavaLanguageVersion.of(21)
    }
}

repositories {
    mavenCentral()
}

dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-web'
}
  • Create the settings file by pasting this into order-service/settings.gradle:
rootProject.name = 'order-service'
  • Create the application configuration by pasting this into order-service/src/main/resources/application.yml:
server:
  port: 8080

spring:
  application:
    name: order-service

product-service:
  url: http://localhost:8081

What is the product-service.url property?

This custom configuration property tells the Order Service where to find the Product Service. By externalizing the URL into config rather than hardcoding it, you can easily change the target when running in different environments (like Docker networking later).

  • Create the application entry point by pasting this into order-service/src/main/java/com/example/orderservice/OrderServiceApplication.java:
package com.example.orderservice;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.web.client.RestTemplate;

@SpringBootApplication
public class OrderServiceApplication {

    public static void main(String[] args) {
        SpringApplication.run(OrderServiceApplication.class, args);
    }

    @Bean
    public RestTemplate restTemplate() {
        return new RestTemplate();
    }
}

Why define a RestTemplate bean?

Declaring RestTemplate as a Spring @Bean makes it injectable across the application. More importantly, the OpenTelemetry agent hooks into Spring-managed RestTemplate instances to automatically add W3C Trace Context headers to outgoing HTTP calls. This is how trace context propagates between services with zero code changes.

Now create the Order Controller. This is where the inter-service call happens.

  • Create the REST controller by pasting this into order-service/src/main/java/com/example/orderservice/OrderController.java:
package com.example.orderservice;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.client.RestTemplate;

import java.time.Instant;
import java.util.Map;
import java.util.UUID;

@RestController
public class OrderController {

    private static final Logger logger = LoggerFactory.getLogger(OrderController.class);

    private final RestTemplate restTemplate;
    private final String productServiceUrl;

    public OrderController(RestTemplate restTemplate,
                           @Value("${product-service.url}") String productServiceUrl) {
        this.restTemplate = restTemplate;
        this.productServiceUrl = productServiceUrl;
    }

    @PostMapping("/orders")
    public Map<String, Object> createOrder(@RequestBody Map<String, Object> request) {
        int productId = (int) request.get("productId");
        logger.info("Creating order for product: {}", productId);

        logger.info("Calling product service for product: {}", productId);
        Map<String, Object> product = restTemplate.getForObject(
            productServiceUrl + "/products/" + productId,
            Map.class
        );

        String orderId = UUID.randomUUID().toString();
        logger.info("Order created successfully: {}", orderId);

        return Map.of(
            "orderId", orderId,
            "productId", productId,
            "productName", product.get("name"),
            "price", product.get("price"),
            "status", "CREATED",
            "timestamp", Instant.now().toString()
        );
    }
}

What does this controller do?

The createOrder method extracts the productId from the request body, then calls the Product Service using restTemplate.getForObject(). This HTTP call is the inter-service communication that creates a distributed trace spanning both services.

The @Value("${product-service.url}") annotation injects the URL you defined in application.yml. The response includes the product name and price fetched from the downstream service.

  • Save OrderController.java.
  • Confirm all Order Service files exist by running:
ls order-service/src/main/java/com/example/orderservice/

You should see OrderController.java and OrderServiceApplication.java listed.

Missing files or wrong directory?

Make sure the package path is com/example/orderservice (not productservice). Each service has its own distinct package name.

help me fix my Order Service directory structure

✔️ Awesome, I've got everything!

Great. Make sure you have saved all five Order Service files before moving on.

ⓧ I'd like to double check the full code

Here are all five Order Service files for comparison:

plugins {
    id 'java'
    id 'org.springframework.boot' version '3.5.14'
    id 'io.spring.dependency-management' version '1.1.7'
}

group = 'com.example'
version = '0.0.1-SNAPSHOT'

java {
    toolchain {
        languageVersion = JavaLanguageVersion.of(21)
    }
}

repositories {
    mavenCentral()
}

dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-web'
}
rootProject.name = 'order-service'
server:
  port: 8080

spring:
  application:
    name: order-service

product-service:
  url: http://localhost:8081
package com.example.orderservice;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.web.client.RestTemplate;

@SpringBootApplication
public class OrderServiceApplication {

    public static void main(String[] args) {
        SpringApplication.run(OrderServiceApplication.class, args);
    }

    @Bean
    public RestTemplate restTemplate() {
        return new RestTemplate();
    }
}
package com.example.orderservice;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.client.RestTemplate;

import java.time.Instant;
import java.util.Map;
import java.util.UUID;

@RestController
public class OrderController {

    private static final Logger logger = LoggerFactory.getLogger(OrderController.class);

    private final RestTemplate restTemplate;
    private final String productServiceUrl;

    public OrderController(RestTemplate restTemplate,
                           @Value("${product-service.url}") String productServiceUrl) {
        this.restTemplate = restTemplate;
        this.productServiceUrl = productServiceUrl;
    }

    @PostMapping("/orders")
    public Map<String, Object> createOrder(@RequestBody Map<String, Object> request) {
        int productId = (int) request.get("productId");
        logger.info("Creating order for product: {}", productId);

        logger.info("Calling product service for product: {}", productId);
        Map<String, Object> product = restTemplate.getForObject(
            productServiceUrl + "/products/" + productId,
            Map.class
        );

        String orderId = UUID.randomUUID().toString();
        logger.info("Order created successfully: {}", orderId);

        return Map.of(
            "orderId", orderId,
            "productId", productId,
            "productName", product.get("name"),
            "price", product.get("price"),
            "status", "CREATED",
            "timestamp", Instant.now().toString()
        );
    }
}
Build and verify inter-service communication

Both services are written. Now you will compile them into runnable JARs, start them up, and confirm the Order Service can successfully call the Product Service.

  • Build the Product Service by running:
cd product-service && gradle bootJar
  • Build the Order Service by running:
cd ../order-service && gradle bootJar
  • Return to the project root by running:
cd ..

Build failing with compilation errors?

Make sure your Java class files are in the correct package directory. The file ProductController.java must be at product-service/src/main/java/com/example/productservice/ProductController.java. If it is at the wrong path, Gradle cannot find it.

Also verify that your build.gradle has no typos in the plugin versions. The Spring Boot plugin version must be 3.5.14.

help me fix my Gradle build errors

Now start both services to test inter-service communication.

  • Start the Product Service in the background by running:
java -jar product-service/build/libs/product-service-0.0.1-SNAPSHOT.jar &
  • Wait a few seconds for it to start, then start the Order Service by running:
java -jar order-service/build/libs/order-service-0.0.1-SNAPSHOT.jar &

You should see Spring Boot startup logs for each service. The Product Service will log that it started on port 8081, and the Order Service on port 8080.

  • Test the inter-service call by sending a request to the Order Service:
curl -X POST http://localhost:8080/orders -H "Content-Type: application/json" -d '{"productId": 1}'

You should see a JSON response containing an orderId, productName of Mechanical Keyboard, a price of 89.99, and a status of CREATED.

What just happened?

Your curl command hit the Order Service on port 8080. The Order Service then made an internal HTTP call to the Product Service on port 8081 to look up product ID 1. The Product Service returned the product details, and the Order Service assembled them into an order response.

This is a distributed request. Right now there is no observability. In the next step, you will attach the OpenTelemetry agent and this same call will generate correlated metrics, logs, and traces automatically.

  • Stop both running services by running:
kill %1 %2

Why stop the services now?

In the next step you will restart both services with the OpenTelemetry Java Agent attached. You need to stop the current processes first so the ports are free.

Connection refused when calling /orders?

Make sure both services finished starting before you send the curl request. Look for the "Started" log line from each service. If only one service started, the other may have a port conflict or a typo in application.yml.

help me debug my microservice communication

Both microservices are running and communicating. Next up, you will attach the OpenTelemetry Java Agent to capture every detail of these inter-service calls as metrics, logs, and distributed traces.

Instrument with the OpenTelemetry Java Agent

Your microservices work, but right now you have no visibility into what happens once a request enters the system. If the Order Service slows down, you have no way to tell whether the problem is in your code, the network call, or the downstream Product Service.

The OpenTelemetry Java Agent solves this with zero code changes. You attach a single JAR file to each service at startup, and it automatically generates metrics, logs, and distributed traces by intercepting framework calls at the bytecode level.

In this step, get ready to:
  • Download the OpenTelemetry Java Agent JAR.
  • Start both services with the agent attached via environment variables.
  • Generate traffic and verify telemetry appears in Grafana.
Download the OpenTelemetry Java Agent

The OpenTelemetry Java Agent is a single JAR file that uses Java bytecode instrumentation (via the -javaagent JVM flag) to intercept framework calls at runtime. It creates spans for every HTTP request, propagates W3C Trace Context headers on outgoing calls, captures Logback log records, and exposes HTTP metrics.

  • Download the OpenTelemetry Java Agent into your project root by running this command:
curl -L -o opentelemetry-javaagent.jar https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar

What does this command do?

The -L flag tells curl to follow GitHub's download redirects. The -o flag saves the output to a file named opentelemetry-javaagent.jar in your current directory.

  • Confirm the file downloaded by running:
ls opentelemetry-javaagent.jar

You should see opentelemetry-javaagent.jar listed. The file is approximately 20-30 MB.

Download failed or file is 0 bytes?

Make sure you have an active internet connection. GitHub releases occasionally rate-limit downloads. Wait a moment and try the curl command again.

Help me troubleshoot the OpenTelemetry Java Agent download failing

Why the Java Agent instead of Spring Boot Starter?

The agent approach requires zero code changes and zero dependencies in your project. You attach it at startup and it instruments everything automatically. This is the model production teams use for brown-field applications where modifying source code is expensive or risky.

The Spring Boot Starter approach requires adding dependencies and configuration to each service. Both work, but the agent gives you observability without touching a single line of application code.

Start both services with the agent attached

Each service needs two pieces of configuration: a unique service name (so they appear as separate nodes in traces) and the OTLP endpoint pointing to the OTel Collector. You pass both as environment variables alongside the -javaagent flag.

  • If either service is still running from the previous step, stop them with Ctrl+C.
  • Open a new terminal tab or window for the Product Service.
  • Make sure you are in the observability-stack project root directory (the folder containing docker-compose.yml).
  • Start the Product Service with the agent by running this command:
OTEL_SERVICE_NAME=product-service \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 \
java -javaagent:opentelemetry-javaagent.jar -jar product-service/build/libs/product-service-0.0.1-SNAPSHOT.jar

What do these environment variables do?

  • OTEL_SERVICE_NAME=product-service gives this service a unique identity in all telemetry data. Traces, metrics, and logs are all tagged with this name.
  • OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 tells the agent to send all telemetry to the OTel Collector's HTTP receiver. Port 4318 is mapped from your Docker Compose stack.
  • -javaagent:opentelemetry-javaagent.jar attaches the agent to the JVM. It intercepts Spring Boot and RestTemplate calls at the bytecode level to generate telemetry automatically.

You should see the Spring Boot startup logs, plus additional lines from the OpenTelemetry agent confirming it has loaded. Wait until you see a line indicating the service started on port 8081.

  • Open a second terminal tab or window for the Order Service.
  • Navigate to the same project root directory (containing docker-compose.yml).
  • Start the Order Service with the agent by running this command:
OTEL_SERVICE_NAME=order-service \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 \
java -javaagent:opentelemetry-javaagent.jar -jar order-service/build/libs/order-service-0.0.1-SNAPSHOT.jar

Wait for the Order Service to start on port 8080. You now have both services running with full auto-instrumentation active.

Seeing "Address already in use" errors?

This means a service from your previous testing is still running on that port. Find and stop the previous process. On macOS/Linux, run lsof -i :8080 or lsof -i :8081 to find the PID, then kill it with kill <PID>.

Help me fix the address already in use error when starting my Spring Boot service

Generate traffic and verify telemetry in Grafana

With both services running and the agent attached, every HTTP request now generates traces, metrics, and correlated logs. Time to send some requests and confirm the data flows through the Collector to each backend.

  • Open a third terminal tab or window.
  • Send several requests to the Order Service, including one for product ID 3 (the intentionally slow one), by running these commands:
curl -X POST http://localhost:8080/orders -H "Content-Type: application/json" -d '{"productId": 1}'
curl -X POST http://localhost:8080/orders -H "Content-Type: application/json" -d '{"productId": 2}'
curl -X POST http://localhost:8080/orders -H "Content-Type: application/json" -d '{"productId": 3}'

What happens behind the scenes?

Each request generates a distributed trace that spans both services. The agent on the Order Service creates a parent span for the incoming POST request, then automatically injects W3C Trace Context headers into the outgoing RestTemplate call. The Product Service agent reads those headers and creates a child span linked to the same trace.

The request for product ID 3 takes over 2 seconds because of the intentional delay. This will show up clearly as a slow span in your traces.

You should see JSON responses from each request, including the product name and order ID. The third request (product 3) takes noticeably longer.

  • Verify that the OTel Collector is receiving and exporting telemetry by checking its logs:
docker logs otel-collector --tail 20

You should see log lines confirming successful exports to Prometheus, Loki, and Tempo. Look for messages mentioning data points, log records, and spans being exported.

  • Open Grafana in your browser at http://localhost:3000.
  • Click the hamburger menu (three horizontal lines) in the top left.
  • Click Explore.
  • In the datasource dropdown at the top, select Tempo.
  • Change the query type to Search.
  • Click Run query (or press Shift+Enter).

You should see a list of traces with both order-service and product-service spans visible. One trace will be noticeably longer than the others (the product ID 3 request).

No traces showing up in Tempo?

Give it 10-15 seconds after sending requests. The Collector batches data before forwarding (configured with a 5 second timeout).

Check that your Docker Compose stack is still running with docker compose ps. All 5 containers should be in the "running" state. Also confirm the OTel Collector logs do not show connection errors to Tempo.

Help me debug why traces are not appearing in Grafana Tempo

Your services are now fully instrumented. Every request generates correlated metrics, logs, and traces flowing through the Collector to their respective backends. Next up, you will explore each pillar independently in Grafana to understand what questions each signal type answers.

Explore the Three Pillars in Grafana

In the previous step, you attached the OpenTelemetry Java Agent to both services and confirmed that telemetry is flowing into your backends. Now it is time to see what that telemetry actually looks like.

Each observability pillar answers a different question about your system. Metrics answer "how is it performing over time?" Logs answer "what happened?" Traces answer "where was time spent?" In this step, you will query each pillar independently in Grafana to understand the unique value each one provides.

In this step, get ready to:
  • Query HTTP latency metrics using PromQL in the Prometheus datasource.
  • Explore structured log lines with embedded trace IDs using LogQL in the Loki datasource.
  • Inspect distributed trace waterfalls in the Tempo datasource and identify a slow span.
Query metrics with Prometheus

Metrics show you the overall health and performance of your services over time. The OTel agent automatically exposes HTTP request latency histograms that Prometheus scrapes from the Collector.

  • Return to Grafana at http://localhost:3000 in your browser.
  • Click Explore in the left side menu.
  • Select Prometheus from the datasource dropdown in the upper left.
  • Click the Code tab in the query editor toolbar to switch to raw PromQL mode.
  • Paste the following query into the query editor to see request latency histogram data:
http_server_request_duration_seconds_bucket
  • Click Run query in the upper right.

You should see multiple time series in the results panel, one for each latency bucket and service combination.

What does this metric show?

The http_server_request_duration_seconds_bucket metric is a histogram automatically generated by the OTel Java Agent. It records how many requests fell into each latency bucket (e.g., under 100ms, under 250ms, under 1s).

Histograms let you compute percentiles like the 95th percentile (p95), which tells you the latency that 95% of requests complete within.

Now calculate the 95th percentile latency to see how your services are really performing.

  • Clear the query editor and paste the following PromQL expression:
histogram_quantile(0.95, rate(http_server_request_duration_seconds_bucket[5m]))
  • Click Run query.

You should see a graph showing the p95 latency over time. Notice that some series show significantly higher latency than others. This is the slow product ID 3 response showing up in your metrics.

What does this PromQL expression do?

  • rate(...[5m]) calculates the per-second rate of bucket increases over the last 5 minutes.
  • histogram_quantile(0.95, ...) computes the 95th percentile from those rates.
  • Together, they answer: "What latency do 95% of requests complete within?"

Not seeing any metric data?

  • Metrics need time to accumulate. If you just started the services, send a few more requests and wait 15-30 seconds for Prometheus to scrape.
  • Confirm the Prometheus datasource is selected (not Loki or Tempo) in the upper left dropdown.
  • Check the time range picker in the upper right. Set it to Last 15 minutes to ensure you are looking at recent data.

Help me debug why my Prometheus metrics aren't showing in Grafana Explore.

Query logs with Loki

Logs capture detailed records of what happened inside each service. The OTel agent automatically injects trace context (trace ID and span ID) into every log record, making it possible to correlate logs with specific traces later.

  • In the datasource dropdown in the upper left, switch to Loki.
  • Click the Code tab if it is not already active.
  • Paste the following LogQL query to fetch logs from the Order Service:
{service_name="order-service"}
  • Click Run query.

You should see structured log lines appear in the results panel. Expand any log line by clicking on it.

  • Click on any log line to expand its details.
  • Look for the trace_id and span_id fields in the log record metadata.

Why are trace IDs in the logs?

The OTel Java Agent automatically injects the current trace ID and span ID into every log record emitted by Logback. This means every log line is linked to the exact request that produced it.

Later, you will use this trace ID to jump from a log line to the full distributed trace, or from a trace to the exact log lines that were emitted during that request.

No logs appearing for order-service?

  • Make sure you selected Loki (not Prometheus or Tempo) in the datasource dropdown.
  • Verify the query uses double quotes around the service name: {service_name="order-service"}.
  • Expand the time range to Last 1 hour if you generated traffic a while ago.

Help me troubleshoot why Loki isn't showing logs from my order-service.

Explore traces with Tempo

Traces show you the full journey of a request across service boundaries. Each trace is a tree of spans, where each span represents a unit of work (handling an HTTP request, making a downstream call, etc.).

  • In the datasource dropdown, switch to Tempo.
  • In the Tempo query editor, select Search as the query type (if not already selected).
  • In the Service Name dropdown, select order-service.
  • Click Run query.

You should see a list of traces returned, each with a trace ID, duration, and service name.

  • Click on any trace to open the waterfall view.

The waterfall shows the parent span (order-service handling POST /orders) and child spans (order-service calling the product-service, then product-service handling GET /products/{id}).

  • Look for a trace with a noticeably longer duration (over 2 seconds). Click on it to open the waterfall.
  • Identify which span accounts for the majority of the time. It will be the product-service span for product ID 3.

How to read a trace waterfall

  • The top-level span is the entry point: order-service receiving the POST /orders request.
  • Nested below it is the outgoing HTTP call from order-service to product-service (created by the auto-instrumented RestTemplate).
  • Nested further is the product-service span handling GET /products/{id}.
  • The horizontal bars show relative time. A long bar on the product-service span for ID 3 reveals the 2-second simulated delay.

Not seeing traces in Tempo?

  • Confirm you selected Tempo as the datasource, not Prometheus or Loki.
  • Make sure you have generated traffic since attaching the OTel agent. The agent only produces traces for requests that happen while it is running.
  • If the Service Name dropdown is empty, wait 30 seconds and refresh the page. Tempo needs time to index incoming traces.

Help me debug why Tempo isn't showing traces from my services.

You have now explored each pillar independently and seen the unique data each one provides. Next up, you will tie all three together to debug the slow request for product ID 3, demonstrating the full incident investigation workflow.

Correlate Signals and Debug a Slow Request

In the previous step, you explored each observability pillar independently. You saw metrics showing latency distributions, logs with embedded trace IDs, and trace waterfalls revealing inter-service calls.

Now you will tie all three pillars together to debug a specific problem. This is the workflow an SRE follows during a real incident: start from a metric that says "something is slow," jump to traces to find "where time was spent," then land on logs to understand "why."

In this step, get ready to:
  • Identify an elevated p95 latency metric as the starting signal.
  • Jump from metrics to traces to locate the slow span.
  • Correlate the trace with logs to diagnose the root cause.
Start from metrics to detect the problem

In a real incident, an alert fires because a latency metric crossed a threshold. You already saw the elevated p95 when exploring Prometheus in the previous step. Now you will use that same signal as the starting point of your debugging workflow.

  • Return to Grafana at http://localhost:3000.
  • Click the hamburger menu icon in the top left, then select Explore.
  • Select Prometheus from the datasource dropdown at the top.
  • Switch the query editor to Code mode (if it is not already) by clicking the Code button on the right side of the query row.
  • Enter this PromQL query to calculate the 95th percentile request latency:
histogram_quantile(0.95, rate(http_server_request_duration_seconds_bucket[5m]))
  • Click Run query (or press Shift+Enter).

What does this query tell you?

The histogram_quantile function calculates the latency at which 95% of requests complete. If this value is elevated (well over 2 seconds), it means some requests are taking significantly longer than others.

The metric tells you "something is slow" but not why or where. In a production environment, this is where an alert would fire and page an on-call engineer.

You should see the graph spike above 2 seconds for the order-service endpoint. This confirms a latency problem exists. Now you need to find out where the time is being spent.

Jump to traces to locate the slow span

Metrics told you a problem exists. Distributed tracing will tell you exactly where in the request lifecycle the time was spent.

  • Switch the datasource dropdown from Prometheus to Tempo.
  • In the Search tab, set the Service Name filter to order-service.
  • Set the Duration filter to greater than 2s to isolate slow requests.
  • Click Run query.

You should see one or more traces with durations exceeding 2 seconds. These are the requests that hit product ID 3.

  • Click on one of the slow traces to open its detail view.
  • Examine the trace waterfall. You should see a parent span from order-service (handling POST /orders) and a child span showing the call to product-service (handling GET /products/3).
  • Notice that the product-service span accounts for the vast majority of the total trace duration (over 2 seconds).

What does the trace tell you?

The waterfall view reveals that the order-service itself is fast. The bottleneck is in the downstream product-service, specifically when handling product ID 3.

You now know "where" the time was spent. The next question is "why" the product-service is slow for that specific product.

  • In the trace detail panel, locate the Trace ID field at the top of the view. It is a long hexadecimal string.
  • Copy the full trace ID value. You will use it to find the exact logs from this request.
Correlate with logs to diagnose the root cause

You know the product-service is slow for product 3. Logs will tell you exactly what the service was doing during that time.

  • Switch the datasource dropdown from Tempo to Loki.
  • Switch the query editor to Code mode if it is not already active.
  • Enter this LogQL query, replacing the placeholder with the trace ID you just copied:
{service_name="product-service"} |= "[[TRACE_ID="your copied trace ID"]]"
  • Click Run query.

What does this query do?

The {service_name="product-service"} part selects all logs from the product-service. The |= operator filters those logs to only show lines containing your specific trace ID.

This isolates the exact log lines that were emitted during the slow request you found in Tempo.

You should see log lines from the product-service for that specific request. Look for the warning log line that says product 3 requires a slow database lookup.

  • Expand one of the log lines by clicking on it.
  • Notice the trace_id and span_id fields embedded in the log record. These were injected automatically by the OpenTelemetry Java Agent.
  • Confirm you can see the warning message about the slow database lookup for product 3. This is the root cause.

Why is this correlation powerful?

You just followed the exact debugging workflow a production SRE uses during an incident. Metrics told you "something is slow." Traces told you "where the time was spent" (product-service, product 3). Logs told you "why" (a slow database lookup).

The trace ID is the thread that connects all three pillars. Without it, you would be searching through thousands of log lines with no way to isolate the relevant ones.

Not seeing log results for your trace ID?

Make sure you pasted the full trace ID string with no extra spaces. The trace ID is case-sensitive.

Check that the time range selector at the top right of Grafana covers the period when you sent your requests. If you generated traffic more than an hour ago, expand the time window.

If Loki returns no results at all, ensure both services are still running with the OTel agent and that the Collector is forwarding logs. Help me debug why my Loki LogQL query returns no results for a trace ID that exists in Tempo.

You just completed a full incident debugging workflow using the three pillars of observability. In a production system, this same pattern lets you go from a pager alert to root cause in minutes instead of hours. Next up, you will take your observability skills further by adding custom instrumentation.

Secret mission

Add Manual Instrumentation

The OTel agent gives you automatic observability, but production services also need custom business telemetry. Can you add manual instrumentation to capture a custom span for order validation logic and a counter metric for total orders created?

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

This project runs entirely locally using Docker containers, so there are no ongoing cloud costs.

Resources you used:

  • Docker containers: Grafana, Prometheus, Loki, Tempo, and the OpenTelemetry Collector (5 services consuming approximately 2-3 GB RAM total).
  • Docker volumes: prometheus-data, loki-data, and tempo-data storing metrics, logs, and trace data.
  • Running Spring Boot processes (Order Service on port 8080 and Product Service on port 8081).
  • Local project directory on your Desktop with all configuration files, service code, and the downloaded OpenTelemetry Java Agent JAR.

Keep everything running

No action needed. Choose this if you are still actively experimenting with the observability stack.

  • Your Docker Compose stack continues running in the background, collecting telemetry from any services you start with the OpenTelemetry Java Agent.
  • Grafana remains accessible at http://localhost:3000 for exploring metrics, logs, and traces.
  • Try adding more services, building Grafana dashboards, or writing more complex PromQL and LogQL queries.

Pause - I'll come back to this later

Shut down running processes to free up memory while keeping your data and files intact for later.

  • Stop the Order Service and Product Service by pressing Ctrl+C in each terminal window where they are running.
  • Stop the Docker Compose stack (without removing data) by running:
docker compose stop
  • When you are ready to resume, navigate back to your project directory and run:
docker compose start
  • Your Prometheus metrics, Loki logs, and Tempo traces are preserved in the Docker volumes.

Delete - I don't want to use this again

Remove all project resources and start fresh. This deletes all containers, volumes, and stored telemetry data permanently.

  • Stop the Order Service and Product Service by pressing Ctrl+C in each terminal window where they are running.
  • Remove all Docker containers and volumes by running this command from your project directory:
docker compose down -v
  • Delete the project directory. On macOS or Linux, run:
cd ~/Desktop
rm -rf observability-stack
  • On Windows, run:
cd ~\Desktop
Remove-Item -Recurse -Force observability-stack
  • Docker Desktop, Java 21, and Gradle remain installed on your machine. These are general-purpose tools useful for other projects.

Nice Work!

Nice Work!

You just built a production-grade observability platform from scratch. Two Spring Boot microservices are now fully monitored with correlated metrics, logs, and distributed traces, all visualized through a single Grafana dashboard.

You've learned how to:

  • Built a complete observability backend (Prometheus, Loki, Tempo, OpenTelemetry Collector, and Grafana) orchestrated via Docker Compose.
  • Instrumented two Spring Boot microservices with zero code changes using the OpenTelemetry Java Agent, automatically capturing HTTP metrics, distributed traces, and correlated log records.
  • Demonstrated the full incident debugging workflow: from a latency metric (detect), to the offending trace (locate), to the exact log line (diagnose).
  • Secret Mission: Added manual instrumentation with custom spans and business metrics alongside auto-instrumentation.

Ready to quiz yourself?