Microservices with Docker and Grafana

Build two Spring Boot microservices with full observability using Grafana, Loki, and Tempo.

Introduction

30 Second Summary

When a request fails in a system with multiple services, finding the root cause feels like searching for a lost package that passed through five different shipping centers. You need a way to trace it through every hop.

In this project, you will build two Spring Boot microservices that talk to each other via REST, containerize them with Docker, and set up a full Grafana observability stack to trace requests end-to-end.

What You'll Build

A local microservices system with production-grade observability where you can follow a single request across services through distributed traces, structured logs, and metrics.

By the end of this project, you'll have:

  • Two Spring Boot microservices (Order Service and Product Service) communicating via REST, containerized with Docker Compose.
  • A full Grafana observability stack with Loki for log aggregation, Tempo for distributed tracing, and Prometheus for metrics.
  • An Angular frontend that triggers requests across your microservices, with traces visible end-to-end in Grafana.
  • Secret Mission: Add RED metrics dashboards and Prometheus alerting rules to monitor your microservices.

Are there any prerequisites?

You should have basic Java and Angular knowledge. Familiarity with REST APIs is helpful but not required.

Build the Product Service

Your development environment is ready. Now it's time to build the first microservice in your system.

The Product Service is the data layer of your architecture. It stores product information and exposes it via REST endpoints. Later, the Order Service will call this service to look up products when creating orders.

In this step, get ready to:
  • Create a Spring Boot project with REST endpoints that return product data.
  • Add OpenTelemetry tracing and Loki logging dependencies for full observability.
  • Containerize the service with a multi-stage Docker build.
Generate the Product Service project

Spring Initializr is a web tool that generates Spring Boot project skeletons. You select your dependencies and it gives you a ready-to-open folder with Maven configured.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project root folder and move into it by running these commands:
mkdir microservices-observability
cd microservices-observability
  • Open start.spring.io in your browser.
  • Set Project to Maven.
  • Set Language to Java.
  • Set Spring Boot version to the latest 3.5.x available.
  • Under Project Metadata, set Group to com.example and Artifact to product-service.
  • Set Java version to 17.
  • Click Add Dependencies and add Spring Web and Spring Boot Actuator.
  • Click Generate to download the zip file.
  • Extract the zip file and move the product-service folder into your microservices-observability folder on the Desktop.

What did Spring Initializr generate?

The downloaded zip contains a Maven project with the standard Spring Boot folder structure. Inside you'll find a pom.xml (your dependency manifest), an application.properties file, and a ProductServiceApplication.java main class.

You'll replace the generated pom.xml in the next substep to add observability dependencies. The generated main class already matches what you need.

  • Confirm the generated ProductServiceApplication.java at product-service/src/main/java/com/example/productservice/ProductServiceApplication.java matches this:
package com.example.productservice;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class ProductServiceApplication {
    public static void main(String[] args) {
        SpringApplication.run(ProductServiceApplication.class, args);
    }
}

What does this code do?

The @SpringBootApplication annotation combines auto-configuration, component scanning, and configuration into one shortcut. The main method launches the embedded web server and your application.

Now you'll add the REST controller that serves product data.

  • Create a new file at product-service/src/main/java/com/example/productservice/ProductController.java and paste the following code:
package com.example.productservice;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.List;
import java.util.Map;

@RestController
@RequestMapping("/api/products")
public class ProductController {

    private static final Logger log = LoggerFactory.getLogger(ProductController.class);

    private final List<Map<String, Object>> products = List.of(
        Map.of("id", 1, "name", "Laptop", "price", 999.99),
        Map.of("id", 2, "name", "Phone", "price", 699.99),
        Map.of("id", 3, "name", "Tablet", "price", 449.99)
    );

    @GetMapping
    public List<Map<String, Object>> getAllProducts() {
        log.info("Fetching all products");
        return products;
    }

    @GetMapping("/{id}")
    public Map<String, Object> getProduct(@PathVariable int id) {
        log.info("Fetching product with id: {}", id);
        return products.stream()
            .filter(p -> (int) p.get("id") == id)
            .findFirst()
            .orElse(Map.of("error", "Product not found"));
    }
}

What does this code do?

  • @RestController tells Spring this class handles HTTP requests and returns JSON responses directly.
  • @RequestMapping("/api/products") sets the base URL path for all endpoints in this controller.
  • The products list is a hardcoded in-memory data store with three items.
  • getAllProducts() handles GET requests to /api/products and returns all three products.
  • getProduct() handles GET requests to /api/products/{id} and returns a single product by ID.
  • Every request is logged with log.info(). These logs will include trace IDs once observability is configured.
  • Save ProductController.java.

File not in the right place?

The file must be at product-service/src/main/java/com/example/productservice/ProductController.java. If the productservice folder doesn't exist, create it. The package declaration in the Java file must match the folder path exactly.

Help me fix my file structure for the Product Service.

Add observability dependencies and configuration

The Product Service needs tracing, metrics, and structured logging to participate in the observability stack. You'll add OpenTelemetry for distributed tracing, Prometheus for metrics collection, and a Loki appender for log shipping.

  • Open product-service/pom.xml and replace its entire contents with the following:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.5.14</version>
        <relativePath/>
    </parent>
    <groupId>com.example</groupId>
    <artifactId>product-service</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>product-service</name>

    <properties>
        <java.version>17</java.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <!-- Tracing: Micrometer to OpenTelemetry bridge -->
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-tracing-bridge-otel</artifactId>
        </dependency>
        <!-- Tracing: OTLP exporter (sends traces to Tempo via OTel Collector) -->
        <dependency>
            <groupId>io.opentelemetry</groupId>
            <artifactId>opentelemetry-exporter-otlp</artifactId>
        </dependency>
        <!-- Metrics: Prometheus registry -->
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-registry-prometheus</artifactId>
        </dependency>
        <!-- Logging: Loki appender -->
        <dependency>
            <groupId>com.github.loki4j</groupId>
            <artifactId>loki-logback-appender</artifactId>
            <version>1.5.2</version>
            <scope>runtime</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>
</project>

What do these dependencies do?

  • micrometer-tracing-bridge-otel bridges Spring Boot's Micrometer Observation API to OpenTelemetry. This is what generates trace spans automatically for every HTTP request.
  • opentelemetry-exporter-otlp ships those trace spans to a collector (Tempo) using the OTLP protocol.
  • micrometer-registry-prometheus exposes metrics in Prometheus format via the Actuator /actuator/prometheus endpoint.
  • loki-logback-appender pushes structured logs directly to Grafana Loki, including trace IDs for correlation.
  • Save pom.xml.

Next, configure the application properties to enable tracing and metrics endpoints.

  • Open product-service/src/main/resources/application.properties and replace its contents with:
spring.application.name=product-service
server.port=8081

# Tracing - send all traces (100% sampling)
management.tracing.sampling.probability=1.0

# OTLP exporter - send traces to Tempo
management.otlp.tracing.endpoint=http://tempo:4318/v1/traces

# Actuator - expose prometheus and health endpoints
management.endpoints.web.exposure.include=health,prometheus,metrics

What does this configuration do?

  • management.tracing.sampling.probability=1.0 tells Spring Boot to trace 100% of requests. The default is only 10%.
  • management.otlp.tracing.endpoint points to Tempo's HTTP OTLP receiver. The hostname tempo resolves via Docker networking (you'll set this up later).
  • management.endpoints.web.exposure.include exposes health checks, Prometheus metrics, and general metrics endpoints over HTTP.
  • Save application.properties.

Finally, configure structured logging that includes trace IDs in every log line and ships logs to Loki.

  • Create a new file at product-service/src/main/resources/logback-spring.xml and paste the following:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <include resource="org/springframework/boot/logging/logback/defaults.xml"/>

    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} - [traceId=%mdc{traceId:-none}] [spanId=%mdc{spanId:-none}] %msg%n</pattern>
        </encoder>
    </appender>

    <appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender">
        <http>
            <url>http://loki:3100/loki/api/v1/push</url>
        </http>
        <format>
            <label>
                <pattern>service_name=${spring.application.name},level=%level</pattern>
            </label>
            <message>
                <pattern>{"timestamp":"%d{yyyy-MM-dd HH:mm:ss.SSS}","level":"%level","logger":"%logger{36}","traceId":"%mdc{traceId:-none}","spanId":"%mdc{spanId:-none}","message":"%msg"}</pattern>
            </message>
        </format>
    </appender>

    <root level="INFO">
        <appender-ref ref="CONSOLE"/>
        <appender-ref ref="LOKI"/>
    </root>
</configuration>

What does this logging config do?

  • The CONSOLE appender prints logs to your terminal with the traceId and spanId embedded in each line. This lets you visually correlate logs with traces.
  • The LOKI appender pushes logs as structured JSON directly to Grafana Loki. Each log entry carries traceId and spanId fields for trace-to-log correlation.
  • Labels like service_name and level let you filter logs by service and severity in Grafana.
  • Save logback-spring.xml.

Seeing XML parsing errors?

Make sure the file starts with <?xml version="1.0" encoding="UTF-8"?> on the very first line with no whitespace before it. XML is strict about this.

Help me debug my logback-spring.xml configuration.

Create the Dockerfile

To run this service inside a Docker container, you need a Dockerfile that builds the Java application and packages it into a lightweight image. A multi-stage build keeps the final image small by separating the build tools from the runtime.

  • Create a new file called Dockerfile in the product-service/ folder and paste the following:
FROM maven:3.9-eclipse-temurin-17 AS build
WORKDIR /app
COPY pom.xml .
RUN mvn dependency:go-offline -B
COPY src ./src
RUN mvn package -DskipTests -B

FROM eclipse-temurin:17-jre
WORKDIR /app
COPY --from=build /app/target/*.jar app.jar
EXPOSE 8081
ENTRYPOINT ["java", "-jar", "app.jar"]

What does this Dockerfile do?

  • Stage 1 (build): Uses a full Maven image to download dependencies and compile the application into a JAR file.
  • mvn dependency:go-offline downloads all dependencies first, which gets cached by Docker. This means subsequent builds only recompile your code.
  • Stage 2 (runtime): Uses a minimal JRE-only image. It copies just the compiled JAR from stage 1, keeping the final image small.
  • EXPOSE 8081 documents which port the service listens on.
  • Save Dockerfile.
  • Verify your file structure by running this from the microservices-observability folder:
ls product-service/

You should see Dockerfile, pom.xml, and a src folder listed in the output.

Missing files in the listing?

Double check that you extracted the Spring Initializr zip into microservices-observability/product-service/ and not into a nested subfolder. The Dockerfile must be at the same level as pom.xml and src/.

Help me fix my Product Service folder structure.

✔️ Awesome, I've got everything!

Great. Double check you've saved all files in the product-service/ folder before moving on.

ⓧ I'd like to double check the full code

Here are all the files you should have in your product-service/ folder:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.5.14</version>
        <relativePath/>
    </parent>
    <groupId>com.example</groupId>
    <artifactId>product-service</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>product-service</name>

    <properties>
        <java.version>17</java.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <!-- Tracing: Micrometer to OpenTelemetry bridge -->
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-tracing-bridge-otel</artifactId>
        </dependency>
        <!-- Tracing: OTLP exporter (sends traces to Tempo via OTel Collector) -->
        <dependency>
            <groupId>io.opentelemetry</groupId>
            <artifactId>opentelemetry-exporter-otlp</artifactId>
        </dependency>
        <!-- Metrics: Prometheus registry -->
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-registry-prometheus</artifactId>
        </dependency>
        <!-- Logging: Loki appender -->
        <dependency>
            <groupId>com.github.loki4j</groupId>
            <artifactId>loki-logback-appender</artifactId>
            <version>1.5.2</version>
            <scope>runtime</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>
</project>
spring.application.name=product-service
server.port=8081

# Tracing - send all traces (100% sampling)
management.tracing.sampling.probability=1.0

# OTLP exporter - send traces to Tempo
management.otlp.tracing.endpoint=http://tempo:4318/v1/traces

# Actuator - expose prometheus and health endpoints
management.endpoints.web.exposure.include=health,prometheus,metrics
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <include resource="org/springframework/boot/logging/logback/defaults.xml"/>

    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} - [traceId=%mdc{traceId:-none}] [spanId=%mdc{spanId:-none}] %msg%n</pattern>
        </encoder>
    </appender>

    <appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender">
        <http>
            <url>http://loki:3100/loki/api/v1/push</url>
        </http>
        <format>
            <label>
                <pattern>service_name=${spring.application.name},level=%level</pattern>
            </label>
            <message>
                <pattern>{"timestamp":"%d{yyyy-MM-dd HH:mm:ss.SSS}","level":"%level","logger":"%logger{36}","traceId":"%mdc{traceId:-none}","spanId":"%mdc{spanId:-none}","message":"%msg"}</pattern>
            </message>
        </format>
    </appender>

    <root level="INFO">
        <appender-ref ref="CONSOLE"/>
        <appender-ref ref="LOKI"/>
    </root>
</configuration>
package com.example.productservice;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class ProductServiceApplication {
    public static void main(String[] args) {
        SpringApplication.run(ProductServiceApplication.class, args);
    }
}
package com.example.productservice;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.List;
import java.util.Map;

@RestController
@RequestMapping("/api/products")
public class ProductController {

    private static final Logger log = LoggerFactory.getLogger(ProductController.class);

    private final List<Map<String, Object>> products = List.of(
        Map.of("id", 1, "name", "Laptop", "price", 999.99),
        Map.of("id", 2, "name", "Phone", "price", 699.99),
        Map.of("id", 3, "name", "Tablet", "price", 449.99)
    );

    @GetMapping
    public List<Map<String, Object>> getAllProducts() {
        log.info("Fetching all products");
        return products;
    }

    @GetMapping("/{id}")
    public Map<String, Object> getProduct(@PathVariable int id) {
        log.info("Fetching product with id: {}", id);
        return products.stream()
            .filter(p -> (int) p.get("id") == id)
            .findFirst()
            .orElse(Map.of("error", "Product not found"));
    }
}
FROM maven:3.9-eclipse-temurin-17 AS build
WORKDIR /app
COPY pom.xml .
RUN mvn dependency:go-offline -B
COPY src ./src
RUN mvn package -DskipTests -B

FROM eclipse-temurin:17-jre
WORKDIR /app
COPY --from=build /app/target/*.jar app.jar
EXPOSE 8081
ENTRYPOINT ["java", "-jar", "app.jar"]

Your Product Service is fully built and ready to be containerized. Next up, you'll build the Order Service that calls this one and see how trace context propagates automatically across services.

Build the Order Service

Your Product Service is ready to serve product data. Now you need a second service that actually processes orders.

The Order Service receives order requests from the frontend and calls the Product Service to fetch product details. This service-to-service communication is where distributed tracing becomes essential. When a single request crosses multiple services, you need to follow its path end-to-end.

In this step, get ready to:
  • Create the Order Service project with REST endpoints that call the Product Service.
  • Configure observability dependencies and Docker networking settings.
  • Containerize the Order Service with a Dockerfile.
Create the Order Service project with dependencies and configuration

The Order Service follows the same project structure as the Product Service. The key difference is an additional configuration property that tells it where to find the Product Service on the Docker network.

  • Create the Order Service directory structure by running these commands in your terminal from the project root:
mkdir -p order-service/src/main/java/com/example/orderservice
mkdir -p order-service/src/main/resources
  • Confirm the directories were created by running:
ls order-service/src/main/java/com/example/orderservice

You should see no output (empty directory), which means the path exists.

  • Create the order-service/pom.xml file and paste in the following:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.5.14</version>
        <relativePath/>
    </parent>
    <groupId>com.example</groupId>
    <artifactId>order-service</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>order-service</name>

    <properties>
        <java.version>17</java.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <!-- Tracing: Micrometer to OpenTelemetry bridge -->
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-tracing-bridge-otel</artifactId>
        </dependency>
        <!-- Tracing: OTLP exporter (sends traces to Tempo via OTel Collector) -->
        <dependency>
            <groupId>io.opentelemetry</groupId>
            <artifactId>opentelemetry-exporter-otlp</artifactId>
        </dependency>
        <!-- Metrics: Prometheus registry -->
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-registry-prometheus</artifactId>
        </dependency>
        <!-- Logging: Loki appender -->
        <dependency>
            <groupId>com.github.loki4j</groupId>
            <artifactId>loki-logback-appender</artifactId>
            <version>1.5.2</version>
            <scope>runtime</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>
</project>

What does this file do?

This pom.xml has the same dependencies as your Product Service. The artifact ID is order-service instead of product-service, but all the observability libraries (Micrometer tracing, OTLP exporter, Prometheus registry, Loki appender) are identical.

Both services need the same instrumentation so their traces, logs, and metrics flow into the same observability stack.

  • Create the order-service/src/main/resources/application.properties file and paste in the following:
spring.application.name=order-service
server.port=8080

# Tracing
management.tracing.sampling.probability=1.0
management.otlp.tracing.endpoint=http://tempo:4318/v1/traces

# Actuator
management.endpoints.web.exposure.include=health,prometheus,metrics

# Product service URL (Docker networking)
product.service.url=http://product-service:8081

What does product.service.url do?

The product.service.url property tells the Order Service where to find the Product Service. The hostname product-service is the container name in Docker Compose.

Docker Compose creates a network where each service is reachable by its service name. This means your services don't need hardcoded IP addresses.

  • Create the order-service/src/main/resources/logback-spring.xml file and paste in the following:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <include resource="org/springframework/boot/logging/logback/defaults.xml"/>

    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} - [traceId=%mdc{traceId:-none}] [spanId=%mdc{spanId:-none}] %msg%n</pattern>
        </encoder>
    </appender>

    <appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender">
        <http>
            <url>http://loki:3100/loki/api/v1/push</url>
        </http>
        <format>
            <label>
                <pattern>service_name=${spring.application.name},level=%level</pattern>
            </label>
            <message>
                <pattern>{"timestamp":"%d{yyyy-MM-dd HH:mm:ss.SSS}","level":"%level","logger":"%logger{36}","traceId":"%mdc{traceId:-none}","spanId":"%mdc{spanId:-none}","message":"%msg"}</pattern>
            </message>
        </format>
    </appender>

    <root level="INFO">
        <appender-ref ref="CONSOLE"/>
        <appender-ref ref="LOKI"/>
    </root>
</configuration>

Why is this identical to the Product Service?

Both services push structured JSON logs to Loki with the same format. The service_name label is pulled dynamically from spring.application.name, so Loki can tell which logs came from which service.

The traceId and spanId fields embedded in each log line are what allow Grafana to jump from a trace directly to its matching logs.

  • Save all three files.

File not saving to the right place?

Double check that application.properties and logback-spring.xml are inside order-service/src/main/resources/, not at the project root. The nested directory structure matters for Spring Boot to find these files.

help me verify my order-service directory structure is correct

Build the REST endpoints with RestClient

The Order Service uses Spring Boot's RestClient to call the Product Service. RestClient is the modern synchronous HTTP client that automatically propagates trace context between services.

Why RestClient instead of RestTemplate?

Spring Boot 3.5.14 promotes RestClient as the modern synchronous HTTP client. It uses a fluent API with method chaining, making code more readable.

The key advantage for observability is that RestClient is auto-instrumented by Micrometer Observation. Trace context (the trace ID and span ID) propagates automatically when one service calls another. You don't need to manually forward any headers.

  • Create the order-service/src/main/java/com/example/orderservice/OrderServiceApplication.java file and paste in the following:
package com.example.orderservice;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.web.client.RestClient;

@SpringBootApplication
public class OrderServiceApplication {
    public static void main(String[] args) {
        SpringApplication.run(OrderServiceApplication.class, args);
    }

    // Create a RestClient bean that Spring auto-instruments for tracing
    @Bean
    public RestClient restClient(RestClient.Builder builder) {
        return builder.build();
    }
}

What does the RestClient bean do?

The @Bean method creates a RestClient using the auto-configured RestClient.Builder. Spring Boot automatically injects Micrometer observation into this builder.

This means every HTTP call made through this RestClient automatically creates a child span and forwards the traceparent header to the downstream service. No manual instrumentation needed.

  • Save OrderServiceApplication.java.
  • Create the order-service/src/main/java/com/example/orderservice/OrderController.java file and paste in the following:
package com.example.orderservice;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.core.ParameterizedTypeReference;
import org.springframework.web.bind.annotation.*;
import org.springframework.web.client.RestClient;

import java.time.LocalDateTime;
import java.util.List;
import java.util.Map;

@RestController
@RequestMapping("/api/orders")
@CrossOrigin(origins = "*")
public class OrderController {

    private static final Logger log = LoggerFactory.getLogger(OrderController.class);

    private final RestClient restClient;
    private final String productServiceUrl;

    // Inject RestClient and the product service URL from application.properties
    public OrderController(RestClient restClient,
                           @Value("${product.service.url}") String productServiceUrl) {
        this.restClient = restClient;
        this.productServiceUrl = productServiceUrl;
    }

    // POST /api/orders - creates an order by fetching product details from Product Service
    @PostMapping
    public Map<String, Object> createOrder(@RequestBody Map<String, Object> orderRequest) {
        int productId = (int) orderRequest.get("productId");
        log.info("Creating order for product: {}", productId);

        // Call Product Service - trace context propagates automatically
        Map<String, Object> product = restClient.get()
            .uri(productServiceUrl + "/api/products/" + productId)
            .retrieve()
            .body(new ParameterizedTypeReference<>() {});

        log.info("Product fetched: {}", product.get("name"));

        return Map.of(
            "orderId", (int) (Math.random() * 10000),
            "product", product,
            "status", "CREATED",
            "timestamp", LocalDateTime.now().toString()
        );
    }

    // GET /api/orders/products - proxy endpoint to list all products
    @GetMapping("/products")
    public List<Map<String, Object>> getProducts() {
        log.info("Fetching products from product service");

        return restClient.get()
            .uri(productServiceUrl + "/api/products")
            .retrieve()
            .body(new ParameterizedTypeReference<>() {});
    }
}

What does this controller do?

  • POST /api/orders accepts an order request with a productId, calls the Product Service to fetch product details, then returns the combined order response.
  • GET /api/orders/products is a proxy endpoint that forwards the request to the Product Service's product list. The Angular frontend will use this.
  • @CrossOrigin(origins = "*") allows the Angular frontend to call this service from a different origin during development.
  • Every log.info() call automatically includes the trace ID in the output (thanks to the logback config). This is how you correlate logs with traces in Grafana.
  • Save OrderController.java.
  • Verify the project compiles by running this from the order-service directory:
cd order-service
mvn compile

You should see BUILD SUCCESS in the terminal output. This confirms all dependencies resolved and your Java files have no syntax errors.

Seeing a compilation error?

Check that OrderServiceApplication.java and OrderController.java are both inside order-service/src/main/java/com/example/orderservice/. The package declaration must match the directory path exactly.

Also verify your pom.xml is in the order-service/ root (not inside src/).

help me fix my Order Service compilation error

Create the Dockerfile

The Order Service Dockerfile uses the same multi-stage build pattern as the Product Service. The only difference is the exposed port.

  • Create the order-service/Dockerfile file and paste in the following:
# Stage 1: Build the application with Maven
FROM maven:3.9-eclipse-temurin-17 AS build
WORKDIR /app
COPY pom.xml .
RUN mvn dependency:go-offline -B
COPY src ./src
RUN mvn package -DskipTests -B

# Stage 2: Run the application with a lightweight JRE
FROM eclipse-temurin:17-jre
WORKDIR /app
COPY --from=build /app/target/*.jar app.jar
EXPOSE 8080
ENTRYPOINT ["java", "-jar", "app.jar"]

What changed from the Product Service Dockerfile?

Only the port number. The Product Service exposes port 8081, while the Order Service exposes port 8080. The multi-stage build pattern (Maven compile in one stage, run in a lightweight JRE in the second) is identical.

Separating the build and runtime stages keeps your final Docker image small. The Maven build tools and source code don't ship in the production container.

  • Save Dockerfile.
  • Navigate back to the project root by running:
cd ..
  • Verify both service directories exist by running:
ls

You should see both product-service and order-service directories listed.

Only see one service directory?

Make sure you created the order-service directory at the same level as product-service, not inside it. Both should be siblings in your project root folder.

help me fix my project directory structure for the microservices project

✔️ Awesome, I've got everything!

Great. Double check you've saved all files in the order-service directory.

ⓧ I'd like to double check the full code

Here is the complete code for each file in the Order Service. Compare yours against these:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.5.14</version>
        <relativePath/>
    </parent>
    <groupId>com.example</groupId>
    <artifactId>order-service</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>order-service</name>

    <properties>
        <java.version>17</java.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-tracing-bridge-otel</artifactId>
        </dependency>
        <dependency>
            <groupId>io.opentelemetry</groupId>
            <artifactId>opentelemetry-exporter-otlp</artifactId>
        </dependency>
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-registry-prometheus</artifactId>
        </dependency>
        <dependency>
            <groupId>com.github.loki4j</groupId>
            <artifactId>loki-logback-appender</artifactId>
            <version>1.5.2</version>
            <scope>runtime</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>
</project>
spring.application.name=order-service
server.port=8080

management.tracing.sampling.probability=1.0
management.otlp.tracing.endpoint=http://tempo:4318/v1/traces

management.endpoints.web.exposure.include=health,prometheus,metrics

product.service.url=http://product-service:8081
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <include resource="org/springframework/boot/logging/logback/defaults.xml"/>

    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} - [traceId=%mdc{traceId:-none}] [spanId=%mdc{spanId:-none}] %msg%n</pattern>
        </encoder>
    </appender>

    <appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender">
        <http>
            <url>http://loki:3100/loki/api/v1/push</url>
        </http>
        <format>
            <label>
                <pattern>service_name=${spring.application.name},level=%level</pattern>
            </label>
            <message>
                <pattern>{"timestamp":"%d{yyyy-MM-dd HH:mm:ss.SSS}","level":"%level","logger":"%logger{36}","traceId":"%mdc{traceId:-none}","spanId":"%mdc{spanId:-none}","message":"%msg"}</pattern>
            </message>
        </format>
    </appender>

    <root level="INFO">
        <appender-ref ref="CONSOLE"/>
        <appender-ref ref="LOKI"/>
    </root>
</configuration>
package com.example.orderservice;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.web.client.RestClient;

@SpringBootApplication
public class OrderServiceApplication {
    public static void main(String[] args) {
        SpringApplication.run(OrderServiceApplication.class, args);
    }

    @Bean
    public RestClient restClient(RestClient.Builder builder) {
        return builder.build();
    }
}
package com.example.orderservice;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.core.ParameterizedTypeReference;
import org.springframework.web.bind.annotation.*;
import org.springframework.web.client.RestClient;

import java.time.LocalDateTime;
import java.util.List;
import java.util.Map;

@RestController
@RequestMapping("/api/orders")
@CrossOrigin(origins = "*")
public class OrderController {

    private static final Logger log = LoggerFactory.getLogger(OrderController.class);

    private final RestClient restClient;
    private final String productServiceUrl;

    public OrderController(RestClient restClient,
                           @Value("${product.service.url}") String productServiceUrl) {
        this.restClient = restClient;
        this.productServiceUrl = productServiceUrl;
    }

    @PostMapping
    public Map<String, Object> createOrder(@RequestBody Map<String, Object> orderRequest) {
        int productId = (int) orderRequest.get("productId");
        log.info("Creating order for product: {}", productId);

        Map<String, Object> product = restClient.get()
            .uri(productServiceUrl + "/api/products/" + productId)
            .retrieve()
            .body(new ParameterizedTypeReference<>() {});

        log.info("Product fetched: {}", product.get("name"));

        return Map.of(
            "orderId", (int) (Math.random() * 10000),
            "product", product,
            "status", "CREATED",
            "timestamp", LocalDateTime.now().toString()
        );
    }

    @GetMapping("/products")
    public List<Map<String, Object>> getProducts() {
        log.info("Fetching products from product service");

        return restClient.get()
            .uri(productServiceUrl + "/api/products")
            .retrieve()
            .body(new ParameterizedTypeReference<>() {});
    }
}
FROM maven:3.9-eclipse-temurin-17 AS build
WORKDIR /app
COPY pom.xml .
RUN mvn dependency:go-offline -B
COPY src ./src
RUN mvn package -DskipTests -B

FROM eclipse-temurin:17-jre
WORKDIR /app
COPY --from=build /app/target/*.jar app.jar
EXPOSE 8080
ENTRYPOINT ["java", "-jar", "app.jar"]

You now have two microservices that can talk to each other. The Order Service calls the Product Service, and trace context flows automatically between them. Next up, you'll build the Angular frontend that triggers these cross-service requests.

Build the Angular Frontend

Your two Spring Boot microservices are ready to receive requests. Now you need a way to trigger those requests and see the results.

In this step, you'll build an Angular frontend that calls the Order Service API. When a user clicks a button, the request flows from Angular to the Order Service, which calls the Product Service. Later, you'll see that entire chain as a distributed trace in Grafana.

In this step, get ready to:
  • Generate a new Angular application with the Angular CLI.
  • Create an HTTP service and UI component that calls the Order Service.
  • Containerize the Angular app with Docker and Nginx.
Generate the Angular application

The Angular CLI scaffolds a complete project structure with routing, build configuration, and a starter component. You'll generate a new app called frontend alongside your existing services.

  • In your terminal, make sure you're in the project root folder (the parent of product-service/ and order-service/).
  • Generate the Angular app by running this command:
ng new frontend --style=css --ssr=false --file-name-style-guide=2016

What do these flags mean?

  • --style=css tells the CLI to use plain CSS for stylesheets.
  • --ssr=false disables server-side rendering. You're building a simple client-side SPA served by Nginx.
  • --file-name-style-guide=2016 uses the familiar app.component.ts naming convention instead of the newer concise format.
  • The CLI will install dependencies automatically. Wait for it to finish.
  • Move into the new project folder by running:
cd frontend
  • Confirm the project was generated by running:
ls src/app

You should see files like app.component.ts, app.config.ts, and app.routes.ts listed in the output.

Command not found?

Make sure you installed the Angular CLI globally in Step 1. Run npm install -g @angular/cli and try again.

help me troubleshoot the ng new command failing

Add the HTTP service and update the component

Angular uses services to keep HTTP logic separate from components. You'll create a service that calls the Order Service's REST API, then update the app component to use it.

First, enable HttpClient in the application config so Angular can make HTTP requests.

  • Open src/app/app.config.ts in your editor.
  • Find this line:
import { provideRouter } from '@angular/router';
  • Add the HttpClient import right below it, so the top of the file looks like this:
import { provideRouter } from '@angular/router';
import { provideHttpClient } from '@angular/common/http';
  • In the providers array, add provideHttpClient() after the existing providers. The full file should look like this:
import { ApplicationConfig, provideZoneChangeDetection } from '@angular/core';
import { provideRouter } from '@angular/router';
import { provideHttpClient } from '@angular/common/http';

import { routes } from './app.routes';

export const appConfig: ApplicationConfig = {
  providers: [
    provideZoneChangeDetection({ eventCoalescing: true }),
    provideRouter(routes),
    provideHttpClient()
  ]
};

What does provideHttpClient do?

provideHttpClient() registers Angular's HttpClient service for dependency injection. Without this, any attempt to make HTTP requests would throw a missing-provider error at runtime.

  • Save app.config.ts.

Now create the order service that handles API calls.

  • Create a new file at src/app/order.service.ts and add the following code:
import { Injectable } from '@angular/core';
import { HttpClient } from '@angular/common/http';
import { Observable } from 'rxjs';

@Injectable({
  providedIn: 'root'
})
export class OrderService {
  // Base URL for order-related API calls
  private apiUrl = '/api/orders';

  constructor(private http: HttpClient) {}

  // Fetch all available products from the Order Service
  getProducts(): Observable<any[]> {
    return this.http.get<any[]>(`${this.apiUrl}/products`);
  }

  // Create a new order for a given product ID
  createOrder(productId: number): Observable<any> {
    return this.http.post<any>(this.apiUrl, { productId });
  }
}

What does this service do?

The service makes two HTTP calls. getProducts() hits GET /api/orders/products which the Order Service proxies to the Product Service. createOrder() hits POST /api/orders which triggers the full cross-service flow.

The relative URL /api/orders works because Nginx will proxy these requests to the Order Service container at runtime.

  • Save order.service.ts.

Next, update the app component to display products and create orders.

  • Open src/app/app.component.ts and replace its entire contents with:
import { Component, OnInit } from '@angular/core';
import { JsonPipe } from '@angular/common';
import { OrderService } from './order.service';

@Component({
  selector: 'app-root',
  imports: [JsonPipe],
  templateUrl: './app.component.html',
  styleUrl: './app.component.css'
})
export class AppComponent implements OnInit {
  // List of available products fetched from the backend
  products: any[] = [];
  // Stores the most recent order response
  orderResponse: any = null;

  constructor(private orderService: OrderService) {}

  ngOnInit() {
    // Load products when the component initializes
    this.orderService.getProducts().subscribe(products => {
      this.products = products;
    });
  }

  // Send a create-order request for the given product
  createOrder(productId: number) {
    this.orderService.createOrder(productId).subscribe(response => {
      this.orderResponse = response;
    });
  }
}
  • Open src/app/app.component.html and replace its entire contents with:
<div class="container">
  <h1>Microservices Order System</h1>

  <h2>Products</h2>
  <div class="products">
    @for (product of products; track product.id) {
      <div class="product">
        <h3>{{ product.name }}</h3>
        <p>${{ product.price }}</p>
        <button (click)="createOrder(product.id)">Order</button>
      </div>
    }
  </div>

  @if (orderResponse) {
    <div class="response">
      <h2>Order Created</h2>
      <pre>{{ orderResponse | json }}</pre>
    </div>
  }
</div>

What does this component do?

On load, ngOnInit fetches products from the backend and displays them as cards. Each card has an **Order** button that sends a POST to create an order. The response JSON is displayed below the product list.

The template uses Angular's built-in @for and @if control flow syntax. The json pipe formats the response object as readable text.

  • Save both app.component.ts and app.component.html.
  • Verify the code compiles by running:
npx ng build

You should see a successful build with output files listed. The compiled app is now in dist/frontend/browser/.

Build errors?

If you see errors about HttpClient or missing providers, double-check that provideHttpClient() is in the providers array of app.config.ts. Make sure the import line matches exactly: import { provideHttpClient } from '@angular/common/http';.

help me fix Angular build errors in my frontend project

Create the Dockerfile and Nginx configuration

The Angular app needs to be served by a web server inside Docker. You'll use Nginx to serve the static files and proxy API requests to the Order Service container.

  • Create a new file called nginx.conf in the frontend/ folder (the same level as package.json) and add:
server {
    listen 80;
    server_name localhost;

    location / {
        root /usr/share/nginx/html;
        index index.html;
        try_files $uri $uri/ /index.html;
    }

    location /api/ {
        proxy_pass http://order-service:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

What does this Nginx config do?

The first location / block serves the Angular app's static files. The try_files directive falls back to index.html for any route Angular handles client-side.

The second location /api/ block forwards any request starting with /api/ to the Order Service container. Docker Compose networking resolves order-service to the correct container IP automatically.

  • Save nginx.conf.
  • Create a new file called Dockerfile in the frontend/ folder and add:
FROM node:20 AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npx ng build --configuration=production

FROM nginx:alpine
COPY --from=build /app/dist/frontend/browser /usr/share/nginx/html
COPY nginx.conf /etc/nginx/conf.d/default.conf
EXPOSE 80

What does this Dockerfile do?

This is a multi-stage build. The first stage uses a Node.js image to install dependencies and compile the Angular app into static HTML, CSS, and JavaScript files.

The second stage uses a lightweight Nginx image to serve those files. It copies the compiled output from dist/frontend/browser into Nginx's web root and applies the custom nginx.conf for API proxying.

  • Save Dockerfile.
  • Confirm both files exist by running:
ls Dockerfile nginx.conf

You should see both files listed in the output.

Files not showing up?

Make sure you're inside the frontend/ directory when running ls. Both Dockerfile and nginx.conf must be at the same level as package.json.

help me find where my Dockerfile should go in the Angular project

✔️ Awesome, I've got everything!

Great. Double check you've saved all your files before moving on.

ⓧ I'd like to double check the full code

Here's the full frontend/Dockerfile:

FROM node:20 AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npx ng build --configuration=production

FROM nginx:alpine
COPY --from=build /app/dist/frontend/browser /usr/share/nginx/html
COPY nginx.conf /etc/nginx/conf.d/default.conf
EXPOSE 80

And here's the full frontend/nginx.conf:

server {
    listen 80;
    server_name localhost;

    location / {
        root /usr/share/nginx/html;
        index index.html;
        try_files $uri $uri/ /index.html;
    }

    location /api/ {
        proxy_pass http://order-service:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

Your Angular frontend is built and containerized. In the next step, you'll wire everything together with Docker Compose and the full observability stack.

Set Up the Observability Stack

You have two microservices and a frontend, each with their own Dockerfile. But running them individually with docker build and docker run for each container would be tedious and error-prone.

Docker Compose lets you define all your services in one file and start them together with a single command. In this step, you'll create that Compose file along with configuration for Grafana, Tempo, Loki, and Prometheus. These four tools form your observability stack.

In this step, get ready to:
  • Create a Docker Compose file orchestrating 7 services.
  • Configure Tempo and Prometheus with their own config files.
  • Provision Grafana data sources with trace-to-log correlation.
Create the Docker Compose file

The Docker Compose file defines every container in your system, the ports they expose, their dependencies, and how they connect via Docker networking.

  • Create a file called docker-compose.yml in your project root (the same level as product-service/, order-service/, and frontend/).
  • Paste the following content into docker-compose.yml:
services:
  product-service:
    build: ./product-service
    ports:
      - "8081:8081"
    depends_on:
      - tempo
      - loki

  order-service:
    build: ./order-service
    ports:
      - "8080:8080"
    depends_on:
      - product-service
      - tempo
      - loki

  frontend:
    build: ./frontend
    ports:
      - "4200:80"
    depends_on:
      - order-service

  tempo:
    image: grafana/tempo:2.10.4
    command: ["-config.file=/etc/tempo/tempo.yaml"]
    volumes:
      - ./config/tempo.yaml:/etc/tempo/tempo.yaml
    ports:
      - "3200:3200"
      - "4318:4318"

  loki:
    image: grafana/loki:2.9.0
    ports:
      - "3100:3100"

  prometheus:
    image: prom/prometheus:v2.50.0
    volumes:
      - ./config/prometheus.yaml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana:10.4.0
    environment:
      GF_AUTH_ANONYMOUS_ENABLED: "true"
      GF_AUTH_ANONYMOUS_ORG_ROLE: "Admin"
    volumes:
      - ./config/grafana/datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
    ports:
      - "3000:3000"
    depends_on:
      - tempo
      - loki
      - prometheus

What does this file do?

This Compose file defines 7 services that form your entire system:

  • product-service and order-service build from your local Dockerfiles. They depend on Tempo and Loki being available first.
  • frontend builds your Angular app and serves it via Nginx on port 4200.
  • tempo receives distributed traces via OpenTelemetry protocol (OTLP) on port 4318.
  • loki stores structured logs pushed by your Spring Boot services.
  • prometheus scrapes metrics from your services' /actuator/prometheus endpoints.
  • grafana provides the unified dashboard where you view traces, logs, and metrics together.

The depends_on field controls startup order. It ensures Tempo and Loki are running before your microservices try to send data.

  • Save docker-compose.yml.
  • Confirm the file exists by running:
ls docker-compose.yml

You should see docker-compose.yml listed in the output.

File not showing up?

Make sure you created the file in the project root directory, not inside one of the service folders. Run ls and confirm you can see product-service/, order-service/, and frontend/ at the same level.

Help me debug my docker-compose.yml file location

✔️ Awesome, I've got everything!

Great. Make sure docker-compose.yml is saved before moving on.

ⓧ I'd like to double check the full code

services:
  product-service:
    build: ./product-service
    ports:
      - "8081:8081"
    depends_on:
      - tempo
      - loki

  order-service:
    build: ./order-service
    ports:
      - "8080:8080"
    depends_on:
      - product-service
      - tempo
      - loki

  frontend:
    build: ./frontend
    ports:
      - "4200:80"
    depends_on:
      - order-service

  tempo:
    image: grafana/tempo:2.10.4
    command: ["-config.file=/etc/tempo/tempo.yaml"]
    volumes:
      - ./config/tempo.yaml:/etc/tempo/tempo.yaml
    ports:
      - "3200:3200"
      - "4318:4318"

  loki:
    image: grafana/loki:2.9.0
    ports:
      - "3100:3100"

  prometheus:
    image: prom/prometheus:v2.50.0
    volumes:
      - ./config/prometheus.yaml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana:10.4.0
    environment:
      GF_AUTH_ANONYMOUS_ENABLED: "true"
      GF_AUTH_ANONYMOUS_ORG_ROLE: "Admin"
    volumes:
      - ./config/grafana/datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
    ports:
      - "3000:3000"
    depends_on:
      - tempo
      - loki
      - prometheus
Create configuration files for Tempo and Prometheus

Your Docker Compose file references config files that don't exist yet. Tempo needs a config to know how to receive traces. Prometheus needs a config to know which services to scrape for metrics.

  • Create the config directory inside your project root by running:
mkdir -p config
  • Create a file called tempo.yaml inside the config/ folder with the following content:
server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: "0.0.0.0:4317"
        http:
          endpoint: "0.0.0.0:4318"

storage:
  trace:
    backend: local
    local:
      path: /tmp/tempo/blocks

What does this config do?

This tells Tempo to listen for incoming traces on two ports. Port 4317 handles gRPC connections and port 4318 handles HTTP connections. Your Spring Boot services send traces to the HTTP endpoint.

Traces are stored locally at /tmp/tempo/blocks inside the container. This is fine for local development since traces don't need to survive container restarts.

  • Save config/tempo.yaml.
  • Now create a file called prometheus.yaml inside the same config/ folder with the following content:
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "order-service"
    metrics_path: "/actuator/prometheus"
    static_configs:
      - targets: ["order-service:8080"]

  - job_name: "product-service"
    metrics_path: "/actuator/prometheus"
    static_configs:
      - targets: ["product-service:8081"]

What does this config do?

Prometheus pulls (scrapes) metrics from your services every 15 seconds. Each job_name defines a scrape target.

The metrics_path points to /actuator/prometheus. This is the endpoint Spring Boot Actuator exposes automatically when you include the micrometer-registry-prometheus dependency. Prometheus uses Docker networking to reach each service by its container name.

  • Save config/prometheus.yaml.
  • Confirm both config files exist by running:
ls config/

You should see tempo.yaml and prometheus.yaml listed.

Only seeing one file?

Double-check that both files are saved inside the config/ directory, not in the project root. The filenames must be exactly tempo.yaml and prometheus.yaml (not .yml).

Help me find my missing config file

✔️ Awesome, I've got everything!

Great. Both config files are ready for Tempo and Prometheus.

ⓧ I'd like to double check the full code

server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: "0.0.0.0:4317"
        http:
          endpoint: "0.0.0.0:4318"

storage:
  trace:
    backend: local
    local:
      path: /tmp/tempo/blocks
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "order-service"
    metrics_path: "/actuator/prometheus"
    static_configs:
      - targets: ["order-service:8080"]

  - job_name: "product-service"
    metrics_path: "/actuator/prometheus"
    static_configs:
      - targets: ["product-service:8081"]
Create the Grafana data source provisioning file

Grafana needs to know where to find Tempo, Loki, and Prometheus. Instead of configuring these manually through the UI every time, you'll use a provisioning file that auto-configures all three data sources on startup.

This file also sets up the trace-to-log correlation. When you view a trace in Tempo, Grafana will automatically link it to the matching logs in Loki.

  • Create the config/grafana directory by running:
mkdir -p config/grafana
  • Create a file called datasources.yaml inside config/grafana/ with the following content:
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    isDefault: true

  - name: Tempo
    type: tempo
    url: http://tempo:3200
    jsonData:
      tracesToLogsV2:
        datasourceUid: loki
        spanStartTimeShift: "-2s"
        spanEndTimeShift: "2s"
        filterByTraceID: true
        tags:
          - key: "service.name"
            value: "service_name"

  - name: Loki
    type: loki
    uid: loki
    url: http://loki:3100
    jsonData:
      derivedFields:
        - datasourceUid: tempo
          matcherRegex: '"traceId":"([a-f0-9]+)"'
          name: traceId
          url: "$${__value.raw}"

What does this config do?

This provisioning file registers three data sources with Grafana automatically:

  • Prometheus is set as the default data source for metrics queries.
  • Tempo includes a tracesToLogsV2 block. This tells Grafana: "when viewing a trace, find matching logs in Loki by filtering on trace ID and the service_name label."
  • Loki includes a derivedFields block. This extracts trace IDs from log lines using a regex and creates clickable links that jump directly to the matching trace in Tempo.

Together, these two correlation settings let you jump from traces to logs and from logs to traces in a single click.

  • Save config/grafana/datasources.yaml.
  • Confirm the file exists by running:
ls config/grafana/

You should see datasources.yaml listed.

Directory not found?

Make sure you ran mkdir -p config/grafana before creating the file. The -p flag creates parent directories if they don't exist.

Help me set up my Grafana config directory

✔️ Awesome, I've got everything!

Great. Your Grafana provisioning is ready to auto-configure trace-to-log correlation on startup.

ⓧ I'd like to double check the full code

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    isDefault: true

  - name: Tempo
    type: tempo
    url: http://tempo:3200
    jsonData:
      tracesToLogsV2:
        datasourceUid: loki
        spanStartTimeShift: "-2s"
        spanEndTimeShift: "2s"
        filterByTraceID: true
        tags:
          - key: "service.name"
            value: "service_name"

  - name: Loki
    type: loki
    uid: loki
    url: http://loki:3100
    jsonData:
      derivedFields:
        - datasourceUid: tempo
          matcherRegex: '"traceId":"([a-f0-9]+)"'
          name: traceId
          url: "$${__value.raw}"

Your entire observability stack is configured and ready to launch. Next up, you'll start all 7 services with a single command and explore distributed traces flowing through your system.

Run Everything and Explore Distributed Traces

You've built two microservices, an Angular frontend, and a full observability stack. Every piece is configured and ready to go.

Now it's time to bring all 7 services to life with a single command. You'll create orders in the browser, then watch the request flow across services as a distributed trace inside Grafana.

In this step, get ready to:
  • Start the entire Docker Compose stack and verify all services are healthy.
  • Create orders in the Angular frontend and view distributed traces in Grafana Tempo.
  • Explore trace-to-log correlation by jumping from a trace in Tempo to matching logs in Loki.
Start the Docker Compose stack

Docker Compose reads your docker-compose.yml file and builds images for your custom services, pulls images for the observability tools, then starts all 7 containers on a shared network.

  • Switch back to your terminal. Make sure you're inside the project root folder (the one containing docker-compose.yml).
  • Build and start all services by running this command:
docker compose up --build

What does this command do?

The --build flag tells Docker to build fresh images for your custom services (product-service, order-service, frontend) before starting them. The observability tools (Grafana, Tempo, Loki, Prometheus) use pre-built images that Docker pulls automatically.

The first build takes several minutes because Docker downloads base images and Maven dependencies. Subsequent runs are much faster thanks to Docker's layer caching.

You'll see logs streaming from all 7 services. Wait until you see Spring Boot startup messages from both services. Look for lines like Started ProductServiceApplication and Started OrderServiceApplication.

  • Open a new terminal tab or window. Verify all containers are running by running:
docker compose ps

You should see all 7 services listed with a status of Up. Each service maps to a port on your machine: frontend on 4200, order-service on 8080, product-service on 8081, Grafana on 3000, Tempo on 3200, Loki on 3100, and Prometheus on 9090.

Services not starting?

If a service shows as Exited, check its logs for the specific error. Run docker compose logs <service-name> to see what went wrong.

  • Make sure Docker Desktop is running. If it's not, start it before retrying.
  • If you see a Maven build failure, double check your pom.xml files for typos in dependency coordinates.
  • If a port is already in use, stop whatever is using it or change the host port in docker-compose.yml.

Help me debug my Docker Compose startup errors

Create orders and view traces in Grafana Tempo

With all services running, every HTTP request that hits your microservices generates a distributed trace. The trace captures timing data as the request flows from the Angular frontend through the Order Service and into the Product Service.

  • Open your browser and navigate to http://localhost:4200 to load the Angular frontend.
  • Click the button to create an order. Do this 3-4 times to generate multiple traces.

What happens when you click the button?

The Angular app sends a POST request to /api/orders. Nginx proxies it to the Order Service. The Order Service then calls the Product Service to fetch product details. Each hop generates a span, and together they form one distributed trace.

  • Open a new browser tab and navigate to http://localhost:3000 to open Grafana.
  • In the left sidebar, click the hamburger menu icon (three horizontal lines).
  • Click Explore.
  • In the data source dropdown at the top, select Tempo.
  • Change the query type to Search and click Run query.

You should see a list of recent traces. Each trace represents one end-to-end request that flowed through your microservices.

  • Click on any trace in the list to open its detail view.

You should see a trace waterfall showing spans from both order-service and product-service. The waterfall shows how long each service spent processing the request.

No traces showing up?

  • Make sure you created at least one order in the Angular frontend first. Traces only appear after requests are made.
  • Check the time range in Grafana. Click the time picker in the top-right corner and select Last 15 minutes.
  • Verify Tempo is receiving traces by checking its logs with docker compose logs tempo.

Help me figure out why my traces aren't appearing in Grafana Tempo

Explore trace-to-log correlation

Seeing traces is powerful, but the real value comes from correlating traces with logs. When something goes wrong in production, you need to jump from a slow trace directly to the logs that explain why it was slow.

  • With a trace open in Tempo, look for a Logs for this span link on any span in the waterfall. Click it.

Grafana navigates to Loki and automatically filters logs by the trace ID. You see only the logs that belong to that specific request.

How does trace-to-log correlation work?

Your datasources.yaml configured two connection points. The Tempo data source has a tracesToLogsV2 block that tells Grafana how to link from a trace span to Loki logs. The Loki data source has a derivedFields block that extracts trace IDs from log lines and links back to Tempo.

This bi-directional linking means you can start from either side. Find a trace, jump to its logs. Or find a log, jump to its full trace.

  • Now test the reverse direction. In the data source dropdown, switch to Loki.
  • In the label filter, select service_name and choose order-service. Click Run query.
  • Expand any log line. You should see a traceId field that's a clickable link. Click it to jump back to the full trace in Tempo.

What should the trace waterfall tell you?

  • The top-level span shows the total request duration in the Order Service.
  • A child span shows the HTTP call from Order Service to Product Service. The gap between spans reveals network latency.
  • Both spans share the same trace ID. This is how OpenTelemetry connects work across service boundaries.

Trace-to-log link not appearing?

  • Check that your config/grafana/datasources.yaml has the tracesToLogsV2 block under the Tempo data source and derivedFields under the Loki data source.
  • If you changed the datasources config after starting Grafana, you need to restart the Grafana container. Run docker compose restart grafana.
  • Make sure the Loki data source has uid: loki set, because the Tempo datasourceUid references it.

Help me fix my trace-to-log correlation in Grafana

Secret mission

RED Metrics Dashboard

Production teams don't just trace individual requests. They monitor the overall health of their services using RED metrics: Rate (requests per second), Errors (percentage of failed requests), and Duration (how long requests take). Build a Grafana dashboard that visualizes these metrics for your microservices, then add a Prometheus alerting rule that fires when things go wrong.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely on your local machine using Docker containers. There are no cloud resources or ongoing costs to worry about.

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • 7 Docker containers (product-service, order-service, frontend, grafana, loki, tempo, prometheus).
  • Docker images built for product-service, order-service, and frontend.
  • Project folder on your Desktop containing all source code and configuration files.
  • Angular CLI installed globally via npm.
  • Docker Desktop application.

Keep everything running

No action needed. Choose this if you're still actively experimenting with your observability stack or want to try more requests in Grafana.

  • Your 7 Docker containers will continue running and consuming system memory.
  • Docker Desktop will remain installed on your machine.
  • Angular CLI will remain available globally for future Angular projects.

Pause - I'll come back to this later

Stop all running containers to free up memory, but keep your images and project files intact so you can restart quickly later.

  • Open the terminal that has your Docker Compose stack running.
  • Press Ctrl+C to stop all containers.
  • To restart later, navigate to the folder containing your docker-compose.yml and run:
docker compose up --build

Delete - I don't want to use this again

Remove all Docker containers, images, and project files from your machine.

  • Navigate to the folder containing your docker-compose.yml file in your terminal.
  • Stop and remove all containers, images, and volumes by running:
docker compose down --rmi all --volumes
  • Delete the project folder by running:

macOS

cd ~/Desktop
rm -rf microservices-observability

Windows

cd ~\Desktop
rmdir /s /q microservices-observability
  • Uninstall Angular CLI globally by running:
npm uninstall -g @angular/cli
  • To uninstall Docker Desktop, open Settings on your machine.
  • On Windows, go to Apps, then Apps & features.
  • Find Docker Desktop in the list and click Uninstall.
  • On macOS, open Finder, go to Applications, and drag Docker to the Trash.

Nice Work!

Nice Work!

Nice work! You just built a production-grade microservices observability stack from scratch.

You've learned how to:

  • Build two Spring Boot microservices with OpenTelemetry instrumentation and containerize them using Docker Compose.
  • Set up Grafana Tempo for distributed tracing and Grafana Loki for structured log aggregation with trace-to-log correlation so you can jump between traces and logs in one click.
  • Build an Angular frontend that triggers cross-service requests and trace the full request flow end-to-end in Grafana.
  • Secret Mission: Added RED metrics dashboards and Prometheus alerting rules to monitor request rate, error rate, and duration across your microservices.

Ready to quiz yourself?