AI-Powered On-Call SRE Agent

Introduction

30 Second Summary

When something breaks in production at 3am, someone has to wake up, open multiple dashboards, and piece together what went wrong. That first 15 minutes of scrambling between tools is where most of the stress lives.

In this project, you will build an AI-powered On-Call SRE Agent that automates incident investigation for any microservice in your cluster. You hand it a service name, and it queries metrics and logs to produce a structured triage report with severity, root cause, and remediation steps.

What You'll Build

A Spring Boot application using Spring AI tool calling to query Prometheus metrics and Loki logs, deployed on local Kubernetes via ArgoCD GitOps.

By the end of this project, you'll have:

• A service-agnostic AI agent that investigates any microservice by dynamically querying Prometheus metrics and Loki logs using standard observability conventions.
• A full observability stack (Prometheus, Grafana, Loki) deployed on local Kubernetes via ArgoCD GitOps workflow.
• An investigation dashboard where you type a service name and receive a structured triage report with severity assessment, root cause analysis, and remediation steps.
• Secret Mission: An Alertmanager webhook integration that automatically triggers the agent when a Prometheus alert fires for any monitored service.

Are there any prerequisites?

This project assumes you have experience with Java and Spring Boot, and basic familiarity with Docker and Kubernetes concepts. You will need Java 21, Docker Desktop with Kubernetes enabled, and an OpenRouter account with an API key.

Set Up the Multi-Module Maven Project

Your SRE Agent and the services it monitors are fundamentally different concerns. The agent handles AI-powered investigation. The demo service generates realistic metrics and logs for the agent to query.

A multi-module Maven project lets you build, test, and deploy each piece independently while sharing common configuration through a parent POM. In this step, you will scaffold the entire project structure including dependency management, Spring AI configuration, and the hexagonal architecture packages.

In this step, get ready to:

• Initialize the multi-module Maven project with parent and child POMs.
• Configure Spring AI with OpenRouter for LLM access.
• Set up hexagonal architecture package structure in the SRE Agent.

Initialize the multi-module Maven project

A multi-module Maven project uses a parent POM to declare shared properties (like Java version and dependency versions) and child modules that inherit them. This means you configure Spring AI and Spring Boot versions once, and both modules pick them up automatically.

• Create the directory structure for both modules by running these commands:

mkdir -p sre-agent/src/main/java/com/sre/agent/domain
mkdir -p sre-agent/src/main/java/com/sre/agent/application
mkdir -p sre-agent/src/main/java/com/sre/agent/infrastructure
mkdir -p sre-agent/src/main/resources/static
mkdir -p sre-agent/src/test/java/com/sre/agent
mkdir -p demo-service/src/main/java/com/sre/demo
mkdir -p demo-service/src/main/resources
mkdir -p demo-service/src/test/java/com/sre/demo

• Confirm the directories exist by running:

ls

You should see demo-service and sre-agent listed in the output.

• Create the parent pom.xml in your project root with the following content:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.4.3</version>
        <relativePath/>
    </parent>

    <groupId>com.sre</groupId>
    <artifactId>ai-sre-agent</artifactId>
    <version>0.1.0</version>
    <packaging>pom</packaging>
    <name>AI SRE Agent</name>

    <modules>
        <module>demo-service</module>
        <module>sre-agent</module>
    </modules>

    <properties>
        <java.version>21</java.version>
        <spring-ai.version>1.0.8</spring-ai.version>
    </properties>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.ai</groupId>
                <artifactId>spring-ai-bom</artifactId>
                <version>${spring-ai.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
</project>

What does this POM do?

The parent POM inherits from spring-boot-starter-parent version 3.4.3, which gives all child modules access to Spring Boot's dependency management.

The <dependencyManagement> section imports the Spring AI BOM at version 1.0.8. This means child modules can add Spring AI dependencies without specifying versions.

The <packaging>pom</packaging> declaration tells Maven this is a parent-only project that does not produce a JAR itself.

• Save pom.xml.

• Create the SRE Agent module's sre-agent/pom.xml with the following content:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>com.sre</groupId>
        <artifactId>ai-sre-agent</artifactId>
        <version>0.1.0</version>
    </parent>

    <artifactId>sre-agent</artifactId>
    <name>SRE Agent Service</name>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.ai</groupId>
            <artifactId>spring-ai-starter-model-openai</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>
</project>

What does this module include?

The spring-ai-starter-model-openai dependency brings in Spring AI's OpenAI-compatible chat client. Since OpenRouter exposes an OpenAI-compatible API, this starter works perfectly with a base-url override.

No version numbers are needed on any dependency because the parent POM's BOM handles version resolution automatically.

• Save sre-agent/pom.xml.

• Create the demo service module's demo-service/pom.xml with the following content:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>com.sre</groupId>
        <artifactId>ai-sre-agent</artifactId>
        <version>0.1.0</version>
    </parent>

    <artifactId>demo-service</artifactId>
    <name>Demo Service</name>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-registry-prometheus</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>
</project>

What does this module include?

The micrometer-registry-prometheus dependency exposes metrics in Prometheus format at /actuator/prometheus. This is what makes the demo service discoverable by the SRE Agent's Prometheus queries.

• Save demo-service/pom.xml.
• Validate the multi-module project compiles by running:

mvn validate

You should see BUILD SUCCESS in the terminal output confirming all three POMs are valid.

Seeing BUILD FAILURE?

Check that your parent pom.xml is in the project root, not inside a subdirectory. The <modules> list must match the exact directory names demo-service and sre-agent.

Also confirm the child POMs reference the correct parent <groupId>com.sre</groupId> and <version>0.1.0</version>.

help me debug a Maven multi-module BUILD FAILURE

Configure Spring AI with OpenRouter

Spring AI's OpenAI starter connects to the standard OpenAI endpoint by default. By overriding the base-url property to point at OpenRouter, you get access to multiple LLM providers through a single API.

• Create sre-agent/src/main/resources/application.yml with the following configuration:

spring:
  application:
    name: sre-agent
  ai:
    openai:
      base-url: https://openrouter.ai/api/v1
      api-key: ${OPENROUTER_API_KEY}
      chat:
        options:
          model: anthropic/claude-sonnet-4
          temperature: 0.1
          max-tokens: 4096

server:
  port: 8081

sre:
  prometheus:
    base-url: http://prometheus:9090
  loki:
    base-url: http://loki:3100

What does this configuration do?

• spring.ai.openai.base-url overrides the default OpenAI endpoint to use OpenRouter's OpenAI-compatible API at https://openrouter.ai/api/v1.
• ${OPENROUTER_API_KEY} reads your API key from an environment variable at runtime. Your key never appears in code.
• temperature: 0.1 keeps responses deterministic and factual, which is critical for incident triage.
• The sre.prometheus and sre.loki properties define where the agent finds the observability stack inside Kubernetes.

• Save sre-agent/src/main/resources/application.yml.

• Create demo-service/src/main/resources/application.yml with the following content:

spring:
  application:
    name: demo-service

server:
  port: 8080

management:
  endpoints:
    web:
      exposure:
        include: health,prometheus,metrics
  metrics:
    tags:
      application: ${spring.application.name}
      app: ${spring.application.name}

Why these metric tags?

The app tag is the key that makes this service discoverable by the SRE Agent. When the agent constructs PromQL queries like {app="demo-service"}, Prometheus can match metrics from this service because the tag is set automatically on every metric.

• Save demo-service/src/main/resources/application.yml.

YAML indentation errors later?

YAML is sensitive to indentation. Make sure you use spaces (not tabs) and that nested keys are indented exactly 2 or 4 spaces consistently. If your application fails to start later, the configuration file is the first place to check.

help me fix a YAML parsing error in my Spring Boot application.yml

Set up the hexagonal architecture and entry points

Hexagonal architecture separates your code into three layers: domain (business logic and models), application (orchestration), and infrastructure (external integrations like REST controllers and tool implementations). The directory structure you created earlier already reflects these layers.

• Create sre-agent/src/main/java/com/sre/agent/SreAgentApplication.java with the following content:

package com.sre.agent;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class SreAgentApplication {
    public static void main(String[] args) {
        SpringApplication.run(SreAgentApplication.class, args);
    }
}

What does this class do?

This is the Spring Boot entry point for the SRE Agent module. The @SpringBootApplication annotation enables auto-configuration, component scanning, and Spring Boot's opinionated defaults.

• Save SreAgentApplication.java.
• Create sre-agent/src/main/java/com/sre/agent/domain/TriageReport.java with the following content:

package com.sre.agent.domain;

import java.time.Instant;
import java.util.List;

public record TriageReport(
        String serviceName,
        String severity,
        String summary,
        String rootCause,
        List<String> evidence,
        List<String> remediationSteps,
        Instant timestamp
) {}

Why a Java record in the domain layer?

The TriageReport record defines the shape of every investigation result. Spring AI's structured output feature will parse the LLM's response directly into this record, so the field names guide what the model returns.

Placing it in the domain package means it has zero dependencies on infrastructure or framework code. Both the application layer and the REST controller can reference it without circular imports.

• Save TriageReport.java.
• Create demo-service/src/main/java/com/sre/demo/DemoServiceApplication.java with the following content:

package com.sre.demo;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class DemoServiceApplication {
    public static void main(String[] args) {
        SpringApplication.run(DemoServiceApplication.class, args);
    }
}

What does this class do?

This is the entry point for the demo target service. It runs on port 8080 (configured in its application.yml) and will later expose simulation endpoints that generate realistic metrics and error logs for the SRE Agent to investigate.

• Save DemoServiceApplication.java.
• Verify the entire project compiles by running:

mvn compile

You should see BUILD SUCCESS with both modules listed in the reactor summary.

Compilation failing?

If you see a package resolution error for spring-ai-starter-model-openai, make sure your parent POM's <dependencyManagement> section includes the Spring AI BOM with version 1.0.8.

If you see "cannot find symbol" for the record keyword, confirm your <java.version> property is set to 21 in the parent POM.

help me fix a Maven compilation error in my Spring AI multi-module project

✔️ Awesome, I've got everything!

Great. Double check you have saved all your files before moving on.

ⓧ I'd like to double check the full code

Here is the complete code for each file at this point.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.4.3</version>
        <relativePath/>
    </parent>

    <groupId>com.sre</groupId>
    <artifactId>ai-sre-agent</artifactId>
    <version>0.1.0</version>
    <packaging>pom</packaging>
    <name>AI SRE Agent</name>

    <modules>
        <module>demo-service</module>
        <module>sre-agent</module>
    </modules>

    <properties>
        <java.version>21</java.version>
        <spring-ai.version>1.0.8</spring-ai.version>
    </properties>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.ai</groupId>
                <artifactId>spring-ai-bom</artifactId>
                <version>${spring-ai.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
</project>

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>com.sre</groupId>
        <artifactId>ai-sre-agent</artifactId>
        <version>0.1.0</version>
    </parent>

    <artifactId>sre-agent</artifactId>
    <name>SRE Agent Service</name>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.ai</groupId>
            <artifactId>spring-ai-starter-model-openai</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>
</project>

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>com.sre</groupId>
        <artifactId>ai-sre-agent</artifactId>
        <version>0.1.0</version>
    </parent>

    <artifactId>demo-service</artifactId>
    <name>Demo Service</name>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-registry-prometheus</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>
</project>

spring:
  application:
    name: sre-agent
  ai:
    openai:
      base-url: https://openrouter.ai/api/v1
      api-key: ${OPENROUTER_API_KEY}
      chat:
        options:
          model: anthropic/claude-sonnet-4
          temperature: 0.1
          max-tokens: 4096

server:
  port: 8081

sre:
  prometheus:
    base-url: http://prometheus:9090
  loki:
    base-url: http://loki:3100

spring:
  application:
    name: demo-service

server:
  port: 8080

management:
  endpoints:
    web:
      exposure:
        include: health,prometheus,metrics
  metrics:
    tags:
      application: ${spring.application.name}
      app: ${spring.application.name}

package com.sre.agent;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class SreAgentApplication {
    public static void main(String[] args) {
        SpringApplication.run(SreAgentApplication.class, args);
    }
}

package com.sre.agent.domain;

import java.time.Instant;
import java.util.List;

public record TriageReport(
        String serviceName,
        String severity,
        String summary,
        String rootCause,
        List<String> evidence,
        List<String> remediationSteps,
        Instant timestamp
) {}

package com.sre.demo;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class DemoServiceApplication {
    public static void main(String[] args) {
        SpringApplication.run(DemoServiceApplication.class, args);
    }
}

Your project skeleton is ready with both modules compiling cleanly. Next up, you will build the SRE Agent's service-agnostic tools that query Prometheus and Loki using standard observability conventions.

Build the Service-Agnostic SRE Agent Tools

Your project skeleton is in place with hexagonal architecture layers ready to fill. Now you need to give the AI agent the ability to see what's happening inside a running microservice.

In an SRE workflow, investigation starts with two data sources: metrics ("how fast is it failing?") and logs ("what errors are being thrown?"). You'll build two tool classes that the LLM can call to query Prometheus for metrics and Loki for logs. The key design decision: neither tool knows anything about a specific service. The LLM constructs the right queries dynamically based on whatever service name it receives.

In this step, get ready to:

• Implement the PrometheusQueryTool with Spring AI's @Tool annotation for querying metrics.
• Implement the LokiLogSearchTool for searching application logs via LogQL.

Implement the Prometheus query tool

Prometheus exposes an HTTP API that accepts PromQL queries and returns metric data as JSON. Your tool wraps this API so the LLM can request any metric for any service by constructing the appropriate PromQL query itself.

• Create a new file at sre-agent/src/main/java/com/sre/agent/infrastructure/tools/PrometheusQueryTool.java.
• Add the following code:

package com.sre.agent.infrastructure.tools;

import org.springframework.ai.tool.annotation.Tool;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;
import org.springframework.web.client.RestClient;

@Component
public class PrometheusQueryTool {

    private final RestClient restClient;

    public PrometheusQueryTool(@Value("${sre.prometheus.base-url}") String prometheusUrl) {
        this.restClient = RestClient.builder()
                .baseUrl(prometheusUrl)
                .build();
    }

    @Tool(description = "Query Prometheus for current metric values using PromQL. Use standard label selectors like {job=\"service-name\"} or {app=\"service-name\"}. Common queries: rate(http_server_requests_seconds_count{app=\"X\",status=~\"5..\"}[5m]) for error rate, histogram_quantile(0.99, rate(http_server_requests_seconds_bucket{app=\"X\"}[5m])) for p99 latency, process_cpu_seconds_total for CPU, jvm_memory_used_bytes for memory.")
    public String queryMetrics(String promqlQuery) {
        String response = restClient.get()
                .uri("/api/v1/query?query={query}", promqlQuery)
                .retrieve()
                .body(String.class);
        return response;
    }

    @Tool(description = "Query Prometheus for metric values over a time range. Returns time-series data for trend analysis. Use to detect spikes or degradation patterns.")
    public String queryMetricsRange(String promqlQuery, String start, String end, String step) {
        String response = restClient.get()
                .uri("/api/v1/query_range?query={query}&start={start}&end={end}&step={step}",
                        promqlQuery, start, end, step)
                .retrieve()
                .body(String.class);
        return response;
    }
}

What does this code do?

This class is a Spring component that Spring AI registers as a callable tool for the LLM.

• @Tool annotations describe what each method does. The LLM reads these descriptions to decide when and how to call them.
• queryMetrics() hits Prometheus's /api/v1/query endpoint for instant metric snapshots.
• queryMetricsRange() hits /api/v1/query_range for time-series data to detect trends and spikes.
• The RestClient base URL comes from ${sre.prometheus.base-url} in your application.yml. The tool itself has zero knowledge of what services exist.

Why is the tool service-agnostic?

Notice the tool accepts a raw promqlQuery string, not a service name. The LLM decides what PromQL to construct. If you tell the agent to investigate "payment-service", it builds queries like rate(http_server_requests_seconds_count{app="payment-service",status=~"5.."}[5m]) on its own.

This means you never need to update the tool code when new services are deployed. The agent works with any service that follows standard Prometheus label conventions.

• Save PrometheusQueryTool.java.
• You won't see a runtime change yet. The next tool class completes the pair, and both will be registered together in the next step.

Seeing import errors in your IDE?

• Make sure your IDE has finished importing the Maven project. In IntelliJ, click the Maven refresh icon in the right sidebar. In VS Code, wait for the Java extension to finish syncing.
• Verify the file is in the correct package path: sre-agent/src/main/java/com/sre/agent/infrastructure/tools/.
• The org.springframework.ai.tool.annotation.Tool import comes from the spring-ai-starter-model-openai dependency in your sre-agent/pom.xml. If it's unresolved, double-check your parent BOM includes Spring AI 1.0.8.

Still stuck? Ask for help.

Implement the Loki log search tool

Loki is the log counterpart to Prometheus. It stores structured logs indexed by labels, and exposes an HTTP API that accepts LogQL queries. Your second tool gives the LLM the ability to search for error logs, stack traces, and warning patterns for any service.

• Create a new file at sre-agent/src/main/java/com/sre/agent/infrastructure/tools/LokiLogSearchTool.java.
• Add the following code:

package com.sre.agent.infrastructure.tools;

import org.springframework.ai.tool.annotation.Tool;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;
import org.springframework.web.client.RestClient;

@Component
public class LokiLogSearchTool {

    private final RestClient restClient;

    public LokiLogSearchTool(@Value("${sre.loki.base-url}") String lokiUrl) {
        this.restClient = RestClient.builder()
                .baseUrl(lokiUrl)
                .build();
    }

    @Tool(description = "Search application logs in Loki using LogQL. Use label selectors like {app=\"service-name\"} to target a service, then pipe filters like |= \"ERROR\" or |= \"Exception\". Example: {app=\"order-service\"} |= \"ERROR\" | json. Use this to find error patterns, exception stack traces, and warning messages.")
    public String searchLogs(String logqlQuery, String since) {
        long end = System.currentTimeMillis() * 1_000_000L;
        long start = end - parseDuration(since);

        String response = restClient.get()
                .uri("/loki/api/v1/query_range?query={query}&start={start}&end={end}&limit=50",
                        logqlQuery, String.valueOf(start), String.valueOf(end))
                .retrieve()
                .body(String.class);
        return response;
    }

    private long parseDuration(String since) {
        if (since.endsWith("m")) {
            return Long.parseLong(since.replace("m", "")) * 60 * 1_000_000_000L;
        }
        if (since.endsWith("h")) {
            return Long.parseLong(since.replace("h", "")) * 3600 * 1_000_000_000L;
        }
        return 15 * 60 * 1_000_000_000L;
    }
}

What does this code do?

• searchLogs() queries Loki's /loki/api/v1/query_range endpoint with a LogQL expression. The LLM constructs the label selectors dynamically, like {app="demo-service"} |= "ERROR".
• The since parameter lets the LLM specify a lookback window like "15m" or "1h". The parseDuration() helper converts this to nanoseconds for Loki's timestamp format.
• If the LLM omits a valid duration suffix, the default fallback is 15 minutes. This ensures the tool always returns some data even if the LLM passes an unexpected format.

• Save LokiLogSearchTool.java.
• Confirm both tool files exist by checking that your sre-agent/src/main/java/com/sre/agent/infrastructure/tools/ directory contains PrometheusQueryTool.java and LokiLogSearchTool.java.

RestClient not resolving?

• RestClient requires Spring Boot 3.2+. Your project uses 3.4.3, so this should resolve automatically via spring-boot-starter-web.
• If your IDE still shows errors, try running mvn compile from the project root. A successful compile confirms your code is correct even if the IDE lags behind.

Need more help debugging?

✔️ Awesome, I've got everything!

Great. Double-check you've saved both files before moving on.

ⓧ I'd like to double check the full code

Here is PrometheusQueryTool.java in full:

package com.sre.agent.infrastructure.tools;

import org.springframework.ai.tool.annotation.Tool;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;
import org.springframework.web.client.RestClient;

@Component
public class PrometheusQueryTool {

    private final RestClient restClient;

    public PrometheusQueryTool(@Value("${sre.prometheus.base-url}") String prometheusUrl) {
        this.restClient = RestClient.builder()
                .baseUrl(prometheusUrl)
                .build();
    }

    @Tool(description = "Query Prometheus for current metric values using PromQL. Use standard label selectors like {job=\"service-name\"} or {app=\"service-name\"}. Common queries: rate(http_server_requests_seconds_count{app=\"X\",status=~\"5..\"}[5m]) for error rate, histogram_quantile(0.99, rate(http_server_requests_seconds_bucket{app=\"X\"}[5m])) for p99 latency, process_cpu_seconds_total for CPU, jvm_memory_used_bytes for memory.")
    public String queryMetrics(String promqlQuery) {
        String response = restClient.get()
                .uri("/api/v1/query?query={query}", promqlQuery)
                .retrieve()
                .body(String.class);
        return response;
    }

    @Tool(description = "Query Prometheus for metric values over a time range. Returns time-series data for trend analysis. Use to detect spikes or degradation patterns.")
    public String queryMetricsRange(String promqlQuery, String start, String end, String step) {
        String response = restClient.get()
                .uri("/api/v1/query_range?query={query}&start={start}&end={end}&step={step}",
                        promqlQuery, start, end, step)
                .retrieve()
                .body(String.class);
        return response;
    }
}

And here is LokiLogSearchTool.java in full:

package com.sre.agent.infrastructure.tools;

import org.springframework.ai.tool.annotation.Tool;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;
import org.springframework.web.client.RestClient;

@Component
public class LokiLogSearchTool {

    private final RestClient restClient;

    public LokiLogSearchTool(@Value("${sre.loki.base-url}") String lokiUrl) {
        this.restClient = RestClient.builder()
                .baseUrl(lokiUrl)
                .build();
    }

    @Tool(description = "Search application logs in Loki using LogQL. Use label selectors like {app=\"service-name\"} to target a service, then pipe filters like |= \"ERROR\" or |= \"Exception\". Example: {app=\"order-service\"} |= \"ERROR\" | json. Use this to find error patterns, exception stack traces, and warning messages.")
    public String searchLogs(String logqlQuery, String since) {
        long end = System.currentTimeMillis() * 1_000_000L;
        long start = end - parseDuration(since);

        String response = restClient.get()
                .uri("/loki/api/v1/query_range?query={query}&start={start}&end={end}&limit=50",
                        logqlQuery, String.valueOf(start), String.valueOf(end))
                .retrieve()
                .body(String.class);
        return response;
    }

    private long parseDuration(String since) {
        if (since.endsWith("m")) {
            return Long.parseLong(since.replace("m", "")) * 60 * 1_000_000_000L;
        }
        if (since.endsWith("h")) {
            return Long.parseLong(since.replace("h", "")) * 3600 * 1_000_000_000L;
        }
        return 15 * 60 * 1_000_000_000L;
    }
}

Your SRE Agent now has eyes into both metrics and logs. Next up, you'll wire these tools into a Spring AI ChatClient that orchestrates them into a full incident investigation workflow.

Wire the Agent with Spring AI ChatClient

You have two powerful observability tools ready to go. The PrometheusQueryTool can query metrics and the LokiLogSearchTool can search logs. But tools alone don't investigate incidents.

In this step, you'll build the orchestration layer that connects everything together. A ChatClient configured with a system prompt tells the LLM how to behave as an SRE engineer. It receives a service name, decides which tools to call (and with what queries), then produces a structured triage report.

In this step, get ready to:

• Implement the IncidentTriageService with a ChatClient configured for SRE investigation.
• Add a REST controller exposing the triage endpoint.
• Verify the project compiles successfully.

Create the IncidentTriageService

The IncidentTriageService is where the agentic workflow lives. It configures the ChatClient with a system prompt that instructs the LLM to follow a step-by-step investigation procedure. It also registers both tools so the LLM can call them autonomously during an investigation.

• Create a new file called IncidentTriageService.java inside sre-agent/src/main/java/com/sre/agent/application/ and paste the following code:

package com.sre.agent.application;

import com.sre.agent.domain.TriageReport;
import com.sre.agent.infrastructure.tools.LokiLogSearchTool;
import com.sre.agent.infrastructure.tools.PrometheusQueryTool;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.stereotype.Service;

import java.time.Instant;

@Service
public class IncidentTriageService {

    private final ChatClient chatClient;

    public IncidentTriageService(ChatClient.Builder chatClientBuilder,
                                  PrometheusQueryTool prometheusQueryTool,
                                  LokiLogSearchTool lokiLogSearchTool) {
        this.chatClient = chatClientBuilder
                .defaultSystem("""
                    You are a Senior SRE engineer investigating a production incident.
                    Your goal is to quickly identify the root cause and recommend remediation.
                    You are service-agnostic: you can investigate ANY microservice using standard observability patterns.
                    
                    Investigation procedure:
                    1. Query Prometheus for the service's error rate using standard HTTP metrics (http_server_requests_seconds_count with status=~"5..")
                    2. Query Prometheus for latency p99 (http_server_requests_seconds_bucket)
                    3. Query Prometheus for resource usage (process_cpu_seconds_total, jvm_memory_used_bytes)
                    4. Search Loki logs for recent errors using {app="<service-name>"} |= "ERROR"
                    5. Correlate metrics anomalies with log patterns
                    6. Determine severity (CRITICAL, HIGH, MEDIUM, LOW)
                    7. Identify the most likely root cause
                    8. Recommend specific remediation steps
                    
                    Use the service name provided to construct your PromQL and LogQL queries dynamically.
                    Be concise and actionable. Cite specific metric values and log entries as evidence.
                    """)
                .defaultTools(prometheusQueryTool, lokiLogSearchTool)
                .build();
    }

    public TriageReport investigate(String serviceName) {
        String userPrompt = String.format("""
                Investigate the service "%s" for potential issues.
                Use Prometheus to check its error rate, latency, CPU, and memory metrics.
                Use Loki to check for recent error logs.
                Produce a triage report with severity, root cause, evidence, and remediation steps.
                """, serviceName);

        TriageReport report = chatClient.prompt()
                .user(userPrompt)
                .call()
                .entity(TriageReport.class);

        return new TriageReport(
                serviceName,
                report.severity(),
                report.summary(),
                report.rootCause(),
                report.evidence(),
                report.remediationSteps(),
                Instant.now()
        );
    }
}

What does this code do?

The constructor receives Spring AI's ChatClient.Builder (auto-configured from your application.yml settings) along with both tool beans. It builds a ChatClient with two key configurations:

• .defaultSystem(...) sets a system prompt that tells the LLM to act as a Senior SRE and follow an 8-step investigation procedure. The LLM uses this to decide which tools to call.
• .defaultTools(...) registers the Prometheus and Loki tools. When the LLM decides it needs metric data or log entries, it generates a tool call request that Spring AI executes automatically.
• The investigate() method sends a user prompt with the service name. The .entity(TriageReport.class) call tells Spring AI to parse the LLM's final response into your TriageReport record as structured output.

Why rebuild the TriageReport at the end?

The LLM returns a TriageReport with whatever serviceName and timestamp it chose. By constructing a new record, you guarantee the serviceName matches exactly what was requested and the timestamp reflects the actual completion time.

• Save IncidentTriageService.java.

Add the REST controller

The agent needs an HTTP endpoint so the dashboard (and other services) can trigger investigations. A single POST endpoint accepts any service name as a path variable and returns the triage report as JSON.

• Create a new file called TriageController.java inside sre-agent/src/main/java/com/sre/agent/infrastructure/web/ and paste the following code:

package com.sre.agent.infrastructure.web;

import com.sre.agent.application.IncidentTriageService;
import com.sre.agent.domain.TriageReport;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequestMapping("/api/v1/triage")
public class TriageController {

    private final IncidentTriageService triageService;

    public TriageController(IncidentTriageService triageService) {
        this.triageService = triageService;
    }

    @PostMapping("/{serviceName}")
    public ResponseEntity<TriageReport> triage(@PathVariable String serviceName) {
        TriageReport report = triageService.investigate(serviceName);
        return ResponseEntity.ok(report);
    }
}

What does this code do?

• @RequestMapping("/api/v1/triage") sets the base path for all endpoints in this controller.
• @PostMapping("/{serviceName}") captures whatever service name the caller passes in the URL. Calling POST /api/v1/triage/demo-service triggers an investigation against demo-service.
• The controller delegates entirely to IncidentTriageService. This keeps the web layer thin and your business logic testable independently.

Why POST instead of GET?

An investigation is not idempotent. Each call triggers fresh queries against live observability data and consumes LLM tokens. POST signals that this request has side effects and should not be cached or retried without intent.

• Save TriageController.java.

Verify the project compiles

Before moving on, confirm that all the new classes resolve their imports and Spring can wire everything together at compile time.

• Run the Maven compile from your project root directory (ai-sre-agent) by running this command:

mvn compile

• You should see BUILD SUCCESS in the terminal output for both demo-service and sre-agent modules.

Seeing compilation errors?

• If you see "package com.sre.agent.domain does not exist", confirm that TriageReport.java is in sre-agent/src/main/java/com/sre/agent/domain/. This file was created in the previous step.
• If you see "cannot find symbol: ChatClient", double-check that spring-ai-starter-model-openai is listed as a dependency in sre-agent/pom.xml.
• If you see errors about the infrastructure.tools package, verify that PrometheusQueryTool.java and LokiLogSearchTool.java exist in sre-agent/src/main/java/com/sre/agent/infrastructure/tools/.

Help me debug my Maven compilation errors for the SRE agent project

✔️ Awesome, I've got everything!

Great. Double check you've saved both files and that mvn compile shows BUILD SUCCESS.

ⓧ I'd like to double check the full code

Here is the complete IncidentTriageService.java:

package com.sre.agent.application;

import com.sre.agent.domain.TriageReport;
import com.sre.agent.infrastructure.tools.LokiLogSearchTool;
import com.sre.agent.infrastructure.tools.PrometheusQueryTool;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.stereotype.Service;

import java.time.Instant;

@Service
public class IncidentTriageService {

    private final ChatClient chatClient;

    public IncidentTriageService(ChatClient.Builder chatClientBuilder,
                                  PrometheusQueryTool prometheusQueryTool,
                                  LokiLogSearchTool lokiLogSearchTool) {
        this.chatClient = chatClientBuilder
                .defaultSystem("""
                    You are a Senior SRE engineer investigating a production incident.
                    Your goal is to quickly identify the root cause and recommend remediation.
                    You are service-agnostic: you can investigate ANY microservice using standard observability patterns.
                    
                    Investigation procedure:
                    1. Query Prometheus for the service's error rate using standard HTTP metrics (http_server_requests_seconds_count with status=~"5..")
                    2. Query Prometheus for latency p99 (http_server_requests_seconds_bucket)
                    3. Query Prometheus for resource usage (process_cpu_seconds_total, jvm_memory_used_bytes)
                    4. Search Loki logs for recent errors using {app="<service-name>"} |= "ERROR"
                    5. Correlate metrics anomalies with log patterns
                    6. Determine severity (CRITICAL, HIGH, MEDIUM, LOW)
                    7. Identify the most likely root cause
                    8. Recommend specific remediation steps
                    
                    Use the service name provided to construct your PromQL and LogQL queries dynamically.
                    Be concise and actionable. Cite specific metric values and log entries as evidence.
                    """)
                .defaultTools(prometheusQueryTool, lokiLogSearchTool)
                .build();
    }

    public TriageReport investigate(String serviceName) {
        String userPrompt = String.format("""
                Investigate the service "%s" for potential issues.
                Use Prometheus to check its error rate, latency, CPU, and memory metrics.
                Use Loki to check for recent error logs.
                Produce a triage report with severity, root cause, evidence, and remediation steps.
                """, serviceName);

        TriageReport report = chatClient.prompt()
                .user(userPrompt)
                .call()
                .entity(TriageReport.class);

        return new TriageReport(
                serviceName,
                report.severity(),
                report.summary(),
                report.rootCause(),
                report.evidence(),
                report.remediationSteps(),
                Instant.now()
        );
    }
}

And the complete TriageController.java:

package com.sre.agent.infrastructure.web;

import com.sre.agent.application.IncidentTriageService;
import com.sre.agent.domain.TriageReport;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequestMapping("/api/v1/triage")
public class TriageController {

    private final IncidentTriageService triageService;

    public TriageController(IncidentTriageService triageService) {
        this.triageService = triageService;
    }

    @PostMapping("/{serviceName}")
    public ResponseEntity<TriageReport> triage(@PathVariable String serviceName) {
        TriageReport report = triageService.investigate(serviceName);
        return ResponseEntity.ok(report);
    }
}

Your agent can now accept any service name, autonomously query Prometheus and Loki, and return a structured incident triage report. Next up, you'll build the investigation dashboard so you can trigger and visualize investigations from the browser.

Build the Investigation Dashboard

Your SRE Agent now has a working REST API that accepts a service name and returns a structured triage report. The problem is that nobody wants to use curl commands during an incident.

In this step, you will build a single-page investigation dashboard that lets anyone on your team type a service name, click a button, and see a full triage report rendered with severity badges, evidence cards, and remediation checklists.

In this step, get ready to:

• Create a static HTML dashboard served at the root path of the SRE Agent.
• Build an interactive investigation UI with service name input, loading state, and structured report display.
• Wire JavaScript to call the triage API and render results with color-coded severity.

Create the dashboard HTML file

Spring Boot automatically serves any file placed in src/main/resources/static/ at the root URL path. An index.html in that folder becomes the default page at /.

• Create the folder sre-agent/src/main/resources/static/ if it does not already exist.
• Create a new file called index.html inside that static folder.
• Paste the following HTML into the file to set up the page structure, input form, report display areas, and JavaScript logic:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>SRE Agent - Incident Investigation</title>
    <!-- Tailwind CSS via CDN for styling with zero build tooling -->
    <script src="https://cdn.tailwindcss.com"></script>
</head>
<body class="bg-gray-900 text-gray-100 min-h-screen">
    <div class="max-w-4xl mx-auto px-6 py-10">
        <!-- Header -->
        <header class="mb-10">
            <h1 class="text-3xl font-bold text-white">AI SRE Agent</h1>
            <p class="text-gray-400 mt-2">Automated incident investigation for any microservice</p>
        </header>

        <!-- Service name input and investigate button -->
        <div class="bg-gray-800 rounded-lg p-6 mb-8">
            <label for="serviceName" class="block text-sm font-medium text-gray-300 mb-2">Service Name</label>
            <div class="flex gap-3">
                <input type="text" id="serviceName" placeholder="e.g. demo-service, payment-service"
                       class="flex-1 bg-gray-700 border border-gray-600 rounded-md px-4 py-2 text-white placeholder-gray-400 focus:outline-none focus:ring-2 focus:ring-blue-500">
                <button onclick="investigate()" id="investigateBtn"
                        class="bg-blue-600 hover:bg-blue-700 text-white font-medium px-6 py-2 rounded-md transition-colors">
                    Investigate
                </button>
            </div>
        </div>

        <!-- Loading spinner shown during investigation -->
        <div id="loading" class="hidden text-center py-12">
            <div class="inline-block animate-spin rounded-full h-8 w-8 border-4 border-blue-500 border-t-transparent"></div>
            <p class="text-gray-400 mt-4">Agent is investigating... querying Prometheus and Loki</p>
        </div>

        <!-- Triage report display -->
        <div id="report" class="hidden space-y-6">
            <div class="flex items-center gap-4">
                <span id="severityBadge" class="px-3 py-1 rounded-full text-sm font-bold uppercase"></span>
                <h2 id="reportService" class="text-xl font-semibold text-white"></h2>
                <span id="reportTimestamp" class="text-sm text-gray-500 ml-auto"></span>
            </div>

            <div class="bg-gray-800 rounded-lg p-5">
                <h3 class="text-sm font-medium text-gray-400 uppercase tracking-wide mb-2">Summary</h3>
                <p id="reportSummary" class="text-gray-200"></p>
            </div>

            <div class="bg-gray-800 rounded-lg p-5">
                <h3 class="text-sm font-medium text-gray-400 uppercase tracking-wide mb-2">Root Cause</h3>
                <p id="reportRootCause" class="text-gray-200"></p>
            </div>

            <div class="bg-gray-800 rounded-lg p-5">
                <h3 class="text-sm font-medium text-gray-400 uppercase tracking-wide mb-2">Evidence</h3>
                <ul id="reportEvidence" class="space-y-2"></ul>
            </div>

            <div class="bg-gray-800 rounded-lg p-5">
                <h3 class="text-sm font-medium text-gray-400 uppercase tracking-wide mb-2">Remediation Steps</h3>
                <ol id="reportRemediation" class="space-y-2 list-decimal list-inside"></ol>
            </div>
        </div>

        <!-- Error message display -->
        <div id="error" class="hidden bg-red-900/50 border border-red-700 rounded-lg p-5">
            <p id="errorMessage" class="text-red-300"></p>
        </div>
    </div>

    <script>
        // Triggers investigation by calling the triage API
        async function investigate() {
            const serviceName = document.getElementById('serviceName').value.trim();
            if (!serviceName) return;

            document.getElementById('loading').classList.remove('hidden');
            document.getElementById('report').classList.add('hidden');
            document.getElementById('error').classList.add('hidden');
            document.getElementById('investigateBtn').disabled = true;

            try {
                const response = await fetch(`/api/v1/triage/${serviceName}`, { method: 'POST' });
                if (!response.ok) throw new Error(`Investigation failed: ${response.status}`);
                const report = await response.json();
                renderReport(report);
            } catch (err) {
                document.getElementById('error').classList.remove('hidden');
                document.getElementById('errorMessage').textContent = err.message;
            } finally {
                document.getElementById('loading').classList.add('hidden');
                document.getElementById('investigateBtn').disabled = false;
            }
        }

        // Renders the structured triage report into the DOM
        function renderReport(report) {
            document.getElementById('report').classList.remove('hidden');
            document.getElementById('reportService').textContent = report.serviceName;
            document.getElementById('reportSummary').textContent = report.summary;
            document.getElementById('reportRootCause').textContent = report.rootCause;
            document.getElementById('reportTimestamp').textContent = new Date(report.timestamp).toLocaleString();

            // Color-code the severity badge
            const badge = document.getElementById('severityBadge');
            badge.textContent = report.severity;
            const colors = { CRITICAL: 'bg-red-600', HIGH: 'bg-orange-600', MEDIUM: 'bg-yellow-600', LOW: 'bg-green-600' };
            badge.className = `px-3 py-1 rounded-full text-sm font-bold uppercase ${colors[report.severity] || 'bg-gray-600'}`;

            // Render evidence list
            const evidenceList = document.getElementById('reportEvidence');
            evidenceList.innerHTML = '';
            (report.evidence || []).forEach(item => {
                const li = document.createElement('li');
                li.className = 'text-gray-300 bg-gray-700/50 rounded px-3 py-2 text-sm font-mono';
                li.textContent = item;
                evidenceList.appendChild(li);
            });

            // Render remediation steps as an ordered list
            const remediationList = document.getElementById('reportRemediation');
            remediationList.innerHTML = '';
            (report.remediationSteps || []).forEach(step => {
                const li = document.createElement('li');
                li.className = 'text-gray-300';
                li.textContent = step;
                remediationList.appendChild(li);
            });
        }

        // Allow Enter key to trigger investigation
        document.getElementById('serviceName').addEventListener('keypress', (e) => {
            if (e.key === 'Enter') investigate();
        });
    </script>
</body>
</html>

What does this file do?

This single HTML file is the entire dashboard UI. It uses Tailwind CSS via CDN so you get professional styling with zero build tooling.

• The input section lets the user type any service name (e.g. demo-service, payment-service) and click Investigate.
• The investigate() function POSTs to /api/v1/triage/{serviceName} and passes the response to renderReport().
• The renderReport() function maps severity levels to color-coded badges (red for CRITICAL, orange for HIGH, yellow for MEDIUM, green for LOW) and populates the evidence and remediation lists dynamically.
• A keypress listener on the input field lets the user press Enter to trigger the investigation without clicking the button.

Why Tailwind via CDN instead of a build step?

The SRE Agent is a backend service. Its dashboard is a lightweight operational tool, not a full frontend application. Pulling Tailwind from a CDN means the dashboard is a single self-contained HTML file with no npm, no webpack, and no build pipeline to maintain.

Spring Boot serves it automatically from src/main/resources/static/ at the root path /. No extra configuration needed.

• Save index.html.
• Confirm the file exists by running this from your project root:

ls sre-agent/src/main/resources/static/index.html

You should see the file path printed in the output.

File not found?

• Make sure you created the static folder inside sre-agent/src/main/resources/. Spring Boot only auto-serves from this exact path.
• Check that the file is named exactly index.html (lowercase). Any other name will not be served as the default page at /.

Help me troubleshoot my static file not being found in the Spring Boot resources directory.

Verify the dashboard renders correctly

You can check the HTML layout by opening the file directly in your browser. The full end-to-end investigation flow will work once the observability stack is deployed in a later step.

• Open sre-agent/src/main/resources/static/index.html directly in your browser (drag the file into a browser window, or right-click and choose Open With your browser).

You should see a dark-themed page with an "AI SRE Agent" heading, a service name input field, and a blue "Investigate" button. The report sections are hidden until you trigger an investigation.

Page looks unstyled?

• Tailwind loads from cdn.tailwindcss.com. Make sure you have an internet connection when opening the file.
• Verify the <script src="https://cdn.tailwindcss.com"></script> line is inside the <head> tag.

Help me figure out why my Tailwind styles are not loading on the dashboard.

✔️ Awesome, I've got everything!

Great. Double check you have saved index.html and the page renders with the dark theme and input form.

ⓧ I'd like to double check the full code

Here is the complete index.html file:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>SRE Agent - Incident Investigation</title>
    <script src="https://cdn.tailwindcss.com"></script>
</head>
<body class="bg-gray-900 text-gray-100 min-h-screen">
    <div class="max-w-4xl mx-auto px-6 py-10">
        <header class="mb-10">
            <h1 class="text-3xl font-bold text-white">AI SRE Agent</h1>
            <p class="text-gray-400 mt-2">Automated incident investigation for any microservice</p>
        </header>

        <div class="bg-gray-800 rounded-lg p-6 mb-8">
            <label for="serviceName" class="block text-sm font-medium text-gray-300 mb-2">Service Name</label>
            <div class="flex gap-3">
                <input type="text" id="serviceName" placeholder="e.g. demo-service, payment-service"
                       class="flex-1 bg-gray-700 border border-gray-600 rounded-md px-4 py-2 text-white placeholder-gray-400 focus:outline-none focus:ring-2 focus:ring-blue-500">
                <button onclick="investigate()" id="investigateBtn"
                        class="bg-blue-600 hover:bg-blue-700 text-white font-medium px-6 py-2 rounded-md transition-colors">
                    Investigate
                </button>
            </div>
        </div>

        <div id="loading" class="hidden text-center py-12">
            <div class="inline-block animate-spin rounded-full h-8 w-8 border-4 border-blue-500 border-t-transparent"></div>
            <p class="text-gray-400 mt-4">Agent is investigating... querying Prometheus and Loki</p>
        </div>

        <div id="report" class="hidden space-y-6">
            <div class="flex items-center gap-4">
                <span id="severityBadge" class="px-3 py-1 rounded-full text-sm font-bold uppercase"></span>
                <h2 id="reportService" class="text-xl font-semibold text-white"></h2>
                <span id="reportTimestamp" class="text-sm text-gray-500 ml-auto"></span>
            </div>

            <div class="bg-gray-800 rounded-lg p-5">
                <h3 class="text-sm font-medium text-gray-400 uppercase tracking-wide mb-2">Summary</h3>
                <p id="reportSummary" class="text-gray-200"></p>
            </div>

            <div class="bg-gray-800 rounded-lg p-5">
                <h3 class="text-sm font-medium text-gray-400 uppercase tracking-wide mb-2">Root Cause</h3>
                <p id="reportRootCause" class="text-gray-200"></p>
            </div>

            <div class="bg-gray-800 rounded-lg p-5">
                <h3 class="text-sm font-medium text-gray-400 uppercase tracking-wide mb-2">Evidence</h3>
                <ul id="reportEvidence" class="space-y-2"></ul>
            </div>

            <div class="bg-gray-800 rounded-lg p-5">
                <h3 class="text-sm font-medium text-gray-400 uppercase tracking-wide mb-2">Remediation Steps</h3>
                <ol id="reportRemediation" class="space-y-2 list-decimal list-inside"></ol>
            </div>
        </div>

        <div id="error" class="hidden bg-red-900/50 border border-red-700 rounded-lg p-5">
            <p id="errorMessage" class="text-red-300"></p>
        </div>
    </div>

    <script>
        async function investigate() {
            const serviceName = document.getElementById('serviceName').value.trim();
            if (!serviceName) return;

            document.getElementById('loading').classList.remove('hidden');
            document.getElementById('report').classList.add('hidden');
            document.getElementById('error').classList.add('hidden');
            document.getElementById('investigateBtn').disabled = true;

            try {
                const response = await fetch(`/api/v1/triage/${serviceName}`, { method: 'POST' });
                if (!response.ok) throw new Error(`Investigation failed: ${response.status}`);
                const report = await response.json();
                renderReport(report);
            } catch (err) {
                document.getElementById('error').classList.remove('hidden');
                document.getElementById('errorMessage').textContent = err.message;
            } finally {
                document.getElementById('loading').classList.add('hidden');
                document.getElementById('investigateBtn').disabled = false;
            }
        }

        function renderReport(report) {
            document.getElementById('report').classList.remove('hidden');
            document.getElementById('reportService').textContent = report.serviceName;
            document.getElementById('reportSummary').textContent = report.summary;
            document.getElementById('reportRootCause').textContent = report.rootCause;
            document.getElementById('reportTimestamp').textContent = new Date(report.timestamp).toLocaleString();

            const badge = document.getElementById('severityBadge');
            badge.textContent = report.severity;
            const colors = { CRITICAL: 'bg-red-600', HIGH: 'bg-orange-600', MEDIUM: 'bg-yellow-600', LOW: 'bg-green-600' };
            badge.className = `px-3 py-1 rounded-full text-sm font-bold uppercase ${colors[report.severity] || 'bg-gray-600'}`;

            const evidenceList = document.getElementById('reportEvidence');
            evidenceList.innerHTML = '';
            (report.evidence || []).forEach(item => {
                const li = document.createElement('li');
                li.className = 'text-gray-300 bg-gray-700/50 rounded px-3 py-2 text-sm font-mono';
                li.textContent = item;
                evidenceList.appendChild(li);
            });

            const remediationList = document.getElementById('reportRemediation');
            remediationList.innerHTML = '';
            (report.remediationSteps || []).forEach(step => {
                const li = document.createElement('li');
                li.className = 'text-gray-300';
                li.textContent = step;
                remediationList.appendChild(li);
            });
        }

        document.getElementById('serviceName').addEventListener('keypress', (e) => {
            if (e.key === 'Enter') investigate();
        });
    </script>
</body>
</html>

Your dashboard is ready to go. Once the full stack is deployed on Kubernetes, the Investigate button will call through to your agent, which queries Prometheus and Loki, and the report renders right here. Next up, you will build the demo target service that generates realistic metrics and logs for the agent to investigate.

Build the Demo Target Service

Your SRE Agent knows how to query Prometheus and Loki for any service. But right now, there is nothing generating metrics or logs for it to investigate.

In this step, you will build a minimal demo microservice whose sole purpose is generating realistic incident signals. It exposes Prometheus-format metrics via Micrometer and produces structured error logs that Loki can ingest. The agent treats it like any other service in your cluster.

In this step, get ready to:

• Confirm the existing Micrometer and actuator configuration in your demo-service module.
• Create simulation endpoints that generate error spikes and memory pressure.
• Verify both modules compile successfully.

Review the observability configuration

Your demo-service module already has the foundation for observability from the project setup. Two files make this work.

• Open demo-service/pom.xml in your editor.
• Confirm it includes the micrometer-registry-prometheus dependency alongside spring-boot-starter-actuator.
• Open demo-service/src/main/resources/application.yml.
• Confirm the management.endpoints.web.exposure.include property lists health,prometheus,metrics.

Why these two files matter

The micrometer-registry-prometheus dependency translates all Micrometer metrics into Prometheus exposition format. The actuator exposure config makes those metrics scrapeable at /actuator/prometheus.

The management.metrics.tags.app property in application.yml adds an app label to every metric this service emits. This is the exact label your SRE Agent uses in its PromQL queries ({app="demo-service"}) to dynamically discover metrics for any service by name.

Create the simulation controller

The simulation controller gives you on-demand incident scenarios. Each endpoint generates a different failure pattern that the SRE Agent can later detect through metrics and logs.

• Create a new file at demo-service/src/main/java/com/sre/demo/SimulationController.java.
• Add the class skeleton with imports, fields, and the constructor by pasting this code:

package com.sre.demo;

import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.MeterRegistry;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.ArrayList;
import java.util.List;
import java.util.Map;

@RestController
@RequestMapping("/simulate")
public class SimulationController {

    private static final Logger log = LoggerFactory.getLogger(SimulationController.class);
    private final Counter errorCounter;
    private final List<byte[]> memoryLeak = new ArrayList<>();

    public SimulationController(MeterRegistry registry) {
        // Register a counter metric that Prometheus scrapes
        this.errorCounter = Counter.builder("http_server_requests_errors_total")
                .description("Total HTTP errors")
                .tag("app", "demo-service")
                .register(registry);
    }
}

What does this code do?

The MeterRegistry is Spring Boot's auto-configured Micrometer registry. By injecting it into the constructor, you can register custom metrics.

The Counter.builder("http_server_requests_errors_total") creates a counter metric with the tag app=demo-service. When Prometheus scrapes /actuator/prometheus, it sees this counter. The SRE Agent queries it via PromQL to detect error spikes.

The memoryLeak list stores byte arrays on demand, simulating a real memory leak that causes heap pressure.

• Inside the SimulationController class, below the constructor's closing brace, add the error spike endpoint:

    @PostMapping("/error-spike")
    public ResponseEntity<Map<String, String>> triggerErrorSpike() {
        // Emit realistic error log entries that Loki captures
        log.error("NullPointerException in PaymentProcessor.charge(): Cannot invoke method on null reference");
        log.error("Connection timeout to downstream-service at 10.0.2.15:443 after 30000ms");
        log.error("Circuit breaker OPEN for payment-gateway - 10 consecutive failures");
        // Spike the error counter so Prometheus detects the anomaly
        for (int i = 0; i < 50; i++) {
            errorCounter.increment();
        }
        return ResponseEntity.ok(Map.of("status", "error spike triggered", "errors_injected", "50"));
    }

What does this endpoint do?

The /simulate/error-spike endpoint does two things simultaneously. It writes three realistic error log messages (the kind Loki ingests), and it increments the error counter 50 times in a burst.

When the SRE Agent investigates demo-service, it queries Prometheus for this counter's rate and searches Loki for ERROR-level logs. Both signals point to the same incident, giving the agent evidence to correlate.

• Below the triggerErrorSpike method, add the memory leak simulation and reset endpoints:

    @PostMapping("/memory-leak")
    public ResponseEntity<Map<String, String>> triggerMemoryLeak() {
        // Allocate 100MB to simulate heap pressure
        for (int i = 0; i < 100; i++) {
            memoryLeak.add(new byte[1024 * 1024]);
        }
        log.warn("Heap usage exceeding 80% threshold - current: 87%");
        log.error("GC overhead limit approaching - long GC pauses detected (4.2s)");
        return ResponseEntity.ok(Map.of("status", "memory leak triggered", "mb_allocated", "100"));
    }

    @PostMapping("/reset")
    public ResponseEntity<Map<String, String>> reset() {
        // Release allocated memory and request garbage collection
        memoryLeak.clear();
        System.gc();
        return ResponseEntity.ok(Map.of("status", "reset complete"));
    }

What do these endpoints do?

The /simulate/memory-leak endpoint allocates 100 MB of byte arrays into the memoryLeak list. These persist across requests, causing real JVM heap pressure that shows up in jvm_memory_used_bytes metrics.

The /simulate/reset endpoint clears the leak and requests garbage collection so you can run multiple test cycles without restarting the service.

• Save SimulationController.java.

Why this service is completely decoupled from the agent

The demo-service knows nothing about the SRE Agent. It simply exposes standard Prometheus metrics and writes structured logs. Any microservice with the same conventions (Micrometer metrics, SLF4J logging) would work identically as an investigation target.

This is the power of service-agnostic design. The agent uses label selectors like {app="demo-service"} to find metrics dynamically. Replace that name with any other service and the same investigation procedure applies.

✔️ Awesome, I've got everything!

Great. Make sure you have saved SimulationController.java before moving on.

ⓧ I'd like to double check the full code

package com.sre.demo;

import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.MeterRegistry;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.ArrayList;
import java.util.List;
import java.util.Map;

@RestController
@RequestMapping("/simulate")
public class SimulationController {

    private static final Logger log = LoggerFactory.getLogger(SimulationController.class);
    private final Counter errorCounter;
    private final List<byte[]> memoryLeak = new ArrayList<>();

    public SimulationController(MeterRegistry registry) {
        this.errorCounter = Counter.builder("http_server_requests_errors_total")
                .description("Total HTTP errors")
                .tag("app", "demo-service")
                .register(registry);
    }

    @PostMapping("/error-spike")
    public ResponseEntity<Map<String, String>> triggerErrorSpike() {
        log.error("NullPointerException in PaymentProcessor.charge(): Cannot invoke method on null reference");
        log.error("Connection timeout to downstream-service at 10.0.2.15:443 after 30000ms");
        log.error("Circuit breaker OPEN for payment-gateway - 10 consecutive failures");
        for (int i = 0; i < 50; i++) {
            errorCounter.increment();
        }
        return ResponseEntity.ok(Map.of("status", "error spike triggered", "errors_injected", "50"));
    }

    @PostMapping("/memory-leak")
    public ResponseEntity<Map<String, String>> triggerMemoryLeak() {
        for (int i = 0; i < 100; i++) {
            memoryLeak.add(new byte[1024 * 1024]);
        }
        log.warn("Heap usage exceeding 80% threshold - current: 87%");
        log.error("GC overhead limit approaching - long GC pauses detected (4.2s)");
        return ResponseEntity.ok(Map.of("status", "memory leak triggered", "mb_allocated", "100"));
    }

    @PostMapping("/reset")
    public ResponseEntity<Map<String, String>> reset() {
        memoryLeak.clear();
        System.gc();
        return ResponseEntity.ok(Map.of("status", "reset complete"));
    }
}

Compile both modules to verify everything wires up

The new controller uses Micrometer and SLF4J. Both are already on the classpath via the pom.xml dependencies. A successful compile confirms the imports resolve and the class structure is valid.

• Return to your terminal in the project root (ai-sre-agent folder).
• Compile both modules by running:

mvn compile

• You should see BUILD SUCCESS at the bottom of the output, with both demo-service and sre-agent listed as successfully compiled.

Seeing a compilation error?

If you see an error about MeterRegistry or Counter not being found, check that demo-service/pom.xml includes the micrometer-registry-prometheus dependency. This provides the Micrometer classes.

If you see a package import error, verify the file is at exactly demo-service/src/main/java/com/sre/demo/SimulationController.java and the package declaration at the top says package com.sre.demo;.

Help me debug my Maven compilation error for SimulationController

Your demo target is ready. It produces the exact signals your SRE Agent was built to detect. Next, you will containerize both services and write Kubernetes manifests so the full stack can run together in a cluster.

Containerize and Create Kubernetes Manifests

Your SRE Agent and demo service are working locally, but production systems run in containers on Kubernetes. To get there, you need two things: container images that package your apps, and manifests that tell Kubernetes how to run them.

In this step, you will containerize both services with multi-stage Docker builds, write Kubernetes manifests for the entire stack (including Prometheus, Grafana, and Loki), and create an ArgoCD Application for GitOps deployment.

In this step, get ready to:

• Write Dockerfiles for both services using multi-stage builds.
• Create Kubernetes Deployment, Service, and ConfigMap manifests for the full stack.
• Create a docker-compose.yml for local development without K8s.

Write Dockerfiles for both services

A multi-stage Docker build separates the compile step from the runtime image. The first stage uses a full JDK to run Maven. The second stage copies only the finished JAR into a slim JRE image, keeping the final container small and secure.

• Create a file called Dockerfile in the project root (the same level as pom.xml) by pasting this content:

FROM eclipse-temurin:21-jdk-alpine AS build
WORKDIR /app
COPY pom.xml .
COPY sre-agent/pom.xml sre-agent/
COPY sre-agent/src sre-agent/src
RUN apk add --no-cache maven && \
    mvn -f sre-agent/pom.xml clean package -DskipTests

FROM eclipse-temurin:21-jre-alpine
WORKDIR /app
COPY --from=build /app/sre-agent/target/*.jar app.jar
EXPOSE 8081
ENTRYPOINT ["java", "-jar", "app.jar"]

What does this Dockerfile do?

• The build stage uses eclipse-temurin:21-jdk-alpine which includes Java 21 and Maven to compile the source code.
• The runtime stage uses eclipse-temurin:21-jre-alpine which only has the JRE. This cuts the image size significantly.
• EXPOSE 8081 documents the port the SRE Agent listens on.
• -DskipTests skips test execution during the Docker build since tests run in CI separately.

• Create a second file called Dockerfile.demo in the project root for the demo service:

FROM eclipse-temurin:21-jdk-alpine AS build
WORKDIR /app
COPY pom.xml .
COPY demo-service/pom.xml demo-service/
COPY demo-service/src demo-service/src
RUN apk add --no-cache maven && \
    mvn -f demo-service/pom.xml clean package -DskipTests

FROM eclipse-temurin:21-jre-alpine
WORKDIR /app
COPY --from=build /app/demo-service/target/*.jar app.jar
EXPOSE 8080
ENTRYPOINT ["java", "-jar", "app.jar"]

What's different here?

The pattern is identical to the sre-agent Dockerfile, but it copies demo-service/ source files and exposes port 8080 instead of 8081.

• Save both files.
• Verify the files exist by running:

ls Dockerfile Dockerfile.demo

You should see both filenames listed in the output.

Don't see both files?

• Make sure you created the files in the project root directory (where pom.xml lives), not inside sre-agent/ or demo-service/.
• Check the spelling. Docker looks for files named exactly Dockerfile (capital D, no extension) and Dockerfile.demo.

Help me troubleshoot my Dockerfile setup for the SRE Agent project.

Create Kubernetes manifests for the stack

Kubernetes needs YAML manifests to know what to deploy. You will create Deployments (how many pods to run), Services (networking), and a ConfigMap (Prometheus scrape configuration).

• Create the k8s/ directory by running:

mkdir k8s

• Create k8s/demo-service.yaml with the Deployment and Service for the demo service:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-service
  namespace: sre-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: demo-service
  template:
    metadata:
      labels:
        app: demo-service
    spec:
      containers:
        - name: demo-service
          image: demo-service:latest
          imagePullPolicy: Never
          ports:
            - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: demo-service
  namespace: sre-demo
spec:
  selector:
    app: demo-service
  ports:
    - port: 8080
      targetPort: 8080

Why imagePullPolicy: Never?

Docker Desktop shares its local image store with its built-in Kubernetes cluster. Images you build with docker build are immediately available to K8s without pushing to a registry.

Setting imagePullPolicy: Never tells K8s to only use local images. This eliminates the need for a container registry during local development.

• Create k8s/sre-agent.yaml with the SRE Agent Deployment and Service:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sre-agent
  namespace: sre-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sre-agent
  template:
    metadata:
      labels:
        app: sre-agent
    spec:
      containers:
        - name: sre-agent
          image: sre-agent:latest
          imagePullPolicy: Never
          ports:
            - containerPort: 8081
          env:
            - name: OPENROUTER_API_KEY
              valueFrom:
                secretKeyRef:
                  name: sre-agent-secrets
                  key: openrouter-api-key
---
apiVersion: v1
kind: Service
metadata:
  name: sre-agent
  namespace: sre-demo
spec:
  selector:
    app: sre-agent
  ports:
    - port: 8081
      targetPort: 8081

How does the API key get into the pod?

The secretKeyRef block reads the API key from a Kubernetes Secret named sre-agent-secrets. You will create this secret before deploying in the next step. This keeps credentials out of your manifests and Git history.

• Create k8s/prometheus.yaml with the ConfigMap for scrape targets, plus the Prometheus Deployment and Service:

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: sre-demo
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
    scrape_configs:
      - job_name: 'demo-service'
        metrics_path: /actuator/prometheus
        static_configs:
          - targets: ['demo-service:8080']
        relabel_configs:
          - target_label: app
            replacement: demo-service
      - job_name: 'sre-agent'
        metrics_path: /actuator/prometheus
        static_configs:
          - targets: ['sre-agent:8081']
        relabel_configs:
          - target_label: app
            replacement: sre-agent
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  namespace: sre-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      containers:
        - name: prometheus
          image: prom/prometheus:latest
          ports:
            - containerPort: 9090
          volumeMounts:
            - name: config
              mountPath: /etc/prometheus
      volumes:
        - name: config
          configMap:
            name: prometheus-config
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: sre-demo
spec:
  selector:
    app: prometheus
  ports:
    - port: 9090
      targetPort: 9090

What does the ConfigMap do?

• The scrape_configs section tells Prometheus which services to poll for metrics every 15 seconds.
• The relabel_configs add an app label to every scraped metric. This is how the SRE Agent's PromQL queries find metrics for a given service name.
• The volumeMount injects this config file into the Prometheus container at /etc/prometheus.

• Create k8s/loki.yaml for the log aggregation service:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: loki
  namespace: sre-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: loki
  template:
    metadata:
      labels:
        app: loki
    spec:
      containers:
        - name: loki
          image: grafana/loki:2.9.0
          ports:
            - containerPort: 3100
---
apiVersion: v1
kind: Service
metadata:
  name: loki
  namespace: sre-demo
spec:
  selector:
    app: loki
  ports:
    - port: 3100
      targetPort: 3100

• Create k8s/grafana.yaml for the dashboarding layer:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace: sre-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
        - name: grafana
          image: grafana/grafana:latest
          ports:
            - containerPort: 3000
          env:
            - name: GF_AUTH_ANONYMOUS_ENABLED
              value: "true"
            - name: GF_AUTH_ANONYMOUS_ORG_ROLE
              value: "Admin"
---
apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: sre-demo
spec:
  selector:
    app: grafana
  ports:
    - port: 3000
      targetPort: 3000

Why enable anonymous Grafana access?

The GF_AUTH_ANONYMOUS_ENABLED environment variable lets you access Grafana dashboards without logging in. This is fine for local development. In production you would remove this and use proper authentication.

• Create k8s/argocd-application.yaml to define the ArgoCD Application that points to your Git repository:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sre-demo
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/[[YOUR_GITHUB_USERNAME="YOUR_USERNAME"]]/ai-sre-agent.git
    targetRevision: HEAD
    path: k8s/
  destination:
    server: https://kubernetes.default.svc
    namespace: sre-demo
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

What does this ArgoCD Application do?

• source.path: k8s/ tells ArgoCD to watch the k8s/ directory in your repo for manifest changes.
• destination.server: https://kubernetes.default.svc means ArgoCD deploys to the same cluster it runs in. No external cluster registration needed.
• automated.selfHeal: true means ArgoCD will revert any manual changes made to resources, keeping the cluster in sync with Git.
• CreateNamespace=true lets ArgoCD create the sre-demo namespace automatically.

• Save all files in the k8s/ directory.
• Confirm the manifests exist by running:

ls k8s/

You should see argocd-application.yaml, demo-service.yaml, grafana.yaml, loki.yaml, prometheus.yaml, and sre-agent.yaml listed.

Missing files in k8s/?

• Make sure you created the k8s/ directory first with mkdir k8s before saving files into it.
• Check your working directory. Run ls pom.xml to confirm you are in the project root.

Help me check my Kubernetes manifest file structure.

Create a docker-compose.yml for local development

Not everyone has Kubernetes running locally. A Docker Compose file lets you spin up the entire stack with a single command for quick iteration.

• Create docker-compose.yml in the project root with the following content:

version: '3.8'

services:
  demo-service:
    build:
      context: .
      dockerfile: Dockerfile.demo
    ports:
      - "8080:8080"

  sre-agent:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "8081:8081"
    environment:
      - OPENROUTER_API_KEY=${OPENROUTER_API_KEY}
      - SRE_PROMETHEUS_BASE-URL=http://prometheus:9090
      - SRE_LOKI_BASE-URL=http://loki:3100
    depends_on:
      - prometheus
      - loki

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./k8s/prometheus-config.yml:/etc/prometheus/prometheus.yml

  loki:
    image: grafana/loki:2.9.0
    ports:
      - "3100:3100"

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin

What does this compose file do?

• Both application services build from their respective Dockerfiles using the project root as context.
• The depends_on key ensures Prometheus and Loki start before the SRE Agent.
• Service names (prometheus, loki) become DNS hostnames inside the Docker network. This matches the URLs in the agent's application.yml.
• OPENROUTER_API_KEY is read from your shell environment at runtime.

• Save docker-compose.yml.
• Verify the full project structure by running:

ls Dockerfile Dockerfile.demo docker-compose.yml k8s/

You should see both Dockerfiles, the compose file, and the contents of the k8s/ directory listed.

Docker Compose file not valid?

• YAML is whitespace-sensitive. Make sure you use spaces (not tabs) for indentation.
• Check that port mappings use quotes around the values (e.g. "8080:8080") to prevent YAML from interpreting them as numbers.

Help me debug my docker-compose.yml for the SRE Agent project.

✔️ Awesome, I've got everything!

Great. Double check you have saved all files: Dockerfile, Dockerfile.demo, docker-compose.yml, and six YAML files in k8s/.

ⓧ I'd like to double check the full code

Here is the sre-agent Dockerfile (this is the primary one):

FROM eclipse-temurin:21-jdk-alpine AS build
WORKDIR /app
COPY pom.xml .
COPY sre-agent/pom.xml sre-agent/
COPY sre-agent/src sre-agent/src
RUN apk add --no-cache maven && \
    mvn -f sre-agent/pom.xml clean package -DskipTests

FROM eclipse-temurin:21-jre-alpine
WORKDIR /app
COPY --from=build /app/sre-agent/target/*.jar app.jar
EXPOSE 8081
ENTRYPOINT ["java", "-jar", "app.jar"]

Here is the ArgoCD Application manifest:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sre-demo
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/YOUR_USERNAME/ai-sre-agent.git
    targetRevision: HEAD
    path: k8s/
  destination:
    server: https://kubernetes.default.svc
    namespace: sre-demo
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Your project is fully containerized and ready for Kubernetes. Next up, you will deploy the entire stack with ArgoCD and see GitOps in action.

Deploy with ArgoCD on Docker Desktop Kubernetes

Your SRE Agent, demo service, and entire observability stack are containerized and defined as Kubernetes manifests. Instead of running kubectl apply for each resource individually, you'll use ArgoCD to deploy and manage everything through a GitOps workflow.

ArgoCD watches your Git repository and automatically syncs your cluster to match whatever manifests live in the k8s/ directory. Push a change to GitHub, and ArgoCD deploys it. No manual kubectl commands needed after the initial setup.

In this step, get ready to:

• Install ArgoCD on Docker Desktop's Kubernetes cluster.
• Apply the ArgoCD Application manifest to trigger GitOps sync.
• Verify all workloads are synced and healthy via the ArgoCD dashboard.

Install ArgoCD on Docker Desktop Kubernetes

ArgoCD runs as a set of pods inside your Kubernetes cluster. You'll install it into its own dedicated namespace, then retrieve the admin credentials to access its dashboard.

• Create the ArgoCD namespace and install ArgoCD by running these commands:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

What does this do?

The first command creates a dedicated argocd namespace to isolate ArgoCD's components from your application workloads.

The second command pulls ArgoCD's official manifest and installs all its components (Deployments, Services, ConfigMaps, RBAC rules) into that namespace.

• Docker Desktop's networking layer does not enforce NetworkPolicies correctly. Remove them so ArgoCD's components can communicate by running:

kubectl delete networkpolicies --all -n argocd

Why delete NetworkPolicies?

Docker Desktop's CNI plugin does not properly enforce Kubernetes NetworkPolicies. ArgoCD ships with NetworkPolicy resources that restrict traffic between its components. On Docker Desktop, these policies block legitimate communication and cause pods to crash-loop.

• Wait for all ArgoCD pods to become ready by running:

kubectl wait --for=condition=ready pod --all -n argocd --timeout=120s

You should see output showing each pod's condition met, confirming ArgoCD is fully running.

Pods stuck in Pending or CrashLoopBackOff?

Check that Docker Desktop has enough resources allocated. This stack needs at least 4 GB RAM and 2 CPUs. Open Docker Desktop, go to Settings, then Resources, and increase memory to 6 GB or higher.

If you see permission errors, confirm Kubernetes is enabled in Docker Desktop under Settings then Kubernetes.

Still stuck? Help me debug ArgoCD pods not starting on Docker Desktop Kubernetes

• Now retrieve the ArgoCD admin password. The default username is admin.

macOS

• Retrieve and decode the admin password by running:

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 --decode

• Copy the output. This is your ArgoCD admin password.

Windows

• Retrieve and decode the admin password by running in PowerShell:

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | ForEach-Object { [System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($_)) }

• Copy the output. This is your ArgoCD admin password.

Save that password

Paste the decoded password somewhere safe. You'll need it to log in to the ArgoCD dashboard and CLI in the next substeps.

Install the ArgoCD CLI and log in

The ArgoCD CLI lets you manage applications, trigger syncs, and check status directly from your terminal. You need it installed before you can interact with your ArgoCD server.

• Check if you already have the ArgoCD CLI installed by running:

argocd version --client

✔️ I see a version number

Your ArgoCD CLI is ready. Continue to the port-forward step below.

ⓧ Command not found

macOS

• Install the ArgoCD CLI with Homebrew by running:

brew install argocd

Windows

• Download the ArgoCD CLI from GitHub releases by running in PowerShell:

$version = (Invoke-RestMethod https://api.github.com/repos/argoproj/argo-cd/releases/latest).tag_name
$url = "https://github.com/argoproj/argo-cd/releases/download/$version/argocd-windows-amd64.exe"
Invoke-WebRequest -Uri $url -OutFile "argocd.exe"

• Move argocd.exe to a directory on your PATH, or add its current location to your PATH environment variable.

• Verify the installation succeeded by running:

argocd version --client

You should see the ArgoCD client version printed in the output.

• Start port-forwarding to the ArgoCD server so you can access it locally. Run this in a dedicated terminal window:

kubectl port-forward svc/argocd-server -n argocd 8080:443

Keep this terminal running

The port-forward command keeps running in the foreground. Open a new terminal window or tab for the remaining commands. The ArgoCD server is now accessible at https://localhost:8080.

• In a new terminal, log in to ArgoCD via the CLI by running:

argocd login localhost:8080 --username admin --password your-argocd-password-here --insecure

• Replace your-argocd-password-here with the password you retrieved earlier.
• The --insecure flag is needed because ArgoCD uses a self-signed TLS certificate by default.

You should see a message confirming you are logged in successfully.

Login failed or connection refused?

Make sure the kubectl port-forward command is still running in your other terminal. If it crashed, restart it.

If you see a certificate error, double-check you included the --insecure flag.

Still having trouble? Help me log in to ArgoCD CLI on localhost

Apply the ArgoCD Application and verify deployment

The k8s/argocd-application.yaml manifest you created earlier tells ArgoCD to watch your GitHub repository's k8s/ directory. Once applied, ArgoCD reads every manifest in that directory and deploys the resources to your cluster.

• Before applying, confirm the repoURL in k8s/argocd-application.yaml points to your actual GitHub repository URL. Open the file and verify this line:

    repoURL: https://github.com/YOUR_USERNAME/ai-sre-agent.git

• If you have not already replaced YOUR_USERNAME with your actual GitHub username, do so now and push the change.
• Apply the ArgoCD Application to the cluster by running:

kubectl apply -f k8s/argocd-application.yaml

What happens next?

ArgoCD reads the Application resource, connects to your GitHub repo, finds all manifests in the k8s/ directory, and begins deploying them into the sre-demo namespace.

The syncPolicy.automated setting means ArgoCD syncs continuously without manual triggers. The CreateNamespace=true option lets ArgoCD create the sre-demo namespace if it does not already exist.

• Wait for ArgoCD to sync and deploy all resources. Check the application status by running:

argocd app get sre-demo

• You should see Status: Synced and Health Status: Healthy in the output. This may take 1-2 minutes on the first sync.

• Verify that all pods are running in the sre-demo namespace by running:

kubectl get pods -n sre-demo

You should see pods for sre-agent, demo-service, prometheus, grafana, and loki all showing Running status.

• Open the ArgoCD dashboard in your browser by navigating to https://localhost:8080.
• Your browser may show a certificate warning. Click Advanced then Proceed to localhost (or equivalent) to continue.
• Log in with username admin and the password you retrieved earlier.
• You should see the sre-demo application card showing a green Synced and Healthy status.

Application showing OutOfSync or Degraded?

Run argocd app sync sre-demo to force a manual sync. If specific resources fail, run kubectl describe pod <pod-name> -n sre-demo to see detailed error messages.

If you see ImagePullBackOff errors, your Docker images need to be available locally. Ensure you built the images with docker build and that your Kubernetes manifests use imagePullPolicy: Never or imagePullPolicy: IfNotPresent.

Need more help? Help me troubleshoot ArgoCD sync failures for my sre-demo application

• To access the SRE Agent dashboard locally, open another terminal and run:

kubectl port-forward svc/sre-agent -n sre-demo 8081:8081

The SRE Agent investigation dashboard is now accessible at http://localhost:8081.

Your entire stack is now deployed and managed through GitOps. Any change you push to the k8s/ directory in your GitHub repository will be automatically picked up and applied by ArgoCD. Next up, you'll trigger a simulated incident and watch the SRE Agent investigate it in real-time.

Run an End-to-End Incident Investigation

Your entire stack is deployed and healthy on Kubernetes via ArgoCD. The SRE Agent, demo-service, Prometheus, Grafana, and Loki are all running in your cluster.

Now it's time to see the full agentic workflow in action. You'll simulate a production incident, point the AI agent at the affected service, and watch it produce a structured triage report with zero hardcoded knowledge of the target service.

In this step, get ready to:

• Trigger a simulated error spike on the demo-service.
• Investigate the incident from the SRE Agent dashboard.
• Review the structured TriageReport with severity, root cause, evidence, and remediation steps.

Trigger a simulated incident on the demo-service

The demo-service has simulation endpoints that generate realistic metric spikes and error logs. By hitting /simulate/error-spike, you inject 50 error counter increments and three distinct error log messages into the system.

Prometheus scrapes the new metric values, and Loki ingests the structured error logs. This gives the SRE Agent real data to query when it investigates.

• Open a new terminal window. Set up a port-forward to the demo-service by running this command:

kubectl port-forward svc/demo-service 8080:8080 -n sre-demo

What does this command do?

This forwards traffic from localhost:8080 on your machine to port 8080 on the demo-service pod inside Kubernetes. It lets you reach the service's endpoints without exposing them publicly.

• This command stays running in the foreground. Open another terminal window and trigger the simulated error spike by running:

curl -X POST http://localhost:8080/simulate/error-spike

• You should see a JSON response confirming the spike was triggered:

{"status":"error spike triggered","errors_injected":"50"}

What just happened inside the cluster?

The SimulationController did three things when you hit that endpoint.

• Logged three distinct error messages (NullPointerException, connection timeout, circuit breaker open).
• Incremented the http_server_requests_errors_total counter 50 times.
• Prometheus picks up the counter on its next scrape. Loki ingests the structured log entries in real time.

• Wait about 15-30 seconds for Prometheus to scrape the new metrics before moving to the next substep.

Seeing connection refused?

Make sure the port-forward command is still running in the other terminal. If it exited, run it again.

If you get a pod not found error, verify the demo-service pod is running with kubectl get pods -n sre-demo.

Help me troubleshoot why my port-forward to demo-service isn't working on Docker Desktop Kubernetes.

Open the SRE Agent dashboard and investigate

The SRE Agent serves a single-page investigation dashboard at its root path. You type a service name, click Investigate, and the agent autonomously queries Prometheus and Loki via tool calling to produce a structured triage report.

• Open another terminal window and set up a port-forward to the SRE Agent by running:

kubectl port-forward svc/sre-agent 8081:8081 -n sre-demo

• Open your browser and navigate to:

http://localhost:8081

You should see the AI SRE Agent dashboard with a dark background, a "Service Name" input field, and an "Investigate" button.

• Type demo-service into the **Service Name** input field.
• Click the **Investigate** button.

A spinning loader appears while the agent works. Behind the scenes, the IncidentTriageService sends a prompt to the LLM, which calls your PrometheusQueryTool and LokiLogSearchTool multiple times to gather evidence. This typically takes 10-30 seconds depending on the LLM response time.

What is happening during the investigation?

The full agentic workflow flows like this:

• Your browser sends POST /api/v1/triage/demo-service to the SRE Agent.
• The ChatClient sends the system prompt and user prompt to the LLM via OpenRouter.
• The LLM requests tool calls. Spring AI executes PrometheusQueryTool and LokiLogSearchTool with the queries the LLM constructed.
• The LLM receives the raw metrics and logs, correlates them, and returns a structured TriageReport.

Seeing an error in the dashboard?

If you see "Investigation failed: 500", check the SRE Agent logs for details by running kubectl logs -l app=sre-agent -n sre-demo --tail=50 in a separate terminal.

Common causes: your OPENROUTER_API_KEY environment variable isn't set in the pod, or Prometheus/Loki services aren't reachable from the SRE Agent pod.

Help me debug why my SRE Agent is returning a 500 error when I trigger an investigation.

Review the structured triage report

Once the investigation completes, the dashboard renders a full triage report. This is the same structured TriageReport record your domain model defines, rendered with visual formatting by the frontend.

• Look at the top of the report. You should see a color-coded severity badge (likely HIGH or CRITICAL) alongside the service name and timestamp.
• Read the **Summary** section. The agent should describe the overall health issue it detected.
• Read the **Root Cause** section. The agent typically identifies the NullPointerException or circuit breaker failure from the log evidence.
• Review the **Evidence** section. Each item is a specific metric value or log entry the agent found via its tool calls.
• Review the **Remediation Steps** section. These are actionable recommendations the LLM generated based on the evidence it gathered.

Why is this service-agnostic?

Notice that you typed "demo-service" into the input field. The agent has zero hardcoded knowledge of that service. It dynamically built PromQL queries like {app="demo-service"} and LogQL selectors like {app="demo-service"} |= "ERROR" based purely on standard observability conventions.

Any microservice in your cluster that exposes Prometheus metrics and sends structured logs to Loki can be investigated the same way. Just type a different service name.

Try investigating a non-existent service

Type a random service name like payment-service and click Investigate. The agent still runs the full workflow but finds no metrics or logs. Watch how it handles the absence of data in its report.

You've just verified the entire agentic workflow end-to-end. The SRE Agent autonomously investigated a production-like incident, correlated metrics with logs, and delivered a structured triage report. Next up, take on the Secret Mission to wire this agent to fire automatically when Prometheus alerts trigger.

Secret mission

Add Alertmanager Webhook Integration

Right now, the SRE Agent only investigates when you manually click the button. In production, incidents don't wait for humans. Wire up Prometheus Alertmanager so your agent is triggered automatically the moment an alert fires for any service.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely on Docker Desktop's local Kubernetes cluster, so there are no ongoing cloud costs. The stack does consume local CPU and memory while running.

Resources you used:

• Kubernetes namespace sre-demo (contains Prometheus, Grafana, Loki, Alertmanager, demo-service, sre-agent).
• Kubernetes namespace argocd (contains the ArgoCD server and components).
• Docker images built for sre-agent and demo-service.
• Local project directory ai-sre-agent on your Desktop.

Keep everything running

No action needed. Choose this if you're still actively building or experimenting with the SRE Agent.

• Docker Desktop continues running the Kubernetes cluster with all deployed workloads.
• The full stack (ArgoCD, Prometheus, Grafana, Loki, both services) uses approximately 4-6 GB of RAM while running.
• Your OpenRouter API key remains configured. No API calls are made unless you trigger an investigation.

Pause - I'll come back to this later

Shut down Docker Desktop to free up local memory and CPU. All Kubernetes state persists and comes back when you relaunch Docker Desktop.

• Right-click the Docker Desktop icon in your system tray (Windows) or menu bar (macOS).
• Click Quit Docker Desktop.
• When you relaunch Docker Desktop later, your Kubernetes cluster and all deployed workloads resume automatically.

Delete - I don't want to use this again

Remove all project workloads from Kubernetes, clean up Docker images, and delete local project files.

• Delete the sre-demo namespace (removes Prometheus, Grafana, Loki, Alertmanager, demo-service, and sre-agent) by running:

kubectl delete namespace sre-demo

• Delete the argocd namespace (removes all ArgoCD components) by running:

kubectl delete namespace argocd

• Remove the Docker images built for this project by running:

docker image prune -a --filter "label=project=ai-sre-agent"

Full Kubernetes reset

If you want to completely reset your Kubernetes cluster to a clean state, open Docker Desktop and go to Settings. Select the Kubernetes tab, then click Reset Kubernetes Cluster. This removes all namespaces, pods, and services in one action.

• Delete the local project directory. On macOS/Linux, run:

cd ~/Desktop
rm -rf ai-sre-agent

• On Windows, run:

cd ~\Desktop
Remove-Item -Recurse -Force ai-sre-agent

Well Done!

Well Done!

You just built a production-grade AI-powered operations tool that demonstrates the exact patterns enterprises are deploying today.

Your SRE Agent automates the first 15 minutes of incident investigation for any microservice in your cluster, using agentic AI workflows, standard observability conventions, and clean architecture.

You've learned how to:

• Build a service-agnostic Spring AI agent with tool calling and structured output that dynamically queries Prometheus metrics and Loki logs for any microservice.
• Deploy a full observability stack (Prometheus, Grafana, Loki) on Kubernetes via ArgoCD GitOps, with hexagonal architecture separating domain logic from infrastructure.
• Create an investigation dashboard that accepts any service name and renders a structured triage report with severity assessment, root cause analysis, evidence, and remediation steps.
• Automated alert-to-investigation pipeline via Alertmanager webhooks, triggering the SRE Agent automatically when a Prometheus alert fires.

Ready to quiz yourself?