Multi-Agent SRE Pipeline with kagent

Build a five-agent SRE pipeline on Kubernetes using kagent, Bedrock, and OpenRouter.

Introduction

30 Second Summary

When a production system breaks at 3 AM, the best SRE teams don't have one person scrambling to do everything. They have specialists: someone investigates, someone patches, someone designs the long-term fix, and someone implements it.

In this project, you will build a five-agent SRE operations pipeline on Kubernetes using kagent and OpenSRE, where an Orchestrator agent coordinates four specialist agents to diagnose, fix, design, and build resources in response to a real incident.

What You'll Build

You will deploy kagent on minikube and configure two LLM providers (Amazon Bedrock and OpenRouter) as Kubernetes CRDs, then build a multi-agent pipeline that responds to a deliberately broken workload.

By the end of this project, you'll have:

  • A five-agent kagent deployment on minikube with an Orchestrator, Diagnostician, Remediator, Architect, and Builder, each with purpose-matched LLM providers and distinct tool sets.
  • A working incident response pipeline where OpenSRE generates a structured investigation report that feeds into the kagent Orchestrator, which delegates across specialist agents to diagnose, design, and fix cluster issues.
  • Hands-on experience with the agents-as-tools pattern, HITL approval gates on destructive operations, and multi-provider LLM configuration (Bedrock for reasoning-heavy agents, OpenRouter for execution-focused agents).
  • A secret mission to enable agent memory on the Diagnostician so it retains context across investigations.

Are there any prerequisites?

This project assumes strong familiarity with Kubernetes, Helm, kubectl, and CRDs. You need a running minikube cluster, an AWS account with Bedrock model access enabled, an OpenRouter API key, and Python 3.x installed for OpenSRE.

Install kagent and OpenSRE

Before you can build your multi-agent pipeline, you need two core tools installed on your machine. kagent is the Kubernetes-native agent framework that runs your AI agents as CRDs. OpenSRE is the incident investigation tool that generates structured reports from alerts.

In this step, you'll install both CLIs, deploy kagent to your minikube cluster, and confirm everything is accessible.

In This Step, Get Ready To:
  • Install the kagent CLI and deploy it to minikube.
  • Install the OpenSRE CLI and configure its LLM provider.
  • Open the kagent dashboard and confirm the UI is running.
Install the kagent CLI and Deploy kagent to Minikube

The kagent CLI handles both the installation of kagent into your cluster and ongoing management tasks like opening the dashboard.

  • Confirm your minikube cluster is running.
minikube status

Minikube not running?

Start it with minikube start. If you don't have minikube installed, this project requires a running minikube cluster as a prerequisite. Help me set up minikube.

  • Install the kagent CLI.

macOS

brew install kagent

Windows

curl https://raw.githubusercontent.com/kagent-dev/kagent/refs/heads/main/scripts/get-kagent | bash

Linux

curl https://raw.githubusercontent.com/kagent-dev/kagent/refs/heads/main/scripts/get-kagent | bash

The kagent install process requires an OPENAI_API_KEY environment variable to be set, even if you're not using OpenAI as a provider.

  • Set a placeholder value for the required environment variable.
export OPENAI_API_KEY="placeholder"

Why a placeholder key?

The kagent installer checks for this environment variable during setup. You won't actually use OpenAI in this project.

You'll configure Amazon Bedrock and OpenRouter as your LLM providers in the next step.

  • Install kagent into your minikube cluster using the minimal profile.
kagent install --profile minimal

What does this command do?

This installs the kagent controller, the dashboard UI, a bundled PostgreSQL database, and all the custom resource definitions (CRDs) into a kagent namespace on your cluster.

The minimal profile skips demo agents and sample MCP tools. You'll create your own agents from scratch.

This installation takes a couple of minutes. Wait for it to complete before continuing.

  • Verify that all kagent pods are running.
kubectl get pods -n kagent

I see Running pods

You should see pods for the kagent controller, the UI, and PostgreSQL, all in a Running state. kagent is successfully deployed.

I see errors or pending pods

If pods are stuck in Pending or CrashLoopBackOff, check that your minikube cluster has enough resources allocated.

  • Run kubectl describe pod <pod-name> -n kagent to see detailed error messages.
  • Try restarting minikube with more memory: minikube start --memory=4096.

Help me troubleshoot my kagent installation.

Install OpenSRE

OpenSRE is an open-source SRE investigation tool that generates structured incident reports from alerts. Later in this project, you'll use OpenSRE to produce an investigation report that feeds directly into your kagent Orchestrator agent.

  • Install the OpenSRE CLI.

macOS

brew install Tracer-Cloud/opensre/opensre

Windows

pipx install opensre

Linux

pipx install opensre
  • Run the OpenSRE onboarding wizard to configure your LLM provider.
opensre onboard

What does the onboarding wizard do?

The wizard prompts you to select an LLM provider for OpenSRE's own investigation engine. OpenSRE supports multiple providers including Anthropic, OpenAI, Bedrock, OpenRouter, Ollama, Google Gemini, and NVIDIA NIM. Pick whichever provider you already have credentials for.

  • Verify the installation by checking the help output.
opensre --help

You should see a list of available commands including investigate and onboard.

Access the kagent Dashboard

kagent includes a built-in web dashboard for managing agents, viewing conversations, and approving tool calls.

  • Launch the kagent dashboard.
kagent dashboard

The dashboard opens in your browser at http://localhost:8082.

  • Verify you see the kagent landing page in your browser.
  • Leave this terminal running.
  • Open a new terminal window for all subsequent commands.

Pro tip: keep the dashboard open

The kagent dashboard command port-forwards the UI service to your local machine. Closing this terminal kills the dashboard connection. Keep it running in the background throughout the project.

Your tools are installed and your cluster is ready. Next up, you'll configure Amazon Bedrock and OpenRouter as LLM providers so your agents have the brains to power their decisions.

Configure Bedrock and OpenRouter Model Providers

Your five-agent pipeline needs two different LLM providers, each matched to the type of work its agents perform. Amazon Bedrock will power the reasoning-heavy agents (Diagnostician and Architect) that need strong analytical capabilities for root cause analysis and architectural design. OpenRouter will power the execution-focused agents (Orchestrator, Remediator, Builder) that primarily route tasks, patch resources, and apply manifests.

In this step, you'll create both providers as Kubernetes CRDs so your agents can reference them by name.

In This Step, Get Ready To:
  • Create a native Bedrock ModelConfig CRD with AWS credentials stored as a Kubernetes secret.
  • Create an OpenRouter ModelConfig CRD using kagent's BYO OpenAI-compatible provider pattern.
  • Verify both providers appear in the kagent dashboard.
Create Secrets and the Bedrock ModelConfig

kagent uses Kubernetes secrets to store provider credentials separately from the ModelConfig resource itself. For Bedrock, kagent has a native provider integration that uses AWS IAM credentials directly.

Why Native Bedrock Instead of OpenAI-Compatible Bedrock?

kagent supports Bedrock as a first-class provider, handled natively by Google ADK under the hood. The OpenAI-compatible Bedrock endpoint has had historical issues with header mismatches. Using provider: Bedrock avoids those problems entirely.

  • Create the Kubernetes secret containing your AWS credentials.
kubectl create secret generic bedrock-credentials -n kagent \
  --from-literal=AWS_ACCESS_KEY_ID=your-access-key-here \
  --from-literal=AWS_SECRET_ACCESS_KEY=your-secret-key-here

What Does This Command Do?

This creates a Kubernetes secret named bedrock-credentials in the kagent namespace. Replace your-access-key-here and your-secret-key-here with your actual AWS IAM credentials that have Bedrock model access.

  • Create a file called bedrock-modelconfig.yaml with the following content.
apiVersion: kagent.dev/v1alpha2
kind: ModelConfig
metadata:
  name: bedrock-native
  namespace: kagent
spec:
  provider: Bedrock
  model: us.anthropic.claude-sonnet-4-20250514-v1:0
  apiKeySecret: bedrock-credentials
  bedrock:
    region: us-east-1

What Is a ModelConfig?

A ModelConfig is a kagent custom resource that defines an LLM provider connection. Agents reference a ModelConfig by name instead of embedding credentials directly. The provider: Bedrock field tells kagent to use its native Bedrock integration. The apiKeySecret field points to the Kubernetes secret containing your AWS credentials.

  • Apply the ModelConfig to your cluster.
kubectl apply -f bedrock-modelconfig.yaml
  • Verify the ModelConfig was created.
kubectl get modelconfig bedrock-native -n kagent

You should see the bedrock-native ModelConfig listed with its provider and model details.

Create Secrets and the OpenRouter ModelConfig

OpenRouter provides an OpenAI-compatible API, which means kagent can connect to it using the BYO (Bring Your Own) provider pattern. You set provider: OpenAI and override the base URL to point at OpenRouter's endpoint. This same pattern works with any OpenAI-compatible provider.

Why OpenRouter for the Execution Agents?

OpenRouter gives you access to 300+ models through a single endpoint with automatic fallbacks. The execution-focused agents (Orchestrator, Remediator, Builder) primarily route tasks, patch resources, and apply manifests. They need reliable instruction-following and tool execution more than deep analytical reasoning.

Using two different providers also demonstrates a real production pattern: matching LLM capabilities to agent responsibilities to optimize both cost and performance.

  • Store your OpenRouter API key in a Kubernetes secret.
kubectl create secret generic kagent-openrouter -n kagent \
  --from-literal=OPENROUTER_API_KEY=your-openrouter-api-key-here

What Does This Command Do?

This creates a secret named kagent-openrouter with a key called OPENROUTER_API_KEY. Replace your-openrouter-api-key-here with your actual API key from your OpenRouter account.

  • Create a file called openrouter-modelconfig.yaml with the following content.
apiVersion: kagent.dev/v1alpha2
kind: ModelConfig
metadata:
  name: openrouter-config
  namespace: kagent
spec:
  provider: OpenAI
  model: anthropic/claude-sonnet-4
  apiKeySecret: kagent-openrouter
  apiKeySecretKey: OPENROUTER_API_KEY
  openAI:
    baseUrl: "https://openrouter.ai/api/v1"

What Makes This a BYO Provider?

The key is the combination of provider: OpenAI with a custom openAI.baseUrl. kagent treats this as an OpenAI-compatible endpoint but sends all requests to OpenRouter instead. The apiKeySecretKey field tells kagent which key inside the secret holds the API token, since the secret key name (OPENROUTER_API_KEY) differs from the default.

  • Apply the ModelConfig to your cluster.
kubectl apply -f openrouter-modelconfig.yaml
  • Verify the ModelConfig was created.
kubectl get modelconfig openrouter-config -n kagent

You should see openrouter-config listed alongside the Bedrock config you created earlier.

Confirm Both Providers in the Dashboard

Now that both ModelConfigs exist as Kubernetes resources, they should be visible in the kagent dashboard. This confirms the kagent controller has picked them up and they're available for agents to reference.

  • List all ModelConfigs in the kagent namespace.
kubectl get modelconfig -n kagent

You should see both bedrock-native and openrouter-config in the output.

  • Open the kagent dashboard in your browser (it should still be running at http://localhost:8082 from the previous step).
  • Click + Create, then select New Agent.
  • Look at the Model dropdown. Confirm that both bedrock-native and openrouter-config appear as selectable options.

Don't See Your Providers in the Dropdown?

If either ModelConfig is missing from the dropdown, check that the resources were created in the kagent namespace (not the default namespace). Run kubectl get modelconfig -A to see all ModelConfigs across namespaces.

Also verify the kagent controller pod is running with kubectl get pods -n kagent. Help me troubleshoot my ModelConfig not appearing in the kagent dashboard.

Your two LLM providers are ready. Next up, you'll create the four specialist agents that use these providers to investigate, fix, design, and build Kubernetes resources.

Create the Four Specialist Agents

Your two LLM providers are configured and ready. Now you need the agents that will actually do the work. In this step, you'll create four specialist agents, each with a distinct role, scoped tool set, and purpose-matched LLM provider.

This separation mirrors how real SRE teams operate. Investigators don't have production write access. Builders don't delete resources. By giving each agent only the tools it needs, you enforce the same boundaries at the agent level.

In This Step, Get Ready To:
  • Create a Diagnostician agent with read-only Kubernetes tools on Bedrock.
  • Create a Remediator agent (OpenRouter) and an Architect agent (Bedrock) for fixing and planning.
  • Create a Builder agent with Kubernetes creation tools and HITL approval gates.
Create the Diagnostician Agent on Bedrock

The Diagnostician is your investigation specialist. It has read-only Kubernetes tools and uses Bedrock for strong analytical reasoning. Its job is to examine cluster state, correlate evidence, and produce a structured diagnosis report with root cause, evidence, and severity.

  • Create a file called sre-diagnostician.yaml with the following content:
apiVersion: kagent.dev/v1alpha2
kind: Agent
metadata:
  name: sre-diagnostician
  namespace: kagent
spec:
  type: Declarative
  modelConfig: bedrock-native
  declarative:
    systemMessage: |
      You are an SRE Diagnostician agent. Your role is to systematically investigate
      Kubernetes cluster state using read-only tools. When given an incident report
      or investigation request:
      1. Gather evidence using kubectl-equivalent tools (get resources, describe, events, logs)
      2. Correlate findings across multiple resource types
      3. Produce a structured diagnosis with: root cause, supporting evidence, severity (critical/high/medium/low), and affected resources
      Never modify cluster state. You are strictly read-only.
    tools:
      - name: kagent.tools.k8s.GetResources
      - name: kagent.tools.k8s.DescribeResource
      - name: kagent.tools.k8s.GetEvents
      - name: kagent.tools.k8s.GetPodLogs
      - name: kagent.tools.k8s.GetResourceYAML
      - name: kagent.tools.k8s.GetAvailableAPIResources
  • Apply the manifest:
kubectl apply -f sre-diagnostician.yaml
  • Verify the agent was created:
kubectl get agent sre-diagnostician -n kagent

Why Read-Only Tools for the Diagnostician?

Notice that every tool assigned to the Diagnostician is a read operation: GetResources, DescribeResource, GetEvents, and so on. This agent can look at everything but change nothing.

This enforces the principle of least privilege at the agent level. An investigator should never accidentally "fix" something during diagnosis, which could mask the real root cause.

Why Bedrock for Reasoning, OpenRouter for Execution?

The Diagnostician and Architect both need strong analytical reasoning. They correlate evidence, identify root causes, and design remediation plans. Bedrock with Claude Sonnet excels at this kind of deep analysis.

The Orchestrator, Remediator, and Builder primarily follow instructions and execute tool calls. OpenRouter provides reliable instruction-following through its OpenAI-compatible API. Splitting providers this way also demonstrates kagent's multi-provider capabilities.

Create the Remediator and Architect Agents

The next two agents handle different sides of the fix process. The Remediator applies quick fixes to existing resources using mutation tools on OpenRouter. The Architect is a pure reasoning agent on Bedrock with no Kubernetes tools at all. It designs remediation plans that the Builder will implement later.

  • Create a file called sre-remediator.yaml with the following content:
apiVersion: kagent.dev/v1alpha2
kind: Agent
metadata:
  name: sre-remediator
  namespace: kagent
spec:
  type: Declarative
  modelConfig: openrouter-config
  declarative:
    systemMessage: |
      You are an SRE Remediator agent. Your role is to apply quick fixes to existing
      Kubernetes resources. You can patch configurations, update labels/annotations,
      and delete stuck resources. For each fix:
      1. Confirm what you are about to change
      2. Apply the minimal patch needed
      3. Report what was changed and verify the result
      Only fix existing resources. Do not create new resources.
    tools:
      - name: kagent.tools.k8s.PatchResource
      - name: kagent.tools.k8s.LabelResource
      - name: kagent.tools.k8s.AnnotateResource
      - name: kagent.tools.k8s.DeleteResource
    requireApproval:
      - kagent.tools.k8s.DeleteResource

What Is requireApproval?

The requireApproval field is kagent's Human-in-the-Loop (HITL) mechanism. When the agent tries to call a tool listed here, the kagent dashboard pauses execution and shows Approve/Reject buttons.

If you reject the action, the rejection reason is passed back to the LLM so the agent can adjust its approach. This gives you a safety gate on destructive operations.

  • Create a file called sre-architect.yaml with the following content:
apiVersion: kagent.dev/v1alpha2
kind: Agent
metadata:
  name: sre-architect
  namespace: kagent
spec:
  type: Declarative
  modelConfig: bedrock-native
  declarative:
    systemMessage: |
      You are an SRE Architect agent. Your role is to design remediation plans.
      You receive a diagnosis report and output a structured specification of:
      1. What resources need to be created
      2. Their configurations (keys, values, resource limits, etc.)
      3. The order of operations for applying them
      You do NOT generate YAML manifests. You produce a design document that
      the Builder agent will implement. Focus on correctness, completeness,
      and explaining your design decisions.
    tools: []

Why Does the Architect Have No Tools?

Not every agent needs Kubernetes tools. The Architect is a pure reasoning node in the pipeline. It takes a diagnosis as input and produces a design document as output. All of its value comes from the LLM's analytical capabilities, not from cluster interaction.

This demonstrates that agents can serve as thinking stages in a multi-agent workflow, not just action stages.

  • Apply both manifests:
kubectl apply -f sre-remediator.yaml
kubectl apply -f sre-architect.yaml
  • Verify both agents were created:
kubectl get agent sre-remediator sre-architect -n kagent
Create the Builder Agent on OpenRouter

The Builder is the final specialist. It takes the Architect's design plan, generates Kubernetes manifests, and applies them to the cluster. Because it creates new resources, every tool requires HITL approval.

  • Create a file called sre-builder.yaml with the following content:
apiVersion: kagent.dev/v1alpha2
kind: Agent
metadata:
  name: sre-builder
  namespace: kagent
spec:
  type: Declarative
  modelConfig: openrouter-config
  declarative:
    systemMessage: |
      You are an SRE Builder agent. Your role is to implement remediation plans
      designed by the Architect agent. For each resource in the plan:
      1. Generate the Kubernetes manifest based on the Architect's specification
      2. Confirm the manifest details before applying
      3. Apply the manifest to the cluster
      4. Verify the resource was created successfully
      Always follow the Architect's design exactly. Do not improvise or skip resources.
    tools:
      - name: kagent.tools.k8s.ApplyManifest
      - name: kagent.tools.k8s.CreateResource
      - name: kagent.tools.k8s.CreateResourceFromUrl
    requireApproval:
      - kagent.tools.k8s.ApplyManifest
      - kagent.tools.k8s.CreateResource
      - kagent.tools.k8s.CreateResourceFromUrl

What does this manifest do?

The Builder has three creation tools: ApplyManifest, CreateResource, and CreateResourceFromUrl. All three are listed in requireApproval, meaning every single creation operation will pause for your approval in the dashboard. This is the strictest HITL configuration possible.

  • Apply the manifest:
kubectl apply -f sre-builder.yaml
  • Verify the Builder agent was created:
kubectl get agent sre-builder -n kagent
  • Confirm all four specialist agents are running:
kubectl get agent -n kagent

You should see all four agents listed: sre-diagnostician, sre-remediator, sre-architect, and sre-builder.

Your four specialists are deployed and ready. Next up, you'll create the Orchestrator agent that ties them all together using kagent's agents-as-tools pattern.

Create the Orchestrator with Agents-as-Tools

You now have four specialist agents deployed, each with a focused role, scoped tools, and a purpose-matched LLM provider. The missing piece is the coordinator that ties them all together. In this step, you'll create the Orchestrator agent, which references all four specialists as callable tools using kagent's agents-as-tools pattern.

In This Step, Get Ready To:
  • Create the Orchestrator agent that references all four specialist agents as tools.
  • Verify the full five-agent pipeline is visible and correctly wired in the kagent dashboard.
Create the Orchestrator Agent

The Orchestrator is the entry point for your entire incident response pipeline. Instead of having its own Kubernetes tools, it references the four specialist agents as callable tools. When the Orchestrator receives an incident report, it decides which specialist to delegate to based on the type of issue. It always starts with the Diagnostician for live cluster validation, then routes quick fixes to the Remediator or structural changes through the Architect and Builder.

This is kagent's agents-as-tools pattern in action. The Orchestrator sees each specialist as a tool it can call. When it "calls" the Diagnostician tool, the Diagnostician runs independently with its own model (Bedrock) and its own read-only K8s tools, then returns results back to the Orchestrator.

  • Create a file called sre-orchestrator.yaml with the following content:
apiVersion: kagent.dev/v1alpha2
kind: Agent
metadata:
  name: sre-orchestrator
  namespace: kagent
spec:
  type: Declarative
  modelConfig: openrouter-config
  declarative:
    systemMessage: |
      You are an SRE Orchestrator coordinating incident response across specialist agents.

      When given an incident report or user request:
      1. ALWAYS delegate to sre-diagnostician first for live cluster validation.
      2. Based on the Diagnostician's findings, determine the fix path:
         - For quick fixes (wrong image tags, misconfigurations on existing resources): delegate to sre-remediator.
         - For structural changes (missing resources that need to be created): delegate to sre-architect for a design plan, then delegate that plan to sre-builder for implementation.
      3. After all fixes are applied, summarize the full outcome including what was found, what was fixed, and the current cluster state.

      You do NOT have direct Kubernetes tools. You coordinate by delegating to the right specialist.
    tools:
      - type: Agent
        agent:
          name: sre-diagnostician
      - type: Agent
        agent:
          name: sre-remediator
      - type: Agent
        agent:
          name: sre-architect
      - type: Agent
        agent:
          name: sre-builder

What Makes This Different from a Regular Agent?

Notice the tools section. Instead of listing built-in Kubernetes tools like kagent.tools.k8s.GetResources, the Orchestrator lists other agents using type: Agent with an agent.name reference. Each specialist agent becomes a callable tool that the Orchestrator's LLM can invoke. The specialist then runs independently with its own model, system prompt, and tools before returning results.

Why OpenRouter for the Orchestrator?

The Orchestrator's job is routing and coordination, not deep analysis. It reads an incident report, decides which specialist to call, and summarizes results. This is reliable instruction-following work that doesn't need the heavier reasoning of Bedrock. The Diagnostician and Architect use Bedrock because they need strong analytical reasoning for root cause analysis and architectural design decisions.

  • Apply the Orchestrator manifest:
kubectl apply -f sre-orchestrator.yaml
  • Verify the agent was created:
kubectl get agent sre-orchestrator -n kagent

You should see the sre-orchestrator agent listed with a status indicating it's ready.

Verify All Five Agents in the Dashboard

With all five agents deployed, it's time to confirm the full pipeline is correctly wired. The dashboard gives you a visual way to verify that the Orchestrator recognizes its four specialist agents as tools.

  • Run the following command to list all agents and confirm all five are present:
kubectl get agents -n kagent

You should see all five agents listed: sre-orchestrator, sre-diagnostician, sre-remediator, sre-architect, and sre-builder.

  • Open the kagent dashboard in your browser at http://localhost:8082.
  • Confirm all five agents appear in the agent list.
  • Click on sre-orchestrator to open its details.
  • In the Agent Details panel, verify that the four specialist agents are listed as tools: sre-diagnostician, sre-remediator, sre-architect, and sre-builder.

What Should I See?

The Orchestrator's Agent Details panel should show four tools, each with type: Agent. These are not Kubernetes tools. They are references to your other agents. This confirms the agents-as-tools wiring is complete.

An Agent Is Missing or Not Showing as Ready?

Check that all agent YAMLs use the correct namespace (kagent) and API version (kagent.dev/v1alpha2). Also verify that the modelConfig names match exactly what you created in Step 2 (bedrock-native or openrouter-config). You can describe a specific agent for more detail:

kubectl describe agent sre-orchestrator -n kagent

If you're still stuck, help me debug why my agent isn't showing as ready.

Your five-agent pipeline is fully wired. Next up, you'll deploy a deliberately broken workload and use OpenSRE to generate an incident report that kicks off the entire multi-agent pipeline.

Deploy a Broken Workload and Generate an OpenSRE Report

Your five-agent pipeline is wired up and ready to respond to incidents. Now you need an actual incident to respond to. In this step, you'll deploy a deliberately broken application with two distinct issues, then use OpenSRE to generate a structured investigation report that will feed into your Orchestrator.

The two issues are carefully chosen to exercise different paths in your pipeline. One is a quick fix that the Remediator can patch directly. The other requires the Architect to design a solution and the Builder to implement it.

In This Step, Get Ready To:
  • Deploy a multi-issue broken workload with a wrong image tag and a missing ConfigMap.
  • Generate a structured OpenSRE investigation report from the broken workload.
  • Review the report structure that will serve as input to the kagent Orchestrator.
Deploy a Multi-Issue Broken Workload

You need a workload that breaks in two different ways. The first issue is a wrong image tag (nginx:nonexistent-tag), which causes an ImagePullBackOff error. This is a quick fix that the Remediator can patch directly. The second issue is a reference to a ConfigMap named app-config that doesn't exist. This causes a CreateContainerConfigError and requires the Architect to design the ConfigMap spec, then the Builder to create it.

  • Create a namespace for the broken workload.
kubectl create namespace test-app
  • Create a file called broken-workload.yaml with the following Deployment manifest.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: broken-app
  namespace: test-app
  labels:
    app: broken-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: broken-app
  template:
    metadata:
      labels:
        app: broken-app
    spec:
      containers:
        - name: nginx
          image: nginx:nonexistent-tag
          ports:
            - containerPort: 80
          envFrom:
            - configMapRef:
                name: app-config

What does this manifest do?

This Deployment has two deliberate problems. The image: nginx:nonexistent-tag line references a tag that doesn't exist on Docker Hub, so Kubernetes can't pull the image. The envFrom.configMapRef.name: app-config line references a ConfigMap that you never created. Both issues prevent the pod from starting.

  • Apply the broken workload to your cluster.
kubectl apply -f broken-workload.yaml
  • Confirm both issues are present by checking the pod status.
kubectl get pods -n test-app

You should see the pod in an ImagePullBackOff or CreateContainerConfigError state. Kubernetes surfaces whichever error it encounters first, but both issues exist in the manifest.

  • Run a describe on the pod to see detailed error events for both issues.
kubectl describe pod -n test-app -l app=broken-app

What should I see?

The Events section of the describe output should show errors related to pulling the image (Failed to pull image "nginx:nonexistent-tag") and potentially errors about the missing ConfigMap (configmap "app-config" not found). Both errors confirm the workload is broken in the ways you intended.

Generate an OpenSRE Investigation Report

OpenSRE can investigate Kubernetes incidents and produce structured reports with correlated findings, root cause analysis, and recommended next steps. You'll use it to generate a report that becomes the input context for your kagent Orchestrator.

  • Run the OpenSRE investigate command using the built-in Kubernetes alert fixture as a starting template.
opensre investigate -i tests/e2e/kubernetes/fixtures/datadog_k8s_alert.json

What does this command do?

The opensre investigate command takes an alert fixture (a JSON file representing a Kubernetes alert) and runs an LLM-powered investigation against your cluster. It correlates events, pod statuses, and resource states to produce a structured report.

Report doesn't match your test-app namespace?

The built-in Datadog alert fixture is a generic template. If the investigation output doesn't reference your test-app namespace or broken workload, you may need to run OpenSRE in interactive mode or create a custom alert JSON that points to your namespace. Try running opensre investigate without the -i flag and provide details interactively. Help me create a custom OpenSRE alert fixture for my test-app namespace

  • Save the investigation output. You'll paste this into the kagent dashboard in the next step.
Review the OpenSRE Report

Before feeding the report to the Orchestrator, take a moment to understand its structure. The OpenSRE investigation report gives an SRE (or an AI agent) everything needed to act on an incident.

  • Read through the report and identify the following sections:
  • Root cause analysis identifies the underlying problems causing the incident.
  • Correlated evidence links specific Kubernetes events, pod statuses, and resource states to each finding.
  • Severity assessment rates the impact of each issue.
  • Recommended next steps suggests actions to resolve the issues.

Why use a structured report instead of just describing the problem?

The Orchestrator agent works best with structured input. A well-organized report with distinct findings, evidence, and severity levels helps the Orchestrator make better delegation decisions. It can route quick fixes to the Remediator and structural changes to the Architect based on the report's categorization.

Your broken workload is deployed and your investigation report is ready. Next up, you'll paste this report into the Orchestrator and watch all five agents work together to diagnose, design, and fix the issues.

Run the Multi-Agent Pipeline

Your five agents are deployed, your providers are configured, and a broken workload is waiting in the cluster. Now it is time to bring everything together. In this step, you will feed the OpenSRE investigation report into the Orchestrator and watch the full multi-agent pipeline diagnose, design, and fix the broken workload across all four specialist agents.

In This Step, Get Ready To:
  • Feed the OpenSRE investigation report to the Orchestrator agent via the kagent dashboard.
  • Observe the multi-agent delegation chain and approve destructive operations through HITL gates.
  • Verify that the broken workload is fully fixed and running healthy.
Feed the OpenSRE Report to the Orchestrator

The Orchestrator agent is your pipeline's entry point. It receives an incident report, then decides which specialist agents to delegate to based on the type of issues found. Your OpenSRE investigation report from the previous step provides the structured context the Orchestrator needs to begin coordinating.

  • Open the kagent dashboard at http://localhost:8082.
  • Select the sre-orchestrator agent from the agent list.
  • Paste the following prompt into the chat input, replacing the placeholder with your actual OpenSRE report output:
Here is an incident investigation report from OpenSRE:

[PASTE YOUR OPENSRE REPORT HERE]

Please validate these findings against the live cluster in the test-app namespace, then fix all identified issues.

What happens when you send this prompt?

The Orchestrator reads the incident report and identifies the types of issues described. Its first move is always to delegate to the Diagnostician for live cluster validation. The Orchestrator does not trust the report blindly. It uses the Diagnostician's read-only Kubernetes tools to confirm whether the reported issues actually exist in the cluster right now.

After sending the prompt, watch the chat panel. You should see the Orchestrator begin delegating to the sre-diagnostician agent first. The Diagnostician will use its read-only tools like kagent.tools.k8s.GetResources, kagent.tools.k8s.DescribeResource, and kagent.tools.k8s.GetEvents to validate the OpenSRE findings against the live cluster state.

Observe the Delegation and HITL Approvals

This is where the multi-agent pipeline comes to life. The Orchestrator coordinates a sequence of delegations based on the Diagnostician's findings. Each delegation triggers a different specialist agent with its own LLM provider and tool set. You will need to actively approve certain operations through the HITL (Human-in-the-Loop) approval gates you configured earlier.

Watch for the following delegation sequence in the dashboard chat:

  • The Diagnostician validates both issues: the wrong image tag (nginx:nonexistent-tag causing ImagePullBackOff) and the missing ConfigMap (app-config causing CreateContainerConfigError). It returns a structured diagnosis to the Orchestrator.
  • The Orchestrator delegates the image tag fix to the sre-remediator. This is a quick patch operation. When the Remediator calls kagent.tools.k8s.PatchResource, you will see an Approve / Reject prompt if it triggers a tool with requireApproval. Click Approve to allow the patch.
  • The Orchestrator delegates the missing ConfigMap issue to the sre-architect. The Architect is a pure reasoning agent with no Kubernetes tools. It designs a specification for the ConfigMap that needs to be created, including its name, namespace, and data keys.
  • The Orchestrator passes the Architect's design plan to the sre-builder. The Builder generates a Kubernetes manifest and calls kagent.tools.k8s.ApplyManifest or kagent.tools.k8s.CreateResource. All three of the Builder's tools have requireApproval enabled. Click Approve to allow the ConfigMap creation.

Pro tip: expand the tool call details

Click on the Arguments and Results sections in each tool call within the chat. This lets you see exactly what data the Orchestrator passed to each specialist agent and what each agent returned. It is the best way to understand the full delegation chain.

Why does the Orchestrator split the work this way?

The image tag issue is a quick fix. The existing Deployment just needs a patch to correct the tag. That is the Remediator's specialty. The missing ConfigMap is a structural problem. A new resource needs to be designed and created from scratch. That requires the Architect to plan and the Builder to execute. This split mirrors how real SRE teams operate: quick fixes go through a different approval path than new resource creation.

Agent not delegating or stuck?

If the Orchestrator is not delegating to specialist agents, check that all five agents show as Ready in the dashboard. Also verify that the Orchestrator's Agent CRD lists all four specialists in its tools array with type: Agent. You can inspect this with kubectl get agent sre-orchestrator -n kagent -o yaml. If a specialist agent fails mid-execution, check its logs with kubectl logs -n kagent -l app=kagent. Help me debug my agent delegation issue.

Verify the Fix

After the pipeline completes, the Orchestrator should provide a summary of all actions taken. Now verify that the broken workload is actually fixed by checking the cluster state directly.

  • Check that the pods are running:
kubectl get pods -n test-app

What should I see?

All pods in the test-app namespace should show a Running status. Previously they were stuck in ImagePullBackOff or CreateContainerConfigError.

  • Verify that the missing ConfigMap was created by the Builder:
kubectl get configmap app-config -n test-app

The ConfigMap app-config should now exist in the test-app namespace. This is the resource the Architect designed and the Builder created.

  • Back in the dashboard, review the full conversation thread for the sre-orchestrator.

Scroll through the complete conversation to trace the delegation chain: Orchestrator to Diagnostician for validation, then Orchestrator to Remediator for the image patch, then Orchestrator to Architect for the ConfigMap design, and finally Orchestrator to Builder for the ConfigMap creation. This is the agents-as-tools pattern in action.

Why is this pattern powerful?

Each agent in this pipeline has scoped responsibilities and scoped tool access. The Diagnostician can only read. The Remediator can patch but not create. The Architect can reason but has no cluster access at all. The Builder can create but every action requires human approval. This separation of concerns mirrors production SRE team structures where investigators, fixers, and builders have different levels of access.

Your multi-agent SRE pipeline just diagnosed, designed, and fixed a multi-issue workload across five coordinated agents. Next up, take on the Secret Mission to give your Diagnostician long-term memory across investigations.

Secret mission

Enable Agent Memory on the Diagnostician

What if your Diagnostician could remember past investigations? Right now, every time the pipeline runs, the Diagnostician starts from scratch. Enable long-term memory so it retains context across investigations and can reference past findings when similar issues recur.

Before you go

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. The kagent installation itself is free, and LLM costs only accrue when you send queries through the agents.

Cost warning

The multi-agent pipeline involves multiple LLM calls (one per agent delegation), so each run costs more than a single-agent setup. Bedrock charges per invocation based on token usage (Claude Sonnet ~$3/$15 per million input/output tokens). OpenRouter charges per token based on the model chosen. If you leave minikube running and continue sending queries, expect costs to accumulate. Stop sending queries or pause minikube to avoid further charges.

Resources you used:

  • kagent installation in the kagent namespace (controller, UI, PostgreSQL, CRDs).
  • Kubernetes secrets: bedrock-credentials and kagent-openrouter in the kagent namespace.
  • ModelConfig resources: bedrock-native and openrouter-config.
  • Agent resources: sre-orchestrator, sre-diagnostician, sre-remediator, sre-architect, sre-builder.
  • Test namespace and workload: test-app.
  • OpenSRE CLI installed locally.

Keep everything running

No action needed. Choose this if you're still actively building or want to experiment with additional agent configurations and workloads.

  • Your minikube cluster, kagent installation, and all agent CRDs will remain available.
  • No ongoing costs unless you send queries through the agents.
  • You can reopen the dashboard anytime with kagent dashboard.

Pause - I'll come back to this later

Stop minikube to free up local memory and CPU. All your cluster state, CRDs, and secrets will be preserved on disk.

  • Stop minikube by running minikube stop.
  • When you're ready to resume, run minikube start and all your agents, ModelConfigs, and secrets will still be there.
  • No LLM costs while minikube is stopped.

Delete - I don't want to use this again

Remove all project resources and start fresh. This deletes everything you created during this project.

  • Delete the test namespace and its workload by running kubectl delete namespace test-app.
  • Delete the credential secrets by running kubectl delete secret bedrock-credentials kagent-openrouter -n kagent.
  • Uninstall kagent (controller, UI, PostgreSQL, and all CRDs) by running kagent uninstall.
  • Optionally, delete the minikube cluster entirely by running minikube delete.
  • Optionally, uninstall the OpenSRE CLI by running pipx uninstall opensre or brew uninstall opensre depending on how you installed it.

Mission accomplished

That's a Wrap!

Nice work! You just built a production-style, five-agent SRE operations pipeline on Kubernetes using kagent, with two LLM providers and OpenSRE powering the incident trigger.

Here's what you learned:

  • Configure multi-provider LLM backends as Kubernetes CRDs, using native Bedrock for reasoning-heavy agents and OpenRouter as a BYO OpenAI-compatible provider for execution-focused agents.
  • Build a five-agent orchestration pipeline using kagent's agents-as-tools pattern, with an Orchestrator delegating to specialist Diagnostician, Remediator, Architect, and Builder agents based on issue type.
  • Enforce HITL approval gates on destructive and creation operations, separating read-only investigation from write/create actions across agents.
  • Enable agent memory on the Diagnostician so it retains context across investigations and can reference past findings when similar issues recur.

Ready to quiz yourself?