Build a Kubernetes Agent with kagent

Build a human-approved agent that diagnoses and repairs Kubernetes incidents.

Introduction

30 Second Summary

When a service fails, the clock starts immediately. A rushed fix can cause more damage than the original problem.

In this project, you will build a local Kubernetes incident-response agent with kagent. Ollama runs the local model while Model Context Protocol tools gather evidence before every approval-gated patch.

What You'll Build

You'll demonstrate a failing workload that your responder diagnoses from live evidence before asking whether its proposed repair can run.

By the end of this project, you'll have:

A repeatable incident lab that shows a healthy fragile-app healthy log before switching into ImagePullBackOff or CrashLoopBackOff on demand.
An evidence-driven responder that inspects Pods, events, logs, and manifests to explain each root cause.
A human approval workflow that shows read tool calls, proposed patch arguments, and completed repairs in the dashboard.
Secret Mission: Recreate a crash-loop incident. Reject the proposed patch with a reason. Prove that the workload remains unchanged.

Are there any prerequisites?

The prerequisites are a Linux workstation with Rancher Desktop running Kubernetes, Ollama with qwen3:4b available, and enough local resources for the kagent control plane.

The project includes Helm setup if it is missing.

Before We Start

Your local stack pairs a Kubernetes cluster from Rancher Desktop's containerd runtime with Ollama's qwen3:4b local model. Your enforced boundary pairs namespace-scoped RBAC for the kagent controller plus kagent-tools service account in kagent plus incident-lab with the incident-responder Agent policy plus human approval review for patches to the fragile-app Deployment.

Prepare the Local AI and Install kagent

Your local Ollama model can support kagent only when a Kubernetes Pod can reach the host service. This step builds that network path before the incident drills begin.

The Helm chart scopes the controller separately from the tool server. You will configure both RBAC layers so their access stops at the kagent and incident-lab namespaces.

In this step, get ready to:

Verify the local cluster plus the required command-line tools.
Expose the local Ollama API to Pods through the Rancher Desktop host address.
Install kagent with namespace-scoped permissions.

Validate the local prerequisites

The installation depends on a connected cluster plus the downloaded local model. Helm 4.2.4 supplies the pinned chart installation workflow.

Check the cluster connection plus the local model by running these commands:

kubectl get nodes
ollama ls
helm version

What Do These Checks Prove?

The first command confirms that kubectl can reach the Rancher Desktop cluster.
The second command lists the models already available to Ollama.
The third command reports the installed Helm version.

You should see a node with a Ready status. The Ollama list should include qwen3:4b.

Choose the tab that matches the result from the Helm check.

✔️ I see version 4.2.4

Helm is ready. Keep this terminal available for the remaining setup commands.

ⓧ I see a different version

This project pins Helm to 4.2.4 so every chart command follows the same tested behavior. The installer may request administrator access when it copies the binary.

Install the pinned Helm version by running these commands:

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-4
chmod 700 get_helm.sh
./get_helm.sh --version v4.2.4

What Does the Installer Do?

The first command downloads the official Helm installer. The next commands make it executable before requesting version v4.2.4.

The installer verifies the downloaded checksum before copying the Helm binary.

Confirm the installed version by running this command:

helm version

What Should You See?

The output should identify Helm version 4.2.4.

Still Seeing a Different Version?

Your shell may still be resolving an older Helm binary earlier in its command path. Restart the terminal before checking the version again.

If the installer cannot copy the binary, confirm that you approved its administrator prompt.

Help me diagnose my Helm version.

ⓧ Command not found

Helm is missing from your command path. The official installer downloads the requested release before verifying its checksum.

Install Helm 4.2.4 by running these commands:

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-4
chmod 700 get_helm.sh
./get_helm.sh --version v4.2.4

What Does the Installer Do?

The first command downloads the official Helm installer. The next commands make it executable before requesting version v4.2.4.

The installer verifies the downloaded checksum before copying the Helm binary.

Confirm that Helm is now available by running this command:

helm version

What Should You See?

The output should identify Helm version 4.2.4.

Is Helm Still Missing?

Restart the terminal so your shell reloads its command path. Run the version check again after the restart.

If the installation stopped early, check the terminal for a denied administrator prompt.

Help me finish the Helm installation.

Make Ollama reachable from the cluster

Ollama listens on 127.0.0.1 by default. A Pod inside Rancher Desktop reaches the host through host.docker.internal.

This override makes Ollama listen on every host interface. You will remove the temporary setting during cleanup.

Open the Ollama systemd override editor by running this command:

sudo systemctl edit ollama.service

What Does This Command Open?

Systemd opens a service drop-in editor for Ollama. A drop-in preserves the original service file while adding your network setting.

Add this override in the editor:

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"

What Does This Override Change?

The environment setting binds Ollama to 0.0.0.0:11434. This lets the Rancher Desktop virtual machine reach the API through the host alias.

Save the override in the editor.
Exit the editor.
Reload systemd plus restart Ollama by running these commands:

sudo systemctl daemon-reload
sudo systemctl restart ollama

What Does the Restart Apply?

The reload makes systemd read the new drop-in. The restart launches Ollama with the expanded listening address.

Does Ollama Fail to Restart?

Reopen the override and confirm that the section name is [Service]. Confirm that the environment line contains matching double quotes.

Help me diagnose the Ollama service restart.

Create the incident workspace namespace by running this command:

kubectl create namespace incident-lab

What Does This Namespace Hold?

The incident-lab namespace isolates the fragile workload from the kagent control plane. The terminal should report that the namespace was created.

Before you run the probe, do you expect a Pod to find the local model through the Rancher Desktop host alias?

Test the Pod-to-host connection by running this disposable probe:

kubectl run ollama-probe --image=busybox:1.37.0 --restart=Never --rm -i -- wget -qO- http://host.docker.internal:11434/api/tags

What Does the Probe Test?

The command starts a temporary BusyBox Pod with image busybox:1.37.0.
The Pod requests the Ollama model-list endpoint through http://host.docker.internal:11434/api/tags.
The removal flag deletes the probe after the request finishes.

The JSON response should include qwen3:4b. That result proves the cluster can reach the tool-capable local model.

Does the Probe Fail?

Confirm that Ollama still lists qwen3:4b on the host. A missing model cannot appear in the API response.

Confirm that the systemd override uses 0.0.0.0:11434. A loopback-only bind prevents the Pod from connecting.

Help me diagnose the Ollama probe.

Configure and install kagent

A YAML values file makes the installation repeatable. It selects the local model plus both namespace boundaries.

The same file hardens the Pods plus disables the components outside this incident lab. The tool server exposes only the Kubernetes provider.

Create kagent-values.yaml on your Desktop with your Linux text editor.
Add the provider plus controller security settings:

providers:
  default: ollama
  ollama:
    provider: Ollama
    model: "qwen3:4b"
    config:
      host: "http://host.docker.internal:11434"
      options:
        num_ctx: "8192"

rbac:
  namespaces:
    - kagent
    - incident-lab

podSecurityContext:
  runAsNonRoot: true
  seccompProfile:
    type: RuntimeDefault

securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop:
      - ALL

What Does This Configuration Control?

The provider settings connect the default model configuration to qwen3:4b through the Rancher Desktop host alias.
The namespace list creates controller Roles inside kagent plus incident-lab.
The security contexts prevent privilege escalation plus remove Linux capabilities.

Save kagent-values.yaml.
Confirm that both namespace names appear under rbac.namespaces.

Do the YAML Lines Look Misaligned?

Use spaces for every indentation level. Confirm that ollama sits below providers.

Help me check this YAML section.

Add the component controls below the existing security settings:

kmcp:
  enabled: false

grafana-mcp:
  enabled: false

querydoc:
  enabled: false

k8s-agent:
  enabled: false
kgateway-agent:
  enabled: false
istio-agent:
  enabled: false
promql-agent:
  enabled: false
observability-agent:
  enabled: false
argo-rollouts-agent:
  enabled: false
helm-agent:
  enabled: false
cilium-policy-agent:
  enabled: false
cilium-manager-agent:
  enabled: false
cilium-debug-agent:
  enabled: false

Why Disable These Components?

These settings remove the bundled sample agents plus the unused Model Context Protocol services. The lab keeps its compute focused on one responder.

Save kagent-values.yaml.
Confirm that every component in this section has enabled: false.

Add the tool-server configuration at the end of the file:

kagent-tools:
  enabled: true
  fullnameOverride: kagent-tools
  rbac:
    create: true
    namespaces:
      - kagent
      - incident-lab
    readOnly: false
  tools:
    enabledTools:
      - k8s
  podSecurityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  securityContext:
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: true
    capabilities:
      drop:
        - ALL

What Does the Tool-Server Section Enforce?

The full-name override gives the service account the predictable name kagent-tools.
The nested namespace list creates Roles inside the same two allowed namespaces.
Write mode allows patch operations inside those namespaces. The later Agent definition selects one patch tool plus an approval gate.
The provider list enables only the k8s tool provider.

Save kagent-values.yaml.
Confirm that kagent-tools.rbac.namespaces contains kagent plus incident-lab.

Is the Tool Server Missing Its Scope?

Confirm that the second namespaces list is nested below kagent-tools.rbac. A top-level list scopes the controller only.

Help me check the kagent-tools scope.

✔️ Awesome, I've got everything!

Your saved values file now scopes both RBAC layers plus the local Ollama provider.

ⓧ I'd like to double check the full code

providers:
  default: ollama
  ollama:
    provider: Ollama
    model: "qwen3:4b"
    config:
      host: "http://host.docker.internal:11434"
      options:
        num_ctx: "8192"

rbac:
  namespaces:
    - kagent
    - incident-lab

podSecurityContext:
  runAsNonRoot: true
  seccompProfile:
    type: RuntimeDefault

securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop:
      - ALL

kmcp:
  enabled: false

grafana-mcp:
  enabled: false

querydoc:
  enabled: false

k8s-agent:
  enabled: false
kgateway-agent:
  enabled: false
istio-agent:
  enabled: false
promql-agent:
  enabled: false
observability-agent:
  enabled: false
argo-rollouts-agent:
  enabled: false
helm-agent:
  enabled: false
cilium-policy-agent:
  enabled: false
cilium-manager-agent:
  enabled: false
cilium-debug-agent:
  enabled: false

kagent-tools:
  enabled: true
  fullnameOverride: kagent-tools
  rbac:
    create: true
    namespaces:
      - kagent
      - incident-lab
    readOnly: false
  tools:
    enabledTools:
      - k8s
  podSecurityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  securityContext:
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: true
    capabilities:
      drop:
        - ALL

Comparison Checkpoint

Compare the indentation plus the order of each section with your saved file. The two namespace lists must contain the same names.

The values file is ready to drive both chart installations. The first install can spend a few minutes downloading chart content plus container images.

Return to the terminal session from earlier.
Move to the Desktop location containing kagent-values.yaml.
Install the CRDs plus the kagent release by running these commands:

helm install kagent-crds oci://ghcr.io/kagent-dev/kagent/helm/kagent-crds --namespace kagent --create-namespace --version 0.9.12 --values kagent-values.yaml
helm install kagent oci://ghcr.io/kagent-dev/kagent/helm/kagent --namespace kagent --version 0.9.12 --values kagent-values.yaml

What Do These Installs Create?

The first command installs the kagent custom resource definitions before creating the kagent namespace. The second command installs the controller plus its selected supporting services.

Both commands pin chart version 0.9.12. The main release reads the local provider plus the two RBAC scopes from kagent-values.yaml.

Does a Helm Install Fail?

Confirm that the terminal is at the Desktop location containing kagent-values.yaml. A different location prevents Helm from reading the values file.

Check the reported YAML line for indentation. Fix the line before running the failed install command again.

Help me diagnose the kagent Helm installation.

Check the kagent Pods by running this command:

kubectl get pods -n kagent

What Should the Pod List Show?

The list covers the controller plus the tool server plus the UI plus supporting Pods. Every listed Pod needs to report Ready before you continue.

Image downloads can keep some Pods in a starting state for a few minutes. That delay is expected on the first installation.

Repeat the Pod check until every listed Pod reports Ready.

Before checking the permission boundary, which namespace do you expect the tool server to patch?

Test the allowed namespace plus the denied namespace by running these commands:

kubectl auth can-i patch deployments.apps --as=system:serviceaccount:kagent:kagent-tools -n incident-lab
kubectl auth can-i patch deployments.apps --as=system:serviceaccount:kagent:kagent-tools -n default

What Does the Boundary Check Prove?

The first command should print yes. The service account can patch Deployments inside incident-lab.

The second command should print no. Kubernetes RBAC blocks the same permission inside default.

Do Both Checks Print Yes?

Recheck the nested namespace list under kagent-tools.rbac. An empty list creates cluster-scoped permissions.

Reinstall the kagent release after correcting the values file.

Help me diagnose the RBAC boundary.

Start a second terminal session in the terminal app from earlier.
Expose the kagent dashboard through the second terminal by running this command:

kubectl port-forward -n kagent svc/kagent-ui 8080:8080

What Does the Port Forward Do?

The command maps local port 8080 to the kagent UI service. Keep this terminal session running while you use the dashboard.

Enter http://localhost:8080 in your browser address bar.
Load the dashboard.

You should see the kagent dashboard load. Your local control plane can now reach Ollama through the cluster.

Does the Dashboard Fail to Load?

Confirm that the port-forward terminal is still running. Closing that process removes the local dashboard connection.

Confirm that the Pod list shows the UI Pod as Ready. A starting or failed UI Pod cannot accept the forwarded connection.

Help me diagnose the kagent dashboard connection.

Your local AI control plane is ready with a tested namespace boundary. Next, you will launch a healthy workload before breaking its image on purpose.

Launch the App and Break Its Image

Your local kagent stack can now reach the model. The next ingredient is a controlled incident that gives the responder something concrete to investigate.

A reliable Kubernetes investigation starts with a known-good Deployment. You will observe its healthy Pod before triggering a repeatable image failure.

In this step, get ready to:

Launch the fragile workload from a reusable manifest.
Confirm the known-good behavior through its logs.
Trigger an invalid-image incident in the replacement Pod.

Create the healthy Deployment

A YAML manifest makes the workload repeatable. The container uses BusyBox to print a health message every 30 seconds.

Create fragile-app.yaml in the folder containing kagent-values.yaml by copying this runnable baseline into a new file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: fragile-app
  namespace: incident-lab
spec:
  replicas: 1
  selector:
    matchLabels:
      app: fragile-app
  template:
    metadata:
      labels:
        app: fragile-app
    spec:
      containers:
        - name: fragile-app
          image: busybox:1.37.0
          command:
            - /bin/sh
            - -c
          args:
            - |
              while true; do
                echo "fragile-app healthy"
                sleep 30
              done

What Does This Manifest Do?

The manifest declares one fragile-app replica in incident-lab.
The selector connects the Deployment to Pods carrying the app: fragile-app label.
The shell loop keeps the container running. Its repeated message provides a known-good signal.

Save fragile-app.yaml.
Apply the runnable manifest by running this command:

kubectl apply -f fragile-app.yaml

What Does This Command Do?

The command sends the manifest to the connected cluster. Kubernetes creates the Deployment in incident-lab.

The terminal confirms that the manifest was accepted. Your first version of the workload is now running in the cluster.

Was the Manifest Rejected?

Check the indentation beneath spec plus containers. YAML nesting depends on consistent spaces.

Confirm that incident-lab still exists. The manifest places the Deployment directly in that namespace.

Help me diagnose the initial fragile-app manifest.

The runnable core establishes the workload behavior. The next edit adds the final rollout settings plus the Pod-level security configuration.

In fragile-app.yaml, select everything from apiVersion: apps/v1 through image: busybox:1.37.0.
Replace the selected section with this expanded configuration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: fragile-app
  namespace: incident-lab
  labels:
    app: fragile-app
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: fragile-app
  template:
    metadata:
      labels:
        app: fragile-app
    spec:
      securityContext:
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: fragile-app
          image: busybox:1.37.0
          imagePullPolicy: IfNotPresent

What Does This Configuration Add?

The metadata label gives the Deployment a consistent identity.
The Recreate strategy removes the old Pod before starting its replacement.
The RuntimeDefault seccomp profile limits the system calls available to the Pod.
The IfNotPresent pull policy reuses the local image when it is already available.

Save fragile-app.yaml.
Apply the expanded configuration by running this command:

kubectl apply -f fragile-app.yaml

What Changed in the Cluster?

Kubernetes accepts the new Deployment settings. The Pod template now carries the rollout behavior plus the Pod-level security configuration.

The accepted update proves that the expanded YAML remains valid. The workload keeps its original image plus its healthy command.

Was the Update Rejected?

Confirm that the replacement starts at the first line of fragile-app.yaml. The existing command section must remain directly below imagePullPolicy.

Help me check the expanded Deployment configuration.

The final manifest also constrains the container itself. Resource limits keep the lab lightweight while the container security context removes unnecessary privileges.

In fragile-app.yaml, find the final done line in the args block.
Add the remaining configuration directly below that line:

          resources:
            requests:
              cpu: 10m
              memory: 16Mi
            limits:
              cpu: 100m
              memory: 32Mi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop:
                - ALL

What Do These Limits Enforce?

The resource requests reserve a small amount of CPU plus memory for the container.
The resource limits cap the CPU plus memory available to the container.
The security context blocks privilege escalation. It also makes the container filesystem read-only.
Dropping ALL removes every Linux capability from the container.

Save fragile-app.yaml.
Apply the completed manifest by running this command:

kubectl apply -f fragile-app.yaml

What Does This Apply Confirm?

Kubernetes stores the resource limits plus the hardened container settings. The changed Pod template causes the Deployment to replace its Pod.

Watch the replacement Pod by running this command:

kubectl get pods -n incident-lab -w

What Should You See?

The watch prints each Pod state as the replacement starts. Continue when the new fragile-app Pod reports that it is running.

Stop the watch after the replacement Pod is running.

That is the workload foundation complete. Your hardened Deployment can start successfully from the saved baseline.

Does the Replacement Pod Stay Unready?

Check that resources plus the container securityContext align with command inside the same container entry.

Confirm that the Pod-level securityContext remains directly beneath the Pod spec.

Help me diagnose the hardened fragile-app Pod.

✔️ Awesome, I've got everything!

Great. Your saved fragile-app.yaml now contains the complete healthy baseline.

ⓧ I'd like to double check the full code

Compare your saved fragile-app.yaml with this complete file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: fragile-app
  namespace: incident-lab
  labels:
    app: fragile-app
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: fragile-app
  template:
    metadata:
      labels:
        app: fragile-app
    spec:
      securityContext:
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: fragile-app
          image: busybox:1.37.0
          imagePullPolicy: IfNotPresent
          command:
            - /bin/sh
            - -c
          args:
            - |
              while true; do
                echo "fragile-app healthy"
                sleep 30
              done
          resources:
            requests:
              cpu: 10m
              memory: 16Mi
            limits:
              cpu: 100m
              memory: 32Mi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop:
                - ALL

What Should Match?

Your file should match every field shown here. Keep this healthy manifest unchanged because later recovery actions use it as the baseline.

Confirm the healthy baseline

A running Pod proves that the process stayed alive. Its logs provide stronger evidence because they expose the command's repeated health message.

Before you read the logs, which line should repeat if the saved baseline is working?

Read the Deployment logs by running this command:

kubectl logs deployment/fragile-app -n incident-lab

What Do the Logs Prove?

The output includes fragile-app healthy. This proves that the image started successfully.

The message also confirms that the shell loop is running. You now have evidence for the exact behavior a repair must restore.

Strong baseline established. You have a saved manifest plus a live log that defines healthy behavior.

Are the Healthy Logs Missing?

Wait for the replacement Pod to finish starting before reading the Deployment logs again. The command needs a running Pod behind the Deployment.

Check that the manifest still contains echo "fragile-app healthy" inside the loop.

Help me find the missing healthy log.

Break the image reference

The Deployment image field controls what Kubernetes pulls for each replacement Pod. Changing only that field isolates the failure from the known-good command.

Before you make the change, do you expect the replacement Pod to start successfully?

Change the Deployment image by running this command:

kubectl set image deployment/fragile-app fragile-app=registry.invalid/incident-lab/fragile-app:1.0 -n incident-lab

What Did This Command Change?

The command updates only the fragile-app container image. The Deployment starts a replacement because its Pod template changed.

Watch the replacement Pod by running this command:

kubectl get pods -n incident-lab -w

What Does the Failure Show?

The replacement Pod reaches ImagePullBackOff. Kubernetes could not pull the requested container image.

The .invalid domain makes this failure deterministic. The pull retries with an increasing delay.

Stop the watch after the replacement Pod reports ImagePullBackOff.

This shortfall is intentional. The healthy baseline remains in fragile-app.yaml while the live Deployment now exposes a clean image-pull incident.

Do You See a Different Pod State?

Allow the replacement Pod time to retry the pull. The status can pass through an earlier image-pull state before reaching ImagePullBackOff.

Confirm that the live image value contains registry.invalid/incident-lab/fragile-app:1.0.

Help me diagnose the image-pull incident.

Your controlled incident is ready. Next, you will declare a responder that gathers evidence without changing the broken workload.

Declare the Incident-Response Agent

Your deliberately broken workload is now stuck in ImagePullBackOff inside Kubernetes. Manual checks can reveal this failure once.

A declarative agent makes the evidence path repeatable. Its Model Context Protocol tool set defines what it can inspect.

kagent keeps the only selected write behind human approval. The resulting manifest combines evidence policy with a review gate.

In this step, get ready to:

Declare the responder's model plus its focused Kubernetes tool set.
Gate the patch tool with human approval.
Verify a read-only investigation in the dashboard.

Define the Agent and its tools

An Agent manifest stores the responder as a Kubernetes resource. Its selected tools create a focused path from evidence gathering to a proposed patch.

Return to the editor that holds fragile-app.yaml.
Create incident-agent.yaml beside fragile-app.yaml by using the editor's new-file control.
Define the Agent identity plus its selected tools by adding this code:

apiVersion: kagent.dev/v1alpha2
kind: Agent
metadata:
  name: incident-responder
  namespace: kagent
spec:
  description: Namespace-scoped Kubernetes incident responder with human approval for every exposed write.
  type: Declarative
  declarative:
    modelConfig: default-model-config
    tools:
      - type: McpServer
        mcpServer:
          name: kagent-tool-server
          kind: RemoteMCPServer
          apiGroup: kagent.dev
          toolNames:
            - k8s_get_resources
            - k8s_describe_resource
            - k8s_get_pod_logs
            - k8s_get_events
            - k8s_get_resource_yaml
            - k8s_patch_resource

What Does This Manifest Define?

The metadata creates incident-responder in the kagent namespace.
The default-model-config ModelConfig already points to the Ollama-backed qwen3:4b model.
The five read tools gather resources, descriptions, Pod logs, events, plus current resource YAML.
The k8s_patch_resource entry is the only selected Kubernetes write tool.

Save incident-agent.yaml.
Count five read tool identifiers under toolNames.
Confirm that k8s_patch_resource is the only selected write tool.

Does the Tool List Look Misaligned?

Check that tools sits beneath declarative. Keep every tool name beneath toolNames at the same indentation.

Compare each identifier with the code block because a misspelled tool name prevents kagent from finding it.

Help me check my Agent tool configuration.

Add the approval boundary

Kubernetes RBAC grants namespaced permissions across kagent plus incident-lab. The system message narrows the responder's behavior to the fragile workload.

The approval list adds a separate human-in-the-loop decision before the patch tool can execute. You remain the final reviewer of every proposed write.

Find modelConfig: default-model-config in incident-agent.yaml.
Add the responder policy directly below that line:

    systemMessage: |
      You are the incident responder for the Kubernetes namespace incident-lab.
      Inspect only resources in incident-lab. Never inspect or modify another namespace.
      Gather evidence before proposing a change. Use resource listings, descriptions,
      events, pod logs, and the current resource YAML. State the root cause and cite the
      evidence you found before calling a write tool.

      Your only permitted write target is the Deployment named fragile-app in the
      incident-lab namespace. Your only exposed Kubernetes write tool is
      k8s_patch_resource. Never create, delete, execute commands in, scale, label,
      or annotate resources.

      The known healthy baseline is:
      image: busybox:1.37.0
      command: ["/bin/sh", "-c"]
      args: ["while true; do echo \"fragile-app healthy\"; sleep 30; done"]

      Before calling k8s_patch_resource, explain the exact target and intended patch.
      If the evidence is incomplete or the requested target is outside the allowed scope,
      stop and ask the user for clarification.

How Does the Policy Shape the Responder?

The investigation stays inside incident-lab.
The responder must gather operational evidence before proposing a change.
The permitted write target is Deployment fragile-app in incident-lab.
The known healthy baseline gives the responder exact recovery values from fragile-app.yaml.

Save incident-agent.yaml.
Confirm that the policy names incident-lab as the investigation boundary.
Confirm that the policy names Deployment fragile-app as the only permitted write target.

Does the System Message Break the YAML?

Keep every policy line indented beneath the systemMessage: | line. Preserve the escaped quotation marks inside the healthy baseline.

Help me check the system message indentation.

A gated tool pauses when the Agent tries to invoke it. The deployment settings also give the responder bounded compute plus hardened Pod security.

Find the final k8s_patch_resource entry under toolNames.
Add the approval gate plus deployment settings directly below that entry:

          requireApproval:
            - k8s_patch_resource
    deployment:
      resources:
        requests:
          cpu: 50m
          memory: 128Mi
        limits:
          cpu: 500m
          memory: 256Mi
      podSecurityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop:
            - ALL

What Does the Approval Gate Control?

The requireApproval list pauses every k8s_patch_resource call for your decision.
The resource settings bound the responder's CPU plus memory usage.
The security settings require a non-root process, a read-only filesystem, plus dropped Linux capabilities.

Save incident-agent.yaml.
Confirm that requireApproval contains only k8s_patch_resource.
Confirm that deployment aligns beneath declarative.

Is the Approval Gate in the Wrong Place?

Place requireApproval beside toolNames within mcpServer. Place deployment beside systemMessage plus tools within declarative.

Help me place the approval and deployment blocks.

✔️ Awesome, I've got everything!

Solid work. Your saved Agent manifest now combines focused diagnostics, policy restrictions, hardened execution, plus approval-gated patching.

ⓧ I'd like to double check the full code

The complete incident-agent.yaml file should match this reference.

apiVersion: kagent.dev/v1alpha2
kind: Agent
metadata:
  name: incident-responder
  namespace: kagent
spec:
  description: Namespace-scoped Kubernetes incident responder with human approval for every exposed write.
  type: Declarative
  declarative:
    modelConfig: default-model-config
    systemMessage: |
      You are the incident responder for the Kubernetes namespace incident-lab.
      Inspect only resources in incident-lab. Never inspect or modify another namespace.
      Gather evidence before proposing a change. Use resource listings, descriptions,
      events, pod logs, and the current resource YAML. State the root cause and cite the
      evidence you found before calling a write tool.

      Your only permitted write target is the Deployment named fragile-app in the
      incident-lab namespace. Your only exposed Kubernetes write tool is
      k8s_patch_resource. Never create, delete, execute commands in, scale, label,
      or annotate resources.

      The known healthy baseline is:
      image: busybox:1.37.0
      command: ["/bin/sh", "-c"]
      args: ["while true; do echo \"fragile-app healthy\"; sleep 30; done"]

      Before calling k8s_patch_resource, explain the exact target and intended patch.
      If the evidence is incomplete or the requested target is outside the allowed scope,
      stop and ask the user for clarification.
    tools:
      - type: McpServer
        mcpServer:
          name: kagent-tool-server
          kind: RemoteMCPServer
          apiGroup: kagent.dev
          toolNames:
            - k8s_get_resources
            - k8s_describe_resource
            - k8s_get_pod_logs
            - k8s_get_events
            - k8s_get_resource_yaml
            - k8s_patch_resource
          requireApproval:
            - k8s_patch_resource
    deployment:
      resources:
        requests:
          cpu: 50m
          memory: 128Mi
        limits:
          cpu: 500m
          memory: 256Mi
      podSecurityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop:
            - ALL

Apply and test the no-change investigation

Applying the manifest lets the kagent controller create the responder. Its Ready condition confirms that the declared model plus tools are available.

Return to the terminal from earlier in the folder that contains incident-agent.yaml.
Create the Agent resource by running this command:

kubectl apply -f incident-agent.yaml

What Does This Command Create?

The command sends the incident-agent.yaml manifest to Kubernetes. The kagent controller receives the new incident-responder resource in the kagent namespace.

The terminal confirms that the Agent resource was created.

Does the Agent Fail to Apply?

Check the line reported by Kubernetes in incident-agent.yaml. Indentation near systemMessage or requireApproval can move a field outside its expected parent.

Help me diagnose the Agent apply failure.

Before you check readiness, do you expect the controller to accept the complete Agent manifest?

Wait for the responder's Ready condition by running this command:

kubectl wait --for=condition=Ready agent/incident-responder -n kagent --timeout=180s

What Does the Readiness Check Prove?

The command watches the incident-responder Agent for its Ready condition. The 180s timeout stops the wait if the controller cannot prepare it.

The command returns when the Agent reaches Ready. Your responder can now accept an investigation request.

Does the Readiness Check Time Out?

Confirm that the kagent Pods from earlier remain Ready. Confirm that the local Ollama service still exposes qwen3:4b to the cluster.

Check the Agent manifest for a misspelled tool identifier if the supporting services are healthy.

Help me diagnose why the Agent is not Ready.

A no-change investigation proves that the responder can gather evidence while leaving the incident untouched. The dashboard tool cards expose the arguments plus results behind its diagnosis.

Switch back to the kagent dashboard in the browser from earlier.
Select kagent/incident-responder as the active Agent.

Before you send the request, which read tools do you expect the responder to use as evidence?

Start a no-change investigation by sending this prompt:

Investigate fragile-app in incident-lab. Gather evidence and explain the root cause. Do not change anything.

Why Does This Prompt Stay Read-Only?

The prompt asks for evidence plus a root-cause explanation. Its final sentence explicitly withholds permission to change the workload.

The responder can inspect resources, events, logs, plus the Deployment manifest through its selected read tools.

Expand the completed tool cards in the dashboard.
Inspect each card's arguments.
Inspect each card's results.
Confirm that the response identifies registry.invalid/incident-lab/fragile-app:1.0 as the image behind the failure.
Confirm that no k8s_patch_resource call executed.

You'll see completed read tool calls supporting an invalid-image diagnosis. The Deployment remains in ImagePullBackOff because the investigation performed no write.

Your responder now diagnoses the incident through a visible evidence trail. Next, you will review its exact image repair before allowing the patch to execute.

Diagnose and Approve the Image Repair

Your kagent responder has already investigated the failing Deployment without changing it. The live image still points to registry.invalid/incident-lab/fragile-app:1.0, so the workload remains in ImagePullBackOff.

The repair must pass through k8s_patch_resource, which is the responder's only exposed Kubernetes write tool. You remain the decision point before that patch can execute.

In this step, get ready to:

Ask the responder to justify a repair with cluster evidence.
Review the proposed patch against the approved target.
Verify that the approved repair restores the healthy workload.

Request an evidence-backed repair

A trustworthy remediation starts with evidence from the live cluster. The responder should connect the failing Pod state to image-pull events plus the invalid image in the current manifest.

Switch back to the browser tab showing the kagent dashboard from earlier.
Select kagent/incident-responder.
Request an evidence-backed repair by sending this prompt:

Repair fragile-app using the known healthy baseline. Explain the evidence and show the exact target before changing anything.

What Does This Request Require?

The responder must support its diagnosis with the Pod state, events, plus the current Deployment manifest.
The responder must identify the exact patch target before requesting permission to change it.

Read the responder's diagnosis.
Expand each completed read tool card.
Confirm that the Pod state reports ImagePullBackOff.
Confirm that the events describe an image-pull failure.
Confirm that the Deployment manifest contains registry.invalid/incident-lab/fragile-app:1.0.

You have the evidence chain in place. The responder has grounded its proposed repair in three cluster signals.

Review and approve the patch

A human approval gate pauses the MCP tool call before it reaches the Kubernetes API. This pause gives you time to compare the proposed write with the responder's allowed scope.

Wait for the k8s_patch_resource card to request approval.
Expand the card's arguments.
Confirm that the resource kind is Deployment.
Confirm that the resource name is fragile-app.
Confirm that the namespace is incident-lab.
Confirm that the proposed image is busybox:1.37.0.
Confirm that the patch preserves the healthy command plus loop arguments from the current manifest.

Does a Patch Argument Look Wrong?

Click Reject if any target value falls outside the approved boundary.
Enter a reason that identifies the mismatched argument.
Ask the responder to propose a corrected patch.

Help me review an unsafe patch proposal.

Approving this call changes the live Kubernetes workload. Your argument review keeps that write inside the intended target.

Click Approve when every reviewed value matches.
Wait for the tool card to show a completed result.

What Does Approval Authorize?

Approval releases this specific paused patch call for execution. Future calls to k8s_patch_resource still require a separate review.

The patch changes the live Deployment. Your baseline files remain unchanged.

Verify the healthy recovery

The completed tool card records the responder's result. Rollout status plus workload logs provide independent proof that the repair worked.

Before you run these checks, what should a successful approval change in the rollout and log stream? Hold that prediction.

Return to the terminal from earlier.
Check the repaired Deployment rollout by running this command:

kubectl rollout status deployment/fragile-app -n incident-lab

What Does This Command Check?

The command watches the latest rollout of fragile-app in incident-lab. It returns after Kubernetes reports that the Deployment has completed its rollout.

You should see the rollout status report a successful completion. The replacement Pod now stays in the running state.

Read the repaired workload's log stream by running this command:

kubectl logs deployment/fragile-app -n incident-lab

What Does This Command Confirm?

The command reads logs from the active Pod managed by the fragile-app Deployment. A healthy message proves that the restored image can start the existing loop.

You should see fragile-app healthy in the output. The approved repair has restored the known-good workload.

Is the Workload Still Unhealthy?

Check the approved patch arguments for busybox:1.37.0 if the rollout keeps waiting.
Wait for the replacement Pod to report Running before retrying the log check.

Help me diagnose the failed image repair.

You closed the image incident through an evidence-backed patch plus explicit approval. Next, you will test a process-level failure that demands runtime evidence.

Diagnose and Approve the Crash-Loop Repair

The image repair from the last step returned your Kubernetes Deployment to a healthy baseline. A successful image pull does not prove that the process can stay running.

This drill replaces only the startup command. Your kagent responder must use runtime evidence to separate a process failure from an image failure.

In this step, get ready to:

Trigger a controlled CrashLoopBackOff.
Diagnose the startup failure with events plus previous logs.
Approve a patch that restores the healthy startup command.

Trigger a controlled crash loop

A CrashLoopBackOff occurs when a container repeatedly starts and fails. Kubernetes increases the delay between restart attempts.

Create crash-loop-patch.yaml beside fragile-app.yaml with your editor.
Paste this content into crash-loop-patch.yaml:

spec:
  template:
    spec:
      containers:
        - name: fragile-app
          command:
            - /bin/sh
            - -c
          args:
            - |
              echo "simulated startup failure" >&2
              exit 42

What Does This Patch Do?

The spec.template.spec.containers path targets the Deployment's Pod template.
The container name fragile-app selects the workload container.
The command field starts a shell for the replacement Pod.
The args field writes an explicit failure message to standard error.
The exit 42 line ends the process with a failure status.

Save crash-loop-patch.yaml.

✔️ Awesome, I've got everything!

Great. Your saved patch now creates a deterministic startup failure in fragile-app.

ⓧ I'd like to double check the full code

spec:
  template:
    spec:
      containers:
        - name: fragile-app
          command:
            - /bin/sh
            - -c
          args:
            - |
              echo "simulated startup failure" >&2
              exit 42

This patch keeps the healthy container image. It replaces the long-running loop with a command that exits immediately.

Before you apply the patch, make a quick prediction about whether the replacement container process stays running.

Apply the startup-failure patch by running this command:

kubectl patch deployment fragile-app -n incident-lab --type=strategic --patch-file crash-loop-patch.yaml

You should see a confirmation that the Deployment was patched.

Does the Patch Command Fail?

Confirm that crash-loop-patch.yaml is saved beside fragile-app.yaml.

Compare the indentation in your file with the full-code tab. YAML nesting determines which Deployment fields the patch changes.

Help me diagnose the failed crash-loop patch command.

Before you start the watch, ask yourself which Pod status would prove that Kubernetes is repeatedly restarting the process.

Watch the replacement Pod by running this command:

kubectl get pods -n incident-lab -w

The replacement Pod eventually reports CrashLoopBackOff.

Stop the watch with Ctrl+C after you see CrashLoopBackOff.

Previous-container logs preserve output from the process that just failed. They reveal the message written before the restart.

Read the previous container instance's logs by running this command:

kubectl logs deployment/fragile-app -n incident-lab --previous

You should see simulated startup failure in the output.

What Does the Failure Prove?

The Pod can pull the existing image. Its configured shell then prints the failure message before exiting.

The CrashLoopBackOff status is the restart symptom. The previous log exposes the startup command as the root cause.

Missing the Previous Log?

Wait until the Pod has restarted at least once before reading the previous container instance.

Confirm that the command targets deployment/fragile-app in incident-lab.

Help me find the previous fragile-app logs.

Gather runtime evidence

The responder now needs events plus logs plus the current manifest. Together, these sources separate a successfully pulled image from a failing startup command.

Switch back to the kagent dashboard from earlier.
Enter Investigate and repair the CrashLoopBackOff for fragile-app in incident-lab. Gather events, logs, and the current manifest before proposing a patch. in the responder chat.
Submit the prompt.
Expand each completed tool card.
Inspect the k8s_get_events result.
Inspect the k8s_get_pod_logs result.
Inspect the k8s_get_resource_yaml result.

The responder should identify a startup-command failure. Its diagnosis should explain that the image pull succeeded.

How Does the Evidence Fit Together?

The events establish the Pod's recent lifecycle.
The previous logs expose simulated startup failure from the terminated container.
The current manifest shows the command that prints the message before exiting.
The combined evidence supports restoring the known healthy command plus arguments.

Is the Evidence Incomplete?

Confirm that the UI port-forward from earlier remains active. A stopped port-forward disconnects the dashboard from the local service.

Confirm that kagent/incident-responder remains selected before submitting the investigation prompt again.

Help me diagnose incomplete responder evidence.

Review and approve the repair

The human approval gate pauses the responder's only exposed Kubernetes write tool. Your review determines whether the proposed patch can execute.

Wait for the k8s_patch_resource call to pause for approval.
Expand the pending patch arguments.
Confirm that the target kind is Deployment.
Confirm that the target name is fragile-app.
Confirm that the target namespace is incident-lab.
Confirm that the command field restores ["/bin/sh", "-c"].
Confirm that the args field restores the healthy while-loop from fragile-app.yaml.
Click Approve.

The patch tool changes from a pending approval to a completed result. The responder has now executed the reviewed repair.

Before you run the final checks, predict whether the rollout plus log stream now match the healthy baseline.

Watch the repaired Deployment finish its rollout by running this command:

kubectl rollout status deployment/fragile-app -n incident-lab

You should see confirmation that the Deployment rollout completed successfully.

Is the Rollout Still Waiting?

Return to the completed patch card. Confirm that the reviewed tool call executed successfully.

Recheck the patch arguments for the healthy shell command plus the long-running loop.

Help me diagnose the stalled fragile-app rollout.

Confirm the restored process output by running this command:

kubectl logs deployment/fragile-app -n incident-lab

You should see fragile-app healthy in the log stream. That is the second failure mode repaired.

Missing the Healthy Log?

Wait for the replacement Pod to reach its running state before reading the Deployment logs.

Confirm that the approved patch restored the healthy loop from fragile-app.yaml.

Help me diagnose the missing healthy log.

Secret mission

Reject a Plausible Remediation

Recreate the startup failure and reject a technically correct repair because the change window is closed. Prove that the rejected tool call leaves the workload unchanged until you recover it manually.

Clean Up Your Resources

Choose whether to keep the incident lab active, pause its local processes, or remove its resources. Paid service cost stays at $0 while local compute, storage, electricity, and network bandwidth still apply.

Resources you used:

The active kagent UI port-forward on local port 8080.
The local kagent installation managed by the kagent Helm release at version 0.9.12.
The kagent-crds Helm release at version 0.9.12.
The kagent Kubernetes namespace containing the controller, tool server, UI, incident-responder Agent, evaluation database volume, and namespace-scoped RBAC rules.
The incident-lab namespace containing the healthy fragile-app Deployment.
The temporary Ollama systemd override that exposes its API on 0.0.0.0:11434.
The local project files kagent-values.yaml, incident-agent.yaml, fragile-app.yaml, and crash-loop-patch.yaml.

Keep everything running

No resource deletion is needed. Choose this if you want to repeat the incident drills with the existing Agent and dashboard history.

Return to the second terminal that runs the kagent UI port-forward.
Press Ctrl+C to stop the port-forward.
Leave Rancher Desktop running.
Leave Ollama running.
Keep the four YAML files for future drills.

The namespaces, dashboard history, Agent, and healthy workload remain available for another drill.

The Host Bind Stays Active

The temporary 0.0.0.0:11434 override remains active while Ollama runs. This keeps the API reachable from Rancher Desktop.

Pause - I'll come back to this later

Shut down the running processes to free local compute resources. Your cluster state and model data remain on disk for your next session.

Return to the second terminal that runs the kagent UI port-forward.
Press Ctrl+C to stop the port-forward.
Quit Rancher Desktop from its desktop menu.
Stop Ollama through the Linux service manager you used earlier.

The namespaces, dashboard evidence, model data, and YAML files remain stored locally.

The Override Stays Configured

Pausing stops Ollama, but the 0.0.0.0:11434 override remains configured. The Delete option restores Ollama's default local bind.

Delete - I don't want to use this again

Removing the lab also erases its dashboard history. Your existing Rancher Desktop, Ollama, and Helm installations stay available.

Capture any dashboard evidence you want to keep.
Return to the second terminal that runs the kagent UI port-forward.
Press Ctrl+C to stop the port-forward.
Delete the incident-responder Agent from the cluster with kubectl using incident-agent.yaml.
Delete the fragile-app Deployment from the cluster with kubectl using fragile-app.yaml.
Uninstall the kagent release from the kagent namespace with Helm.
Wait until Helm confirms that the kagent release is removed.
Uninstall the kagent-crds release from the kagent namespace with Helm.

Removing kagent first gives the controller-managed resources time to disappear before you remove the CRDs.

Delete the incident-lab namespace with kubectl.
Confirm that incident-lab no longer appears in the namespace list.
Delete the kagent namespace with kubectl.
Confirm that kagent no longer appears in the namespace list.
Return to the existing Ollama systemd override.
Remove the Environment="OLLAMA_HOST=0.0.0.0:11434" line from the override.
Save the override.
Close the editor.

The next service reload returns Ollama to its default local binding.

Reload systemd through your Linux service manager.
Restart Ollama through your Linux service manager.
Confirm that Ollama still lists qwen3:4b with its model-list command.
Open the folder containing the four YAML files in your file manager.
Delete kagent-values.yaml, incident-agent.yaml, fragile-app.yaml, and crash-loop-patch.yaml.
Confirm that those four files no longer appear in your file manager.
Confirm that the dashboard no longer loads at http://localhost:8080.

You should no longer see either lab namespace in Kubernetes. Ollama should still list qwen3:4b after the restart.

Nice Work!

You did it! You completed a local Kubernetes incident-response workflow with kagent. Your human approval gate kept the responder's only exposed Kubernetes write under your control.

You've learned how to:

Trigger controlled ImagePullBackOff and CrashLoopBackOff incidents through distinct failure mechanisms. Distinguish their root causes with events and logs.
Run a declarative kagent responder at version 0.9.12 on local Ollama with qwen3:4b. Limit it to six focused MCP tools. Enforce namespace-scoped RBAC across kagent and incident-lab.
Inspect operational evidence before remediation. Review the exact k8s_patch_resource arguments before approving each repair.
Secret Mission: Reject a technically correct patch with a recorded reason. Prove that fragile-app remains in CrashLoopBackOff until manual recovery.

Ready to quiz yourself?