Build a Kubernetes Controller

Build a custom Kubernetes controller and watch reconciliation loops self-heal your cluster.

Introduction

30 Second Summary

Think about your home thermostat. You set it to 22 degrees and walk away. The heater kicks on when the temperature drops, and shuts off when it reaches the target. You never manually flick the heater switch. The system continuously drives reality toward your declared goal.

In this project, you will build a custom Kubernetes controller that works exactly like that thermostat. You declare a desired state ("I want 3 echo server pods") and a reconciliation loop continuously creates, fixes, and scales pods to match.

What You'll Build

You will run a controller on your local machine, delete pods on purpose, change replica counts, and watch the system snap back to health every time.

By the end of this project, you'll have:

  • A working custom controller that you can break on purpose and watch self-heal within seconds.
  • A Custom Resource Definition that lets you declare how many replicas you want and what message the server displays, just by editing a YAML file.
  • Real-time reconciliation loop logs showing the observe-compare-act cycle firing every time you perturb the system.
  • Secret Mission: Add status conditions to complete the full feedback loop, reporting observed state back so operators can verify convergence.

Are there any prerequisites?

This project assumes you are comfortable reading and writing Go code. You should also have basic familiarity with Kubernetes concepts like pods, deployments, and kubectl apply.

Before We Start

Before We Start...

Before we dive in, take a moment to lock in what you're about to build and why it matters. Understanding the reconciliation loop pattern upfront makes every step click into place.

Set Up Your Local Cluster

To build a custom Kubernetes controller, you need a local cluster to run it against and the Kubebuilder SDK to scaffold the project.

Without a running cluster, there is nothing to reconcile. In this step, you will install three tools and spin up a local Kubernetes cluster using kind (Kubernetes in Docker).

In this step, get ready to:
  • Install kind, kubectl, and Kubebuilder on your machine.
  • Create a local Kubernetes cluster and confirm it is healthy.
Install kind, kubectl, and Kubebuilder

You need three CLI tools to develop a controller locally. kind creates lightweight Kubernetes clusters inside Docker containers. kubectl lets you interact with the cluster. Kubebuilder scaffolds the controller project with all the boilerplate wiring.

  • Install kind by running this command:
go install sigs.k8s.io/kind@v0.32.0
  • Verify that kind is available by checking its version:
kind version

✔️ I see version 0.32.0 or higher

Kind is ready to go. Move on to the next install.

ⓧ I see an older version

The go install command installs the specific version you request. Run the install command above again to replace the older binary.

  • Run the install command again:
go install sigs.k8s.io/kind@v0.32.0
  • Re-check with kind version to confirm you see v0.32.0.

ⓧ Command not found

Your Go bin directory is not on your PATH. The go install command places binaries in $(go env GOPATH)/bin.

  • Add Go's bin directory to your PATH by running:
export PATH="$PATH:$(go env GOPATH)/bin"
  • Run kind version again to confirm you see v0.32.0.
  • To make this permanent, add the export line to your ~/.zshrc or ~/.bash_profile.
  • Install kubectl by running:
brew install kubectl
  • Verify kubectl is available:
kubectl version --client

✔️ I see a version number

kubectl is ready. Move on to installing Kubebuilder.

ⓧ Command not found

Homebrew may not be installed, or the formula failed.

  • If you do not have Homebrew, install it from https://brew.sh first.
  • Then run brew install kubectl again.
  • Re-check with kubectl version --client to confirm it prints a version.
  • Download and install Kubebuilder by running these commands:
curl -L -o kubebuilder "https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)"
chmod +x kubebuilder && sudo mv kubebuilder /usr/local/bin/

What do these commands do?

The curl command downloads the latest Kubebuilder binary for your OS and architecture. The second line makes it executable and moves it into a system-wide bin directory so you can run kubebuilder from anywhere.

Your terminal will prompt you for your password when sudo runs. Enter your macOS login password and press Enter.

  • Verify Kubebuilder is available:
kubebuilder version

✔️ I see version 4.15.0 or higher

Kubebuilder is installed and ready. All three tools are in place.

ⓧ I see an older version

The install command downloads the latest release. If you see an older version, a cached binary may be shadowing the new one.

  • Check which binary is being used by running:
which kubebuilder
  • If the path is not /usr/local/bin/kubebuilder, remove the old binary at the reported path and re-run the install commands above.

ⓧ Command not found

The binary may not have moved to the correct location.

  • Check that /usr/local/bin is on your PATH by running:
echo $PATH
  • If you do not see /usr/local/bin in the output, add it:
export PATH="$PATH:/usr/local/bin"
  • Run kubebuilder version again to confirm you see v4.15.0.

Install not working?

Make sure Docker is running before continuing. kind needs Docker to create cluster containers. Open Docker Desktop and wait for the whale icon in your menu bar to stop animating.

help me troubleshoot my kind, kubectl, or Kubebuilder installation

Create a local Kubernetes cluster with kind

With all tools installed, you can spin up a full Kubernetes cluster in seconds. kind runs cluster nodes as Docker containers, so there is no cloud cost and no VM overhead.

  • Create a local cluster by running:
kind create cluster

What does this command do?

This creates a single-node Kubernetes cluster running inside a Docker container. The cluster is named kind by default. kind also configures your kubectl context automatically, so kubectl commands target the new cluster without any extra setup.

  • Wait for the command to finish. You will see output ending with a message that the cluster is ready.
  • Confirm the cluster exists by running:
kind get clusters

You should see kind printed as the cluster name.

Before you run the next command, what do you think kubectl cluster-info will show for a local cluster?

  • Confirm the cluster is healthy by running:
kubectl cluster-info

You should see output showing the Kubernetes control plane running at https://127.0.0.1 followed by a port number. This confirms your local cluster is reachable.

Seeing connection refused?

Make sure Docker Desktop is running. kind clusters are Docker containers, so they stop when Docker stops.

  • Check Docker is running by looking for the whale icon in your menu bar.
  • If the cluster was created but Docker restarted since, delete and recreate it:
kind delete cluster
kind create cluster

help me fix kubectl connection issues with my kind cluster

Your local cluster is running and all three tools are ready. Next up, you will scaffold the operator project and define the custom resource that declares your desired state.

Scaffold the Operator and Define the API

Your cluster is running, but it only knows about built-in resources like Pods and Deployments. To teach it about a new resource type called EchoServer, you need a Custom Resource Definition (CRD) and a controller to act on it.

Kubebuilder generates all the boilerplate wiring (the manager, scheme registration, RBAC markers) so you can focus on the reconciliation logic itself. After this step, you will have a CRD installed and a sample resource applied, but the controller will do nothing. That gap is intentional.

In this step, get ready to:
  • Scaffold the Kubebuilder project and define the EchoServer API types.
  • Install the CRD, apply a sample resource, and observe the empty controller.
Scaffold the project and define EchoServer types

Kubebuilder needs a dedicated directory for the operator project. You will initialise the project, create the EchoServer API, edit the type definitions, and generate the CRD manifest.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create a new directory for the operator and move into it by running these commands:
mkdir echoserver-operator
cd echoserver-operator
  • Initialise the Kubebuilder project by running this command:
kubebuilder init --domain learn.dev --repo learn.dev/echoserver-operator

What does this command do?

The --domain flag sets the API group suffix. All your custom resources will live under *.learn.dev.

The --repo flag sets the Go module path. Kubebuilder scaffolds a full Go project with a Makefile, Dockerfile, and config directory.

  • Confirm the scaffolded files exist by running:
ls

You should see files including Makefile, go.mod, and a config/ directory.

  • Create the EchoServer API by running this command:
kubebuilder create api --group apps --version v1alpha1 --kind EchoServer

What will this command prompt?

Kubebuilder will ask two questions interactively:

  • Create Resource [y/n] — press y then Enter.
  • Create Controller [y/n] — press y then Enter.

This generates the types file at api/v1alpha1/echoserver_types.go and the controller at internal/controller/echoserver_controller.go.

Now you will define what an EchoServer resource looks like by editing the types file.

  • Open api/v1alpha1/echoserver_types.go in your code editor.
  • Replace the entire file contents with the following code:
package v1alpha1

import (
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// EchoServerSpec defines the desired state of EchoServer
type EchoServerSpec struct {
	// Replicas is the desired number of echo server pods
	// +kubebuilder:validation:Minimum=1
	// +kubebuilder:validation:Maximum=10
	Replicas int32 `json:"replicas"`

	// Message is the text the echo server should be configured with
	Message string `json:"message"`
}

What does this code do?

  • EchoServerSpec defines the desired state: how many replicas you want and what message the server displays.
  • The +kubebuilder:validation markers tell the CRD generator to enforce min/max on the replica count.
  • In control theory terms, this struct is the setpoint. You declare what you want, and the controller drives reality toward it.
  • Directly below the closing brace of EchoServerSpec, add the status struct and remaining type definitions:
// EchoServerStatus defines the observed state of EchoServer
type EchoServerStatus struct {
	// AvailableReplicas is the number of pods currently available
	AvailableReplicas int32 `json:"availableReplicas,omitempty"`
}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status

// EchoServer is the Schema for the echoservers API
type EchoServer struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   EchoServerSpec   `json:"spec,omitempty"`
	Status EchoServerStatus `json:"status,omitempty"`
}

// +kubebuilder:object:root=true

// EchoServerList contains a list of EchoServer
type EchoServerList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata,omitempty"`
	Items           []EchoServer `json:"items"`
}

func init() {
	SchemeBuilder.Register(&EchoServer{}, &EchoServerList{})
}

What does this code do?

  • EchoServerStatus holds observed state. For now it only tracks AvailableReplicas.
  • The +kubebuilder:subresource:status marker enables the status subresource so spec and status can be updated independently.
  • The init() function registers your types with the scheme so the controller-runtime knows about them.
  • Save api/v1alpha1/echoserver_types.go.
  • Generate the CRD manifest by running this command:
make manifests

You should see output referencing controller-gen generating files under config/crd/bases/.

Seeing a syntax error?

If make manifests reports a Go compilation error, double-check that your echoserver_types.go has matching braces and backtick struct tags on every field.

help me fix my echoserver_types.go so make manifests succeeds

✔️ Awesome, I've got everything!

Great. Make sure you have saved echoserver_types.go before continuing.

ⓧ I'd like to double check the full code

Your api/v1alpha1/echoserver_types.go is built from the two chunks above. Scroll up and confirm both are present in order:

  • Part 1: package v1alpha1 through the closing } of EchoServerSpec.
  • Part 2: EchoServerStatus through func init() at the bottom.

Make sure there is one blank line separating the two parts (after the closing brace of EchoServerSpec).

Install the CRD, apply a sample, and run the controller

The CRD manifest exists on disk, but your cluster does not know about it yet. You will install the CRD, create a sample EchoServer resource declaring 3 replicas, and start the scaffolded controller.

  • Install the CRD into your kind cluster by running:
make install

You should see output ending with the CRD being applied to the cluster.

Seeing a connection error?

Make sure Docker is running and your kind cluster is healthy. Run kubectl cluster-info to confirm connectivity.

help me fix my kind cluster connection for make install

Now you need a sample Custom Resource that declares desired state.

  • Open config/samples/apps_v1alpha1_echoserver.yaml in your editor.
  • Replace its contents with the following YAML:
apiVersion: apps.learn.dev/v1alpha1
kind: EchoServer
metadata:
  name: echo-sample
  namespace: default
spec:
  replicas: 3
  message: "Hello from the reconciliation loop!"

What does this resource declare?

This is the desired state: 3 replicas of an echo server displaying a specific message. In control theory terms, this YAML is the setpoint. The controller's job (once implemented) will be to drive reality toward it.

  • Save the file and apply the sample resource by running:
kubectl apply -f config/samples/apps_v1alpha1_echoserver.yaml
  • Verify the custom resource exists by running:
kubectl get echoservers

You should see echo-sample listed. The cluster now knows about your desired state.

Time to start the controller. Before you run it, what do you think will happen? Will pods appear automatically?

  • Run the scaffolded controller locally by running:
make run

What does make run do?

This compiles and starts the controller-manager binary in the foreground. It connects to your kind cluster using the kubeconfig credentials. You will see log lines indicating the controller is starting and reconciling.

The controller runs in this terminal window. You will need a second terminal for the next commands.

  • Open a new terminal window by pressing Cmd+T (macOS) or Ctrl+Shift+T (Windows/Linux).
  • Navigate back to your project directory by running:
cd ~/Desktop/echoserver-operator
  • Check if any Deployments were created by running:
kubectl get deployments

You will see No resources found in default namespace. despite the EchoServer resource existing with 3 desired replicas.

This is the designed gap. The desired state is declared but no actuator exists to create the Deployment. The Reconcile function body is empty.

Why is this gap important?

Right now your system is open-loop: it has a setpoint (the CR) but no feedback mechanism to drive actual state toward it. In control theory, a controller without an actuator is just a sensor reading a target.

The next step adds the actuator: a Reconcile function that observes, compares, and acts.

  • Stop the running controller by pressing Ctrl+C in the terminal where make run is running.

You have declared desired state and proven that it is not enough on its own. Next up, you will implement the reconciliation loop that observes, compares, and acts to close the gap between desired and actual state.

Implement the Reconciliation Loop

In the previous step, you declared desired state (3 EchoServer replicas) and ran the controller. Nothing happened. The Reconcile function was empty. Desired state existed, but there was no actuator to drive reality toward it.

This is the core lesson of control theory: declaring a setpoint is not enough. You need a control loop that observes actual state, compares it to the setpoint, and takes corrective action. That is exactly what you will build now.

In this step, get ready to:
  • Implement the full Reconcile function with observe, compare, and act phases.
  • Add RBAC permissions and run the controller to see pods appear.
  • Close the feedback loop so Deployment changes trigger re-reconciliation.
Implement the Reconcile function

The Reconcile function is the actuator in your control loop. Every time Kubernetes detects a change to an EchoServer resource, it calls this function. Your job is to observe the current state of the cluster, compare it to what the CR declares, and create or update a Deployment to close the gap.

  • Open internal/controller/echoserver_controller.go in your code editor.
  • Replace the entire file contents with the following code, starting with the package declaration, imports, and struct definition:
package controller

import (
	"context"
	"fmt"

	appsv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	apierrors "k8s.io/apimachinery/pkg/api/errors"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
	"sigs.k8s.io/controller-runtime/pkg/log"

	appsv1alpha1 "learn.dev/echoserver-operator/api/v1alpha1"
)

What are these imports?

  • appsv1 gives you Deployment types so you can create and inspect them.
  • apierrors provides helpers like IsNotFound to check whether a resource exists.
  • controllerutil gives you SetControllerReference to link the Deployment back to its owning EchoServer CR.
  • Directly below the imports, add the struct definition and the three existing RBAC markers that Kubebuilder scaffolded:
// EchoServerReconciler reconciles a EchoServer object
type EchoServerReconciler struct {
	client.Client
	Scheme *runtime.Scheme
}

// +kubebuilder:rbac:groups=apps.learn.dev,resources=echoservers,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=apps.learn.dev,resources=echoservers/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=apps.learn.dev,resources=echoservers/finalizers,verbs=update

What are RBAC markers?

These comment lines tell Kubebuilder what permissions the controller needs. When you run make manifests, Kubebuilder reads these markers and generates a ClusterRole YAML under config/rbac/. The three markers above were already scaffolded. You will add a fourth in the next substep.

  • Now add the OBSERVE phase of the Reconcile function. This fetches the EchoServer CR and checks whether a matching Deployment already exists:
func (r *EchoServerReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	logger := log.FromContext(ctx)

	// OBSERVE: Fetch the EchoServer custom resource
	var echoServer appsv1alpha1.EchoServer
	if err := r.Get(ctx, req.NamespacedName, &echoServer); err != nil {
		if apierrors.IsNotFound(err) {
			logger.Info("EchoServer resource not found, must have been deleted")
			return ctrl.Result{}, nil
		}
		return ctrl.Result{}, err
	}

	logger.Info("Reconciling EchoServer", "name", echoServer.Name, "desiredReplicas", echoServer.Spec.Replicas)

	// OBSERVE: Check if a Deployment already exists
	var deployment appsv1.Deployment
	deploymentName := fmt.Sprintf("%s-deployment", echoServer.Name)
	err := r.Get(ctx, client.ObjectKey{Namespace: echoServer.Namespace, Name: deploymentName}, &deployment)

	if apierrors.IsNotFound(err) {
		// ACT: Create the Deployment (actual state is zero, desired state is non-zero)
		logger.Info("Creating Deployment", "name", deploymentName)
		deployment = *r.buildDeployment(&echoServer, deploymentName)
		if err := controllerutil.SetControllerReference(&echoServer, &deployment, r.Scheme); err != nil {
			return ctrl.Result{}, err
		}
		if err := r.Create(ctx, &deployment); err != nil {
			return ctrl.Result{}, err
		}
		logger.Info("Deployment created successfully")
		return ctrl.Result{}, nil
	} else if err != nil {
		return ctrl.Result{}, err
	}

What does this code do?

  • r.Get(ctx, req.NamespacedName, &echoServer) fetches the EchoServer CR that triggered this reconciliation.
  • r.Get(ctx, client.ObjectKey{...}, &deployment) checks if a Deployment named <echoserver-name>-deployment already exists.
  • If no Deployment exists, it builds one and creates it. SetControllerReference stamps an owner reference on the Deployment so Kubernetes knows it belongs to this EchoServer.
  • Continue in the same function. Add the COMPARE and ACT phases that detect drift and correct it:
	// COMPARE: Check if the Deployment matches the desired state
	needsUpdate := false

	if *deployment.Spec.Replicas != echoServer.Spec.Replicas {
		logger.Info("Replica count drift detected", "actual", *deployment.Spec.Replicas, "desired", echoServer.Spec.Replicas)
		deployment.Spec.Replicas = int32Ptr(echoServer.Spec.Replicas)
		needsUpdate = true
	}

	desiredMessage := echoServer.Spec.Message
	currentMessage := ""
	if len(deployment.Spec.Template.Spec.Containers) > 0 {
		for _, env := range deployment.Spec.Template.Spec.Containers[0].Env {
			if env.Name == "MESSAGE" {
				currentMessage = env.Value
				break
			}
		}
	}
	if currentMessage != desiredMessage {
		logger.Info("Message drift detected", "actual", currentMessage, "desired", desiredMessage)
		deployment.Spec.Template.Spec.Containers[0].Env = []corev1.EnvVar{{Name: "MESSAGE", Value: desiredMessage}}
		needsUpdate = true
	}

	// ACT: Update the Deployment if drift was detected
	if needsUpdate {
		logger.Info("Updating Deployment to match desired state")
		if err := r.Update(ctx, &deployment); err != nil {
			return ctrl.Result{}, err
		}
		logger.Info("Deployment updated successfully")
	} else {
		logger.Info("Deployment already matches desired state, no action needed")
	}

	return ctrl.Result{}, nil
}

What does the compare phase do?

It reads the existing Deployment's replica count and MESSAGE env var, then checks whether they match the CR's spec. If either has drifted, it corrects the Deployment. This is the same logic a thermostat uses: read the temperature, compare to the setpoint, and adjust.

  • Below the Reconcile function, add the buildDeployment helper that constructs the desired Deployment object:
func (r *EchoServerReconciler) buildDeployment(echoServer *appsv1alpha1.EchoServer, name string) *appsv1.Deployment {
	labels := map[string]string{"app": "echoserver", "controller": echoServer.Name}
	return &appsv1.Deployment{
		ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: echoServer.Namespace},
		Spec: appsv1.DeploymentSpec{
			Replicas: int32Ptr(echoServer.Spec.Replicas),
			Selector: &metav1.LabelSelector{MatchLabels: labels},
			Template: corev1.PodTemplateSpec{
				ObjectMeta: metav1.ObjectMeta{Labels: labels},
				Spec: corev1.PodSpec{
					Containers: []corev1.Container{{
						Name: "nginx", Image: "nginx:1.27",
						Ports: []corev1.ContainerPort{{ContainerPort: 80}},
						Env: []corev1.EnvVar{{Name: "MESSAGE", Value: echoServer.Spec.Message}},
					}},
				},
			},
		},
	}
}

What does buildDeployment produce?

A Deployment running nginx:1.27 containers with a MESSAGE environment variable. The labels app=echoserver and controller=<name> tie pods back to the EchoServer CR for easy identification.

  • Finally, add the int32Ptr helper and a simple SetupWithManager function at the bottom of the file:
func int32Ptr(i int32) *int32 { return &i }

// SetupWithManager sets up the controller with the Manager.
func (r *EchoServerReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&appsv1alpha1.EchoServer{}).
		Named("echoserver").
		Complete(r)
}

Why does SetupWithManager only watch EchoServer?

Right now the controller watches EchoServer CRs only. If someone manually edits the Deployment, the controller will not know. You will fix this gap in the final substep by adding Owns so changes to owned Deployments also trigger reconciliation.

  • Save internal/controller/echoserver_controller.go.
  • Verify the code compiles by running:
go build ./...

If you see no output, the build succeeded. Any typos will surface as compiler errors pointing to the exact line.

Seeing a compile error?

  • Check that every import in the import block matches exactly. A missing or extra import triggers imported and not used or undefined errors.
  • Ensure the closing brace of the Reconcile function appears before buildDeployment. A missing brace gives a cascade of type errors.

Help me fix a compile error in my controller

Add Deployment RBAC and run the controller

Your Reconcile code creates and updates Deployments. When running in-cluster, the controller needs explicit permission to do this. RBAC markers declare those permissions so Kubebuilder can generate the ClusterRole manifest.

  • In internal/controller/echoserver_controller.go, find the three existing RBAC marker lines (just above the Reconcile function).
  • Find this line:
// +kubebuilder:rbac:groups=apps.learn.dev,resources=echoservers/finalizers,verbs=update
  • Add the following RBAC marker directly below it, granting the controller full access to Deployments:
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete

What does this marker produce?

When you run make manifests, Kubebuilder regenerates config/rbac/role.yaml to include a rule allowing get, list, watch, create, update, patch, and delete on apps/deployments. This is the permission your controller needs in production.

  • Regenerate the RBAC manifests by running:
make manifests

You should see controller-gen run and exit cleanly. The updated ClusterRole now lives in config/rbac/role.yaml.

Before you start the controller, what do you think will happen when it starts and sees the echo-sample CR that already exists on the cluster?

  • Start the controller by running:
make run

Watch the terminal output. You should see log lines showing the controller detecting your echo-sample CR, noticing no Deployment exists, and creating one.

  • Open a second terminal and check for pods by running:
kubectl get pods

You should see three pods with names starting with echo-sample-deployment, all in Running state. The reconciliation loop detected the gap between desired state (3 replicas) and actual state (0 pods) and drove the system to convergence.

No pods appearing?

  • Check the make run terminal for error messages. A common issue is a typo in the import path for appsv1alpha1 that compiles but fails at runtime.
  • Confirm your CR is still applied by running kubectl get echoservers. You should see echo-sample listed.
  • If you see a permission denied error, verify the RBAC marker is exactly // +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete and you ran make manifests after adding it.

Help me debug why my controller isn't creating pods

Close the feedback loop with Owns

Right now the controller only reconciles when the EchoServer CR itself changes. If someone manually scales the Deployment or deletes a pod directly, the controller will not notice until the next unrelated event triggers it. The Owns directive closes this gap.

  • Stop the running controller by pressing Ctrl+C in the make run terminal.
  • In internal/controller/echoserver_controller.go, find the SetupWithManager function at the bottom.
  • Find this line:
	return ctrl.NewControllerManagedBy(mgr).
		For(&appsv1alpha1.EchoServer{}).
		Named("echoserver").
		Complete(r)
  • Add Owns(&appsv1.Deployment{}) between For(...) and Named(...), so it looks like this:
	return ctrl.NewControllerManagedBy(mgr).
		For(&appsv1alpha1.EchoServer{}).
		Owns(&appsv1.Deployment{}).
		Named("echoserver").
		Complete(r)

What does Owns do?

Owns tells the controller-runtime to watch Deployments that have an owner reference pointing to an EchoServer. When any of those Deployments change (scaled, deleted, updated), the framework enqueues a reconciliation for the owning EchoServer. This closes the feedback loop: the controller reacts to downstream changes, not just spec changes.

  • Save the file and restart the controller by running:
make run

The controller starts and immediately reconciles. Your three pods are still running from before. The feedback loop is now closed.

Seeing an import error for appsv1?

The Owns(&appsv1.Deployment{}) line references the appsv1 import which should already exist at the top of your file. If you see an undefined: appsv1 error, check that appsv1 "k8s.io/api/apps/v1" is in your import block.

Help me fix the Owns import error

✔️ Awesome, I've got everything!

Great. Double check you have saved internal/controller/echoserver_controller.go and make run is running cleanly with no errors.

ⓧ I'd like to double check the full code

Your file should contain these sections in order. Scroll through and confirm each is present:

  • Package declaration and import block (including controllerutil and appsv1alpha1).
  • EchoServerReconciler struct with client.Client and Scheme.
  • Four RBAC markers (three for echoservers, one for deployments).
  • Reconcile function with OBSERVE (Get CR, Get Deployment), ACT (Create if not found), COMPARE (replicas + message), and ACT (Update if drifted).
  • buildDeployment helper returning a Deployment with nginx:1.27 and MESSAGE env var.
  • int32Ptr helper.
  • SetupWithManager with For, Owns, Named, and Complete.

Your controller is now a fully functioning control loop. It observes cluster state, compares it to the declared spec, and takes corrective action. In the next step, you will prove this by deliberately breaking the system and watching the controller repair it.

Test Self-Healing

Your controller is running and has driven the cluster to the desired state of 3 replicas. But a controller that works once and never gets tested under stress is not trustworthy.

In control theory, you verify a controller by disturbing the system and measuring how quickly it returns to the setpoint. This step makes the reconciliation loop viscerally real by breaking things on purpose and watching the controller repair them.

In this step, get ready to:
  • Perturb the system by deleting pods and manually scaling the Deployment, then watch the controller correct the drift.
  • Modify the desired state in the EchoServer CR and watch the controller update the Deployment to match.
Perturb the system and watch it self-heal

Your cluster currently has 3 running pods managed by echo-sample-deployment. You will destroy one and see what happens.

  • In the terminal where you run kubectl commands (not the one running the controller), list the current pods by running:
kubectl get pods

You should see 3 pods with names starting with echo-sample-deployment- followed by random suffixes. Copy one of the pod names.

Before you delete it, think about what you expect to happen. Will the pod stay deleted, or will something recreate it?

  • Delete one of the pods by running this command, replacing the pod name with one you copied:
kubectl delete pod [[POD_NAME="one of the pod names from the output above"]]
  • Immediately check the pods again by running:
kubectl get pods

You should see a brand new pod replacing the one you deleted. The old name is gone and a new pod with a fresh suffix appears, often with an age of just a few seconds.

Who recreated that pod?

The built-in Deployment controller inside Kubernetes did this. It has its own reconciliation loop that watches ReplicaSets and ensures the right number of pods are running. Your custom controller did not need to act because the Deployment spec was never changed.

This is nested control loops in action. Your controller owns the Deployment, and the Deployment controller owns the pods.

Now for a harder test. What happens if someone bypasses your controller and changes the Deployment directly?

  • Manually scale the Deployment down to 1 replica by running:
kubectl scale deployment/echo-sample-deployment --replicas=1

Before you check the pod count, predict what will happen. The Deployment now says 1 replica, but your EchoServer CR still says 3. Which one wins?

  • Wait 2-3 seconds, then check the pods by running:
kubectl get pods

You should see 3 pods again. Your controller detected the drift and corrected the Deployment back to 3 replicas.

  • Switch to the terminal running your controller and look at the logs.

You should see a log line saying Replica count drift detected followed by Updating Deployment to match desired state. That is your observe-compare-act loop firing in real time.

Why did this work?

Because you added Owns(&appsv1.Deployment{}) in SetupWithManager, any change to the owned Deployment triggers a re-reconciliation. When kubectl scale changed the Deployment, your controller was immediately notified.

The controller compared the Deployment's replica count (1) to the EchoServer spec (3), found they did not match, and updated the Deployment back to 3. The desired state always wins.

Still seeing only 1 pod?

Make sure your controller is still running in the other terminal. If make run crashed, restart it. The controller must be running to detect and correct drift.

Also check that the controller logs show the reconciliation firing. If you see no logs after the scale command, the watch may not be connected.

Why isn't my controller correcting the replica drift?

Modify the desired state and watch the controller follow

The previous test proved the controller repairs unauthorized changes to actual state. Now you will change the desired state itself and prove the controller drives actual state to match the new target.

  • Edit the EchoServer CR to change the replica count from 3 to 5 by running:
kubectl edit echoserver echo-sample

What happens when you run this?

Your default terminal editor (usually vi or nano) opens with the full YAML representation of the EchoServer resource.

  • Find the spec: section.
  • Change replicas: 3 to replicas: 5.
  • Save and close the editor (in vi, press Esc, type :wq, press Enter).

Before you check the pods, predict what you expect. You told the system "I want 5 replicas." Does the controller scale up immediately?

  • Check that the controller scaled up to 5 pods by running:
kubectl get pods

You should see 5 pods running. Two new ones will have a very recent age compared to the originals.

Still seeing only 3 pods?

Check that you saved the edit correctly. Run kubectl get echoserver echo-sample -o yaml and confirm spec.replicas is 5. If it still shows 3, the editor did not save properly.

Why didn't my replica count change after kubectl edit?

Now test one more dimension of the reconciliation loop. The controller also watches the message field.

  • Edit the EchoServer CR again to change the message by running:
kubectl edit echoserver echo-sample
  • Find the message: field under spec: and change it to any new value, such as Self-healing works!.
  • Save and close the editor.
  • Verify the Deployment picked up the new message by running:
kubectl describe deployment echo-sample-deployment

Scroll to the Containers section and look for the Environment field. You should see MESSAGE set to your new value.

  • Switch to your controller terminal and check the logs.

You should see Message drift detected followed by Deployment updated successfully. The controller noticed the gap between the CR's message field and the Deployment's env var, then corrected it.

What just happened across both tests?

You tested the two fundamental perturbation types. First, you changed actual state while desired state stayed fixed (the kubectl scale test). Second, you changed desired state and let the controller drive actual state to match (the kubectl edit tests).

Both paths triggered the same Reconcile function. It always observes, compares, and acts. The loop does not care how the drift happened. It only cares that drift exists.

Message not updating on the Deployment?

Make sure you edited the message field under spec:, not under metadata: or status:. The controller only reads from the spec.

My EchoServer message change is not reflected in the Deployment env var.

You have now proven your controller through perturbation. It repairs unauthorized changes, follows spec updates, and reports every action in its logs. Next up, you will extend the controller to report what it observes back into the CR's status, completing the full control loop.

Secret mission

Add Status Conditions

Your controller acts on drift, but it never reports what it observed. In control theory, the sensor feeds measured values back so operators can verify convergence. Can you close the information loop by adding status conditions that reflect real observed state?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally using a kind cluster inside Docker containers. There are no ongoing cloud costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • A local kind cluster (Docker containers running Kubernetes nodes).
  • Docker images cached locally (kind node image, nginx:1.27).
  • The echoserver-operator project folder on your Desktop.

Keep everything running

No action needed. Choose this if you're still actively building or experimenting with your controller.

  • The kind cluster persists until you explicitly delete it. It has no ongoing cost.
  • Docker images remain cached on disk but do not consume CPU or network unless containers are running.
  • Your project folder stays on your Desktop for future editing.

Pause - I'll come back to this later

Suspend the kind cluster's containers to free up CPU and memory without losing any state.

  • Pause the kind cluster containers by running:
docker pause $(kind get nodes -q)
  • When you're ready to resume, unpause the containers by running:
docker unpause $(kind get nodes -q)
  • After unpausing, verify the cluster is healthy by running:
kubectl cluster-info
  • Your project folder and Docker images stay on disk unchanged.

Delete - I don't want to use this again

Remove all project resources and start fresh. This frees Docker resources and disk space.

  • Delete the kind cluster by running:
kind delete cluster
  • Remove cached Docker images used by the project by running:
docker image prune -a --filter "label=io.x-k8s.kind.cluster"
  • Delete the project folder from your Desktop. On macOS or Linux, run:
rm -rf ~/Desktop/echoserver-operator
  • On Windows, run:
rmdir /s /q %USERPROFILE%\Desktop\echoserver-operator
  • Optionally, remove the Kubebuilder binary by running:
sudo rm /usr/local/bin/kubebuilder
  • Optionally, remove the kind binary by running:
rm $(go env GOPATH)/bin/kind

That's a Wrap!

That's a Wrap!

Nice work! You just built a custom Kubernetes controller from scratch and proved that reconciliation loops are the engine behind self-healing infrastructure.

You've learned how to:

  • Build a custom Kubernetes controller using Kubebuilder, implementing the full reconciliation loop pattern: observe actual state, compare to desired state, and act to close the gap.
  • Prove self-healing by deliberately breaking the system (deleting pods, scaling Deployments manually, editing the CR) and watching the control loop detect drift and repair it every time.
  • Connect Kubernetes design to control systems engineering: the Reconcile function is the actuator, the watch mechanism is the sensor, and the spec is the setpoint.
  • Secret Mission: Added status conditions to complete the feedback loop, reporting observed state back so operators can verify convergence.

Ready to quiz yourself?