Kubernetes Networking with Cilium + MetalLB

Build a local Kubernetes lab with eBPF networking, load balancing, and observability.

Introduction

30 Second Summary

Every application you deploy to Kubernetes depends on the network to function. But when traffic mysteriously drops or services can't reach each other, the network layer is often a black box. Understanding what happens to every packet inside your cluster is what separates confident operators from guesswork.

In this project, you will build a fully instrumented local Kubernetes networking lab. You'll replace the default networking layer with Cilium (powered by eBPF), deploy MetalLB for load balancing, and enable Hubble to observe every network flow in real time.

What You'll Build

A production-grade local Kubernetes networking lab with 4 nodes (1 control-plane + 3 workers), Cilium as the CNI, MetalLB providing real LoadBalancer IPs, and Hubble capturing every packet flowing through your cluster.

By the end of this project, you'll have:

  • A multi-node Kind cluster running Cilium as the CNI with eBPF-based networking and Hubble observability enabled.
  • A MetalLB deployment providing real external IPs via L2 advertisement on your local Docker network.
  • Real-time Hubble flow observations showing DNS resolution, service routing, and pod-to-pod traffic at the packet level.
  • Secret Mission: Deploy a NetworkPolicy that blocks traffic, then use Hubble to forensically diagnose the dropped packets.

Are there any prerequisites?

This project assumes you have Docker, kubectl, and Helm already installed on Linux. You should be familiar with Kubernetes pods, services, and deployments.

Kind, the Cilium CLI, and the Hubble CLI will all be installed during the project.

Install Kind and Create a Multi-Node Cluster

A Kubernetes cluster needs a networking layer to give pods IP addresses and route traffic between them. Without a Container Network Interface (CNI) plugin installed, nodes stay in a NotReady state because the cluster has no way to wire up pod networking.

In this step, you will create a 4-node local cluster using Kind and deliberately leave the default CNI disabled. This sets the stage for Cilium to take over networking in the next step.

In this step, get ready to:
  • Install the Kind binary and create a cluster config that disables the default CNI.
  • Spin up a 4-node Kind cluster and confirm nodes are NotReady.
  • Install the Cilium CLI for validating Cilium installations later.
Install Kind v0.32.0

Kind (Kubernetes IN Docker) runs cluster nodes as Docker containers on your local machine. You need the Kind binary to create and manage these local clusters.

  • Download the Kind v0.32.0 binary, make it executable, and move it onto your PATH by running these commands:
# Download the Kind binary for Linux amd64
[ $(uname -m) = x86_64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.32.0/kind-$(uname)-amd64

# Make it executable and move to PATH
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind

What does this command do?

The curl command downloads the Kind binary from the official release URL for your architecture.

chmod +x marks it as executable, and sudo mv places it in /usr/local/bin so you can run kind from any terminal session.

  • Confirm Kind installed correctly by running:
kind version
  • You should see kind v0.32.0 in the output.

Seeing command not found?

Make sure /usr/local/bin is in your PATH. Run echo $PATH to check.

If you are on ARM64 (aarch64), replace the download command with the arm64 variant: change x86_64 to aarch64 and amd64 to arm64 in the curl URL.

Help me troubleshoot why the kind command is not found on my Linux machine.

Now you need a config file that tells Kind what shape of cluster to create. The key setting is disabling the default CNI so Cilium can be installed as the networking layer instead.

  • Create a file called kind-config.yaml with the following content:
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
  - role: worker
  - role: worker
networking:
  disableDefaultCNI: true
  podSubnet: "10.244.0.0/16"

What does this config do?

  • nodes defines 4 Docker containers: 1 control-plane and 3 workers. Each acts as a full Kubernetes node.
  • disableDefaultCNI: true prevents Kind from installing its built-in CNI (kindnet). This leaves networking unconfigured on purpose.
  • podSubnet: "10.244.0.0/16" explicitly sets the pod CIDR range that Cilium will use to assign IPs to pods.
  • Save kind-config.yaml.
  • Confirm the file exists by running:
ls kind-config.yaml
  • You should see kind-config.yaml in the output.

✔️ Awesome, I've got everything!

Great. Double check you have saved kind-config.yaml before moving on.

ⓧ I'd like to double check the full code

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
  - role: worker
  - role: worker
networking:
  disableDefaultCNI: true
  podSubnet: "10.244.0.0/16"
Create the cluster and verify nodes are NotReady

With the config ready, you can spin up the cluster. Because disableDefaultCNI is set to true, the nodes will appear as NotReady until a CNI plugin gives them the ability to assign pod IPs.

  • Create the Kind cluster by running this command:
kind create cluster --config=kind-config.yaml

What does this command do?

Kind reads your config file and launches 4 Docker containers, each running a full Kubernetes node image (kindest/node:v1.36.1). It also configures kubectl to point at the new cluster by setting the context to kind-kind.

  • Wait for the command to finish. This takes 30 to 60 seconds.
  • Verify your cluster exists by running:
kind get clusters
  • You should see kind printed as the cluster name.
  • Check the node status by running:
kubectl get nodes
  • You should see 4 nodes listed, all showing NotReady in the STATUS column.

Why are the nodes NotReady?

Kubernetes requires a CNI plugin to set up pod networking. Without one, the kubelet on each node reports NotReady because it cannot assign IP addresses to pods.

This is exactly what we want. In the next step, you will install Cilium as the CNI and watch the nodes transition to Ready.

Cluster creation failed?

Make sure Docker is running. Kind uses Docker to create node containers. Run docker ps to confirm Docker responds.

If you see a port conflict or the cluster already exists, delete any previous cluster first with kind delete cluster and try again.

Help me debug why my Kind cluster creation is failing.

Install the Cilium CLI

The Cilium CLI is a separate tool from Cilium itself. It validates installations, checks cluster health, and manages Hubble observability.

  • Download and install the Cilium CLI by running these commands:
# Fetch the latest stable version tag
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi

# Download the binary and checksum
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}

# Verify integrity and install
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}

What does this script do?

  • It fetches the latest stable Cilium CLI version string from the project's GitHub repository.
  • It downloads the correct tarball for your CPU architecture along with a SHA-256 checksum.
  • It verifies the download integrity, extracts the binary into /usr/local/bin, and cleans up the downloaded files.
  • Confirm the Cilium CLI installed correctly by running:
cilium version --client
  • You should see a version number in the output. The exact version depends on when you run the install script.

Seeing permission denied or command not found?

If you get a permission error, make sure you included sudo before the tar command. The /usr/local/bin directory requires root access.

If the checksum verification fails, delete the files and rerun the curl command. This usually means a partial download.

Help me fix the Cilium CLI installation on Linux.

Your cluster is up, but intentionally broken. Next up, you will install Cilium as the CNI and watch those NotReady nodes come alive.

Deploy Cilium as the CNI

Your 4-node cluster is running, but every node shows NotReady. That is because there is no Container Network Interface plugin installed yet. Without a CNI, pods cannot receive IP addresses and cannot communicate with each other.

In this step, you will install Cilium as your cluster's CNI. Cilium uses eBPF programs attached directly to the Linux kernel to route packets between pods. You will also enable Hubble for network observability from the start.

In this step, get ready to:
  • Add the Cilium Helm repo and preload the container image into Kind nodes.
  • Install Cilium via Helm with Hubble relay and UI enabled.
  • Validate that all nodes transition to Ready and a test pod gets an IP.
Add the Cilium Helm repo and preload the image

Cilium is distributed as a Helm chart. Before installing it, you need to add the chart repository and preload the container image into every Kind node. Preloading avoids slow image pulls from within the cluster.

  • Add the Cilium Helm repository by running this command:
helm repo add cilium https://helm.cilium.io/
  • Pull the Cilium v1.19.4 container image to your local Docker cache by running:
docker pull quay.io/cilium/cilium:v1.19.4
  • Load the image into all Kind cluster nodes by running:
kind load docker-image quay.io/cilium/cilium:v1.19.4

What do these commands do?

Kind nodes are Docker containers that run their own containerd runtime. They cannot pull images from your host's Docker cache automatically.

The kind load docker-image command copies the image from your local Docker daemon into the containerd image store of every node in the cluster. This ensures Cilium pods start instantly without needing network access to a registry.

  • Verify the image is loaded by checking the output from the kind load command. You should see a confirmation message for each node.

Image load taking too long?

The docker pull step downloads approximately 200 MB. If your connection is slow, wait for it to finish before running the load command.

If you see a "not found" error from kind load, confirm the image tag matches exactly: quay.io/cilium/cilium:v1.19.4.

help me troubleshoot loading the Cilium image into Kind

Install Cilium via Helm with Hubble enabled

Now you will install Cilium into the cluster with a single Helm command. This command configures Cilium to use the Kubernetes IPAM mode (which respects the 10.244.0.0/16 pod subnet you defined in kind-config.yaml) and enables Hubble observability from the start.

  • Install Cilium into the kube-system namespace by running:
helm install cilium cilium/cilium --version 1.19.4 \
  --namespace kube-system \
  --set image.pullPolicy=IfNotPresent \
  --set ipam.mode=kubernetes \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true

What do these Helm flags do?

  • image.pullPolicy=IfNotPresent tells Cilium to use the image you preloaded instead of pulling from the internet.
  • ipam.mode=kubernetes configures Cilium's IP Address Management to use the Kubernetes-native pod CIDR you set in the Kind config.
  • hubble.relay.enabled=true deploys Hubble Relay, which aggregates flow data from all nodes into a single query endpoint.
  • hubble.ui.enabled=true deploys the Hubble UI dashboard for visual network flow inspection.
  • Wait for Helm to confirm the release was deployed. You should see a message indicating the Cilium release has been installed.

Helm install timed out?

If the install hangs, the most common cause is the image not being preloaded. Run docker images | grep cilium to confirm the image exists locally, then re-run the kind load command from the previous substep.

If you see a namespace error, make sure you did not accidentally delete the kube-system namespace. It exists by default on every Kubernetes cluster.

help me fix a Cilium Helm install failure on Kind

Validate Cilium and watch nodes become Ready

With Cilium installed, the cluster now has a functioning network layer. Every node should transition from NotReady to Ready as Cilium assigns pod CIDRs and configures networking.

  • Validate that Cilium is fully operational by running:
cilium status --wait

What does this command do?

The --wait flag tells the Cilium CLI to keep checking until all components report healthy. It will wait up to 5 minutes by default.

When everything is working, you should see Cilium: OK, Operator: OK, and Hubble Relay: OK in the output, along with deployment readiness counts confirming that hubble-relay and hubble-ui are running.

  • Confirm all 4 nodes are now Ready by running:
kubectl get nodes
  • You should see all 4 nodes (1 control-plane and 3 workers) with status Ready.
  • Deploy a test pod and verify it receives an IP from the Cilium IPAM by running:
kubectl run test-pod --image=nginx
kubectl wait --for=condition=Ready pod/test-pod --timeout=60s
kubectl get pod test-pod -o wide

What should I see?

The kubectl get pod test-pod -o wide output shows the pod's IP address in the IP column. This address should be within the 10.244.0.0/16 range you configured in kind-config.yaml. That confirms Cilium's IPAM is allocating addresses from the correct subnet.

Nodes still showing NotReady?

Run kubectl get pods -n kube-system and look for Cilium pods. If they show ImagePullBackOff, the image was not preloaded correctly. Re-run the docker pull and kind load commands, then delete the failing pods so they restart with the local image.

If cilium status --wait times out after 5 minutes, check the Cilium agent logs with kubectl logs -n kube-system -l k8s-app=cilium for error details.

help me troubleshoot Cilium not becoming ready on my Kind cluster

Your cluster now has a fully functioning CNI. Every pod gets an IP address, and Hubble is collecting network flow data in the background. Next up, you will deploy MetalLB to give your services real LoadBalancer IPs.

Deploy MetalLB for LoadBalancer Services

Your cluster has Cilium handling pod networking, and all four nodes are Ready. But there is still a gap in your cluster's capabilities.

If you create a Kubernetes Service of type LoadBalancer right now, it stays stuck in "Pending" forever. That is because there is no cloud provider to hand out external IPs. MetalLB fills this gap for bare-metal and local clusters by allocating real IP addresses and announcing them via ARP so your host machine can reach cluster services directly.

In this step, get ready to:
  • Install MetalLB via Helm into the cluster.
  • Configure an IP address pool on the Kind Docker network.
  • Expose a sample app with a working LoadBalancer IP.
Install MetalLB via Helm and wait for controller readiness

MetalLB needs two components running in your cluster: a controller (which assigns IPs) and a speaker (which advertises them via ARP on each node).

  • Add the MetalLB Helm repo and install MetalLB into the metallb-system namespace by running these commands:
# Add the MetalLB Helm chart repository
helm repo add metallb https://metallb.github.io/metallb

# Install MetalLB into its own namespace
helm install metallb metallb/metallb \
  --namespace metallb-system \
  --create-namespace \
  --wait

What do these flags do?

  • helm repo add registers the MetalLB chart repository so Helm can find it.
  • --namespace metallb-system --create-namespace installs MetalLB into a dedicated namespace (created if it does not exist).
  • --wait tells Helm to block until all pods are Ready before returning. This ensures the controller and speaker are fully operational before you apply configuration.
  • Confirm the MetalLB pods are running by running:
kubectl get pods -n metallb-system

You should see a controller pod and one speaker pod per node (4 speakers total), all in Running status with 1/1 Ready.

Pods stuck in Pending or CrashLoopBackOff?

If the controller pod is not Ready after 90 seconds, check its logs with kubectl logs -n metallb-system -l app.kubernetes.io/component=controller. Common causes include image pull failures (check your internet connection) or webhook certificate issues.

If speaker pods are crashing, ensure your Kind nodes have the NET_RAW capability. Kind nodes run as privileged containers, so this should not be an issue unless Docker is configured with restrictive seccomp profiles.

Help me debug my MetalLB pods not starting

Discover the Kind Docker network and create an IPAddressPool

MetalLB needs to know which IP addresses it can hand out. These IPs must live on the same network as the Kind nodes so that ARP announcements reach your host.

  • Find the subnet of the Kind Docker bridge network by running:
docker network inspect kind -f '{{(index .IPAM.Config 0).Subnet}}'

You should see 172.18.0.0/16 (or a similar /16 subnet). Your Kind nodes and the MetalLB LoadBalancer IPs all live on this network.

Why does the subnet matter?

MetalLB uses L2 mode (ARP) to announce LoadBalancer IPs. For your host to reach those IPs, they must be on the same Layer 2 broadcast domain as the Kind nodes. The Kind Docker bridge network provides this shared broadcast domain.

You will carve out a small slice at the top of the subnet (172.18.255.200-172.18.255.250) so MetalLB IPs do not collide with node IPs that Docker assigns from the bottom of the range.

  • Create the IPAddressPool manifest by running:
cat <<EOF > metallb-ipaddresspool.yaml
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: kind-pool
  namespace: metallb-system
spec:
  addresses:
    - 172.18.255.200-172.18.255.250
EOF

What does this manifest do?

  • IPAddressPool tells MetalLB which IP addresses it can allocate to LoadBalancer services.
  • The range 172.18.255.200-172.18.255.250 gives MetalLB 51 addresses at the top of the Kind subnet, well away from node IPs at the bottom.
  • Create the L2Advertisement manifest to tell MetalLB to announce IPs via ARP:
cat <<EOF > metallb-l2advertisement.yaml
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: kind-l2advert
  namespace: metallb-system
spec:
  ipAddressPools:
    - kind-pool
EOF

What is an L2Advertisement?

An L2Advertisement tells MetalLB's speaker pods to respond to ARP requests for any IP in the referenced pool. When your host asks "who has 172.18.255.200?" on the Docker bridge, a speaker responds with its MAC address, routing traffic into the cluster.

  • Apply both manifests by running:
kubectl apply -f metallb-ipaddresspool.yaml
kubectl apply -f metallb-l2advertisement.yaml

You should see two resources created: ipaddresspool.metallb.io/kind-pool and l2advertisement.metallb.io/kind-l2advert.

Seeing a webhook error on apply?

If you see an error about the validating webhook not being ready, the MetalLB controller's webhook server has not finished initializing. Wait 10-20 seconds and try the kubectl apply commands again. The --wait flag during install should prevent this, but network latency can sometimes cause a brief window.

Help me fix MetalLB webhook validation errors

Deploy a sample app and expose it as LoadBalancer

Time to prove MetalLB works. You will deploy nginx and expose it through a LoadBalancer service that gets a real external IP.

  • Create an nginx deployment and expose it as a LoadBalancer service by running:
# Deploy a single nginx pod
kubectl create deployment nginx --image=nginx

# Expose it as a LoadBalancer service on port 80
kubectl expose deployment nginx --port=80 --type=LoadBalancer

What does this do?

  • kubectl create deployment nginx --image=nginx creates a Deployment running one nginx pod.
  • kubectl expose deployment nginx --port=80 --type=LoadBalancer creates a Service of type LoadBalancer. MetalLB immediately allocates an IP from your pool and announces it via ARP.
  • Check that MetalLB assigned an external IP by running:
kubectl get svc nginx

You should see an EXTERNAL-IP from your pool (e.g. 172.18.255.200), not <pending>.

  • Verify connectivity from your host by curling the external IP:
curl [[NGINX_EXTERNAL_IP="the EXTERNAL-IP from kubectl get svc nginx"]]

You should see the nginx welcome page HTML in your terminal output. This confirms that traffic flows from your host, through the Docker bridge, to the MetalLB speaker, and into the nginx pod.

EXTERNAL-IP still showing Pending?

If the service stays in Pending, MetalLB has not assigned an IP. Check the following:

  • Confirm the IPAddressPool was created: kubectl get ipaddresspool -n metallb-system. You should see kind-pool.
  • Confirm the L2Advertisement exists: kubectl get l2advertisement -n metallb-system. You should see kind-l2advert.
  • Check MetalLB controller logs: kubectl logs -n metallb-system -l app.kubernetes.io/component=controller.

Help me debug why my LoadBalancer service has no external IP

✔️ Awesome, I've got everything!

Great. Double check you have saved both YAML files and that your curl returned the nginx welcome page.

ⓧ I'd like to double check the full code

Here are the complete manifests you should have:

metallb-ipaddresspool.yaml

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: kind-pool
  namespace: metallb-system
spec:
  addresses:
    - 172.18.255.200-172.18.255.250

metallb-l2advertisement.yaml

apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: kind-l2advert
  namespace: metallb-system
spec:
  ipAddressPools:
    - kind-pool

Your cluster now has a working load balancer. Any service you create with type LoadBalancer gets a real, routable IP from your pool. Next up, you will use Hubble to observe the network flows as traffic moves through MetalLB and Cilium into your pods.

Observe Network Flows with Hubble

Your cluster has a working CNI, load balancer, and a live service. But when something breaks in production, you need to see what is actually happening at the packet level.

In this step, you will install the Hubble CLI on your host machine and use it to observe real-time network flows. You will watch traffic traverse from your host through MetalLB to the nginx pod, and inspect DNS resolution at the packet level.

In this step, get ready to:
  • Install the Hubble CLI and connect to Hubble Relay.
  • Observe real-time traffic flows through your cluster.
  • Inspect DNS resolution flows with CoreDNS.
Install the Hubble CLI and port-forward to Hubble Relay

Hubble collects flow data from every Cilium agent in your cluster. The Hubble CLI lets you query that data from your local machine by connecting to Hubble Relay.

  • Download and install the Hubble CLI binary by running these commands:
HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/main/stable.txt)
HUBBLE_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then HUBBLE_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum}
sha256sum --check hubble-linux-${HUBBLE_ARCH}.tar.gz.sha256sum
sudo tar xzvfC hubble-linux-${HUBBLE_ARCH}.tar.gz /usr/local/bin
rm hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum}

What does this do?

This fetches the latest stable Hubble release version, downloads the correct binary for your CPU architecture, verifies the checksum, and installs the hubble binary into /usr/local/bin.

  • Verify the installation succeeded by running:
hubble version
  • You should see a version number printed in the output.

Command not found?

Make sure /usr/local/bin is in your PATH. You can check with echo $PATH. If it is missing, add it with export PATH=$PATH:/usr/local/bin and retry.

help me debug my Hubble CLI installation

  • Start port-forwarding to the Hubble Relay service by running:
cilium hubble port-forward&

What does this command do?

This forwards port 4245 on your local machine to the Hubble Relay service running inside your cluster. The Hubble CLI connects to localhost:4245 by default.

The trailing & runs the port-forward in the background so you can keep using your terminal.

  • Confirm Hubble Relay connectivity by running:
hubble status
  • You should see output showing a healthy connection with all 4 nodes connected.

What should I see?

The output shows a healthcheck status of Ok via localhost:4245. It also displays the current number of stored flows, flows per second, and a connected nodes count showing 4/4. This confirms the Hubble CLI can reach all agents in your cluster through the Relay.

Seeing connection refused?

If Hubble Relay is not yet fully started, wait 30 seconds and retry. You can check the relay pod status with kubectl -n kube-system get pods -l k8s-app=hubble-relay. The pod must be in Running state with 1/1 containers ready.

help me troubleshoot Hubble Relay connection issues

Generate traffic and observe flows

Now that Hubble is connected, every packet flowing through your cluster is being recorded. Let's generate some traffic and watch it in real time.

  • Get the EXTERNAL-IP of your nginx service by running:
kubectl get svc nginx --no-headers -o custom-columns=IP:.status.loadBalancer.ingress[0].ip
  • Curl the LoadBalancer IP to generate traffic by running:
curl $(kubectl get svc nginx --no-headers -o custom-columns=IP:.status.loadBalancer.ingress[0].ip)
  • You should see the nginx welcome page HTML in your terminal output.
  • Now observe the traffic flows that Hubble captured by running:
hubble observe --last 20

What does this show?

Each line in the output represents a single network flow. You will see the timestamp, source pod (or external identity), destination pod, port, protocol, and a verdict.

The verdict FORWARDED means Cilium allowed the packet to pass. Look for flows showing traffic hitting the nginx pod on port 80.

  • Filter the observation to only show flows in the default namespace by running:
hubble observe --namespace default --last 20
  • Filter by verdict to see only successfully forwarded packets by running:
hubble observe --namespace default --verdict FORWARDED --last 20
  • You should see flows showing your curl request arriving at the nginx pod with the FORWARDED verdict, source and destination IP addresses, and port 80.

Why filter by verdict?

In production, clusters generate thousands of flows per second. Filtering by namespace, pod label, or verdict lets you isolate the exact traffic you care about.

The --verdict FORWARDED flag hides background health checks and system traffic so you only see successful application flows.

Not seeing any flows?

Make sure you ran the curl command before running hubble observe. Hubble shows recently captured flows, so traffic must have occurred first. Try running the curl again, then re-run the observe command.

help me see flows in Hubble observe output

Inspect DNS flows and service routing

DNS resolution is one of the most common failure points in Kubernetes networking. Hubble can show you every DNS query and response at the packet level, revealing exactly how CoreDNS resolves service names.

  • Deploy a DNS utilities pod by running:
kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml
  • Wait for the dnsutils pod to become Ready by running:
kubectl wait --for=condition=Ready pod/dnsutils --timeout=60s
  • Perform a DNS lookup from inside the pod by running:
kubectl exec -i -t dnsutils -- nslookup kubernetes.default
  • You should see the DNS server address (CoreDNS ClusterIP) and the resolved address for the kubernetes.default service.
  • Also look up the nginx service to generate a DNS flow for your own app:
kubectl exec -i -t dnsutils -- nslookup nginx.default
  • Now use Hubble to inspect the DNS flows by running:
hubble observe --protocol dns --last 30

What do the DNS flows show?

Each DNS flow shows the dnsutils pod as the source and a CoreDNS pod as the destination. You can see the port (53), the flow direction, and the FORWARDED verdict confirming the query was allowed.

This is how you trace DNS failures in production. If a pod cannot reach a service by name, the DNS flows reveal whether the query reached CoreDNS and whether a response was returned.

  • Filter DNS flows to only show traffic from the dnsutils pod by running:
hubble observe --protocol dns --pod dnsutils --last 20
  • You should see flows showing the dnsutils pod communicating with CoreDNS pods, with both the query and response directions visible.

DNS flows not appearing?

Confirm the dnsutils pod is running with kubectl get pod dnsutils. If it shows Pending, the pod has not been scheduled yet. Also verify that Hubble port-forwarding is still active in the background. If the background process was terminated, restart it with cilium hubble port-forward&.

help me see DNS flows in Hubble

You now have full packet-level visibility into your cluster. You can trace any request from client to service to backend pod, and diagnose DNS issues in real time. Next up, the Secret Mission challenges you to use this observability to perform network policy forensics.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally, so there are no ongoing financial costs. The Kind cluster nodes do consume RAM and CPU as Docker containers while running.

Resources you used:

  • Kind cluster (4 Docker containers acting as Kubernetes nodes).
  • CLI tools installed to /usr/local/bin/: kind, cilium, hubble
  • Local YAML config files: kind-config.yaml, metallb-ipaddresspool.yaml, metallb-l2advertisement.yaml, cilium-netpol-allow-good.yaml.
  • Docker images pulled during the project (quay.io/cilium/cilium:v1.19.4, kindest/node, MetalLB images).

Keep everything running

No action needed. Choose this if you are still actively experimenting with Cilium, MetalLB, or Hubble in your cluster.

  • Your Kind cluster continues running as Docker containers, consuming memory and CPU.
  • All Kubernetes resources (pods, services, CiliumNetworkPolicies) remain in place.
  • The CLI tools (kind, cilium, hubble) remain installed on your system.

Pause - I'll come back to this later

Stop the Kind cluster containers to free up memory while keeping the cluster intact for later.

  • Stop all Kind node containers by running:
docker stop $(kind get nodes)
  • When you want to resume, start the containers again by running:
docker start $(kind get nodes)

What happens when you pause?

Stopping the Docker containers shuts down the Kubernetes nodes. All cluster state is preserved on disk. When you start the containers again, the kubelet and all pods restart automatically.

Delete - I don't want to use this again

Remove all project resources and start fresh. This frees up disk space and removes the cluster entirely.

  • Delete the Kind cluster by running:
kind delete cluster
  • Remove the local YAML files by running:
rm kind-config.yaml metallb-ipaddresspool.yaml metallb-l2advertisement.yaml cilium-netpol-allow-good.yaml
  • Remove the CLI tools by running:
sudo rm /usr/local/bin/kind /usr/local/bin/cilium /usr/local/bin/hubble
  • Remove the Docker images pulled during the project by running:
docker rmi quay.io/cilium/cilium:v1.19.4
docker rmi $(docker images 'kindest/node' -q)
docker rmi $(docker images 'quay.io/metallb/*' -q)

What about kubectl and Helm?

kubectl, Helm, and Docker were already on your system before this project. The steps above only remove what this project installed.

Nice Work!

Nice Work!

You just built a production-grade Kubernetes networking lab from scratch. You now have a 4-node cluster with eBPF-powered networking, bare-metal load balancing, and real-time packet observability.

You've learned how to:

  • Built a multi-node Kind cluster with the default CNI disabled and deployed Cilium as the eBPF-powered network layer providing pod IP addressing via IPAM.
  • Deployed MetalLB with L2 advertisement to assign real LoadBalancer IPs on a local Docker network and serve external traffic.
  • Used Hubble to observe real-time network flows including DNS resolution, service routing, and pod-to-pod traffic at the packet level.
  • (Secret Mission) Created CiliumNetworkPolicy rules to intentionally block traffic, then used Hubble forensics to observe DROPPED and FORWARDED verdicts side-by-side to diagnose policy enforcement at the eBPF level.

Ready to quiz yourself?