Build a Node Collector with DaemonSets

Deploy a node identity collector to every Kubernetes node with a DaemonSet.

Introduction

30 Second Summary

A job can be running successfully while part of your system stays completely uncovered. That hidden gap matters when every machine needs its own watcher.

In this project, you will build a node identity collector in a two-node Kubernetes playground. A one-replica Deployment exposes the coverage gap before a DaemonSet takes over every eligible node.

What You'll Build

Your finished demo shows collector Pods reporting separate node identities across both nodes before Kubernetes restores full coverage after a deletion.

By the end of this project, you'll have:

  • A collector log that displays each node identity through the Downward API.
  • A two-node comparison that makes the one-replica coverage gap visible before the DaemonSet covers every eligible node.
  • A self-healing demonstration that shows full node coverage returning after you delete a collector Pod.
  • Secret Mission: Restrict the DaemonSet to one selected node using a node label plus nodeSelector.

Are there any prerequisites?

You only need a modern web browser with internet access. Killercoda provides the Kubernetes cluster, kubectl, and the Theia browser IDE with no local installation.

Before We Start

Before any hands-on work, commit to building a BusyBox collector that reads each Pod's node name through the Downward API. A one-replica Deployment reveals a coverage gap that the DaemonSet closes by maintaining one collector on every eligible node.

Open the Two-Node Playground

A local Kubernetes setup would use this session on installation instead of node scheduling. Killercoda provides the ready-made cluster you need in a browser.

The playground gives you a two-node Kubernetes v1.36 cluster with kubectl already connected.

The Theia browser IDE sits in the same session. You can use it later without configuring a local editor.

In this step, get ready to:
  • Launch the FREE two-node Kubernetes playground.
  • Confirm that kubectl can reach the Kubernetes server.
  • Verify that the control-plane node and worker node are Ready.
Launch the FREE playground

The FREE session has a maximum of 1 hour. This project fits within that window.

  • Visit the Killercoda Kubernetes Playgrounds page in your browser.
  • Complete Killercoda's on-screen sign-in flow.
  • Select the environment named 1.36 | Two Node.
  • Wait until the browser terminal accepts input.
  • Confirm that the Theia browser IDE has loaded alongside the terminal.

Good start. Your browser session is ready for the cluster checks.

Verify kubectl can reach the cluster

The kubectl client sends requests to the cluster's Kubernetes API. A server version in the response proves that this connection works.

Before you run this check, do you expect kubectl to report only its client or a connected server too?

  • Verify the connection by running this command:
kubectl version

What Does This Check Prove?

  • The kubectl version command requests version information from the local client and the Kubernetes server.
  • A server version confirms that the terminal can communicate with the cluster's API.

The output should include a server version beginning with v1.36. That confirms kubectl is connected to the expected playground cluster.

No Server Version?

  • Wait until the playground finishes preparing the cluster.
  • Repeat the version check in the browser terminal.

If the server still does not respond, help me troubleshoot the playground connection.

Confirm both nodes are Ready

The collector eventually needs one eligible node for each Pod. The node table confirms that both playground machines can accept workloads.

Before you run this check, how many Ready rows do you expect to see?

  • Check the cluster's nodes by running this command:
kubectl get nodes

How Do You Read the Node Table?

  • Each row represents one node that belongs to the cluster.
  • The status value shows whether Kubernetes considers that node healthy enough to accept workloads.
  • The role information distinguishes the control-plane node from the worker node.

You should see two node rows with Ready status. One is the control-plane node. The other is the worker node.

Fewer Than Two Ready Nodes?

  • Wait until the playground finishes preparing both nodes.
  • Repeat the node check from above.

If either node remains unavailable, help me troubleshoot the two-node playground.

That's your environment ready. Next, you'll use a one-replica Deployment to expose the gap between a running collector and full node coverage.

Expose the Deployment Coverage Gap

Your two-node Kubernetes cluster is Ready after the last step. You can now test how a controller turns desired state into running Pods.

A one-replica Deployment gives Kubernetes a replica target. This step checks whether that target also guarantees node coverage.

In this step, get ready to:
  • Build a one-replica collector manifest that receives its node name through the Downward API.
  • Run the Deployment in your two-node cluster and inspect its assigned node.
  • Compare the collector log with the two Ready nodes to reveal the coverage gap.
Create the one-replica manifest

A Deployment matches its selector labels with the labels in its Pod template. This matching contract identifies the Pods that belong to its desired state.

  • In the Theia file list from earlier, create a new file named deployment.yaml.

Good, your collector manifest now has a clear home in the playground workspace.

A Kubernetes manifest can be assembled in connected sections before it is applied. This first section defines the controller identity, replica target, label contract, and scheduling permissions.

  • Add the Deployment identity and scheduling rules to deployment.yaml by pasting this first section:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: node-identity-collector
  labels:
    app: node-identity-collector
spec:
  replicas: 1
  selector:
    matchLabels:
      app: node-identity-collector
  template:
    metadata:
      labels:
        app: node-identity-collector
    spec:
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
        - key: node-role.kubernetes.io/master
          operator: Exists
          effect: NoSchedule

What does this section do?

  • The apiVersion and kind values identify this resource as a Deployment.
  • The replicas: 1 value sets the desired Pod count to one.
  • The matching app: node-identity-collector labels connect the selector with the Pod template.
  • The two tolerations permit the collector to run on the control-plane node when the scheduler selects it.
  • Save deployment.yaml.
  • Check that deployment.yaml remains visible in the Theia file list after the save.

Does the first section look misaligned?

  • Check that each nested YAML level uses spaces consistently.
  • Confirm that both toleration entries sit beneath the same tolerations key.

Ask for help checking the structure: Why is the first section of my Kubernetes Deployment manifest misaligned?

The Pod template also needs a container that can report its assigned node. BusyBox version 1.38.0 provides the small shell process that prints this identity.

  • Complete the Pod template below the second effect: NoSchedule line by pasting this section:
      containers:
        - name: collector
          image: busybox:1.38.0
          command: ["sh", "-c"]
          args:
            - |
              echo "Node identity: ${NODE_NAME}"
              while true; do sleep 3600; done
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName

How does the collector work?

  • The pinned BusyBox image gives the collector a small shell environment.
  • The container prints its assigned node identity once when it starts.
  • The sleep loop keeps the collector running so Kubernetes can continue managing it.
  • The NODE_NAME environment variable receives the Pod's node name from spec.nodeName.
  • Save deployment.yaml.
  • Check that containers aligns with tolerations beneath the Pod spec.

Does the container section sit outside the Pod spec?

  • Keep six leading spaces before containers so it lines up with tolerations.
  • Keep fieldPath: spec.nodeName nested beneath fieldRef.

Get help with the completed structure: Why is the container section in my deployment.yaml outside the Pod spec?

✔️ Awesome, I've got everything!

Your saved deployment.yaml manifest is ready for the cluster.

ⓧ I'd like to double check the full code

apiVersion: apps/v1
kind: Deployment
metadata:
  name: node-identity-collector
  labels:
    app: node-identity-collector
spec:
  replicas: 1
  selector:
    matchLabels:
      app: node-identity-collector
  template:
    metadata:
      labels:
        app: node-identity-collector
    spec:
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
        - key: node-role.kubernetes.io/master
          operator: Exists
          effect: NoSchedule
      containers:
        - name: collector
          image: busybox:1.38.0
          command: ["sh", "-c"]
          args:
            - |
              echo "Node identity: ${NODE_NAME}"
              while true; do sleep 3600; done
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName

How does the full manifest fit together?

The first half defines which Pods the Deployment manages. The second half defines what each managed Pod runs.

Together, these sections create one collector that can report the node selected by Kubernetes.

Apply the Deployment and inspect placement

Applying the manifest turns the saved desired state into a live cluster resource. Kubernetes then schedules the requested Pod onto an eligible node.

  • Apply the manifest to the cluster by running:
kubectl apply -f deployment.yaml

What does this command do?

The command sends the configuration in deployment.yaml to the Kubernetes API. Kubernetes creates the Deployment because that resource does not exist yet.

Good, your saved manifest is now a live Deployment managed by the cluster.

Did the manifest fail to apply?

  • Check that the terminal is using the same playground workspace where you saved deployment.yaml.
  • Compare your indentation with the full manifest above if the command reports a YAML parsing problem.
  • Confirm that both selector and Pod-template labels use app: node-identity-collector.

Ask for help with the apply failure: Why is kubectl unable to apply my deployment.yaml manifest?

Before you inspect the result, predict where Kubernetes placed the collector. Do you expect one node or both nodes to have a collector?

  • Inspect the collector's status and assigned node by running:
kubectl get pods -o wide

What does the wide view show?

The wide Pod listing includes placement information alongside each Pod's status. This lets you connect the collector with the node chosen by the scheduler.

  • Repeat the previous check until the collector status reaches Running.

You will see one running collector Pod assigned to one node. The other Ready node has no matching collector.

That shortfall is the result you were looking for. A replica target of one produced a working Pod without guaranteeing coverage across both nodes.

Is the collector still starting?

  • Give the cluster time to obtain the BusyBox image before repeating the Pod check.
  • Confirm that the image value in deployment.yaml is busybox:1.38.0.
  • Check that the container command and argument indentation match the full manifest.

Get help with the collector status: Why is my node identity collector Pod not reaching Running status?

Read the collector's node identity

The wide listing proves where the Pod runs. The collector log proves that the Downward API supplied the same node identity inside the container.

  • Confirm that the cluster still has two Ready nodes by running:
kubectl get nodes

Why check the nodes again?

This command gives you the cluster-wide baseline for the comparison. Both the control-plane node and worker node should remain Ready.

You should see the same two Ready nodes from the previous step.

Before you read the collector log, predict how many Node identity: lines one replica can produce.

  • Read the output from every container with the collector label by running:
kubectl logs -l app=node-identity-collector --all-containers=true

How does the log command find the collector?

The label selector finds Pods carrying the collector's shared app label. The all-containers option includes output from every container in each matching Pod.

You will see exactly one Node identity: line naming the node that runs the collector. That node name matches the placement shown in the wide Pod listing.

Is the node identity missing?

  • Confirm that the collector Pod reached Running before reading its logs.
  • Check that the Pod-template label matches app: node-identity-collector exactly.
  • Verify that NODE_NAME receives its value from spec.nodeName in deployment.yaml.

Ask for help tracing the missing value: Why does my collector log not show the Kubernetes node identity?

You have proved that one replica keeps one collector running while leaving the second node unrepresented. Next, you will make per-node coverage part of the cluster's desired state.

Deploy a Collector on Every Node

Your one-replica Deployment proved the collector works. However, it left one of the two Kubernetes nodes uncovered.

A DaemonSet turns node coverage into desired state. This step removes the temporary controller before deploying one collector to every eligible node.

In this step, get ready to:
  • Remove the one-replica Deployment before introducing the DaemonSet.
  • Convert the existing collector manifest into a DaemonSet without a replica count.
  • Prove that two collectors run on different nodes through placement details and prefixed logs.
Clear the old desired state

Both controllers use the same app: node-identity-collector label. Removing the Deployment gives the DaemonSet sole responsibility for the matching Pods.

  • Switch back to the Killercoda browser terminal from earlier.
  • Delete the existing Deployment by running:
kubectl delete -f deployment.yaml

What Does This Command Do?

The command reads deployment.yaml to identify the existing Deployment. Kubernetes then removes that controller plus its managed collector Pod.

You should see confirmation that the Deployment was deleted. The single collector is now gone from the cluster.

Deployment Not Deleted?

  • Confirm that the browser terminal is still connected to the two-node playground.
  • Check that deployment.yaml remains in the current terminal directory.

Help me resolve a failed Deployment deletion.

Turn replica control into node control

The collector configuration already contains its labels, tolerations, container command, and Downward API field. The controller header is the only part that needs to change.

  • Switch back to the Theia browser IDE from earlier.
  • Select all the content in deployment.yaml.
  • Copy the selected manifest.
  • Create daemonset.yaml beside deployment.yaml using the file explorer in the left sidebar.

You should see daemonset.yaml in the file explorer. Its editor tab starts empty.

  • Paste the copied manifest into daemonset.yaml.
  • Find this header at the top of the copied manifest:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: node-identity-collector
  labels:
    app: node-identity-collector
spec:
  replicas: 1
  selector:

What Does This Header Control?

The current kind: Deployment selects the replica-based controller. The replicas: 1 field asks that controller to maintain one Pod.

  • Replace that header with this DaemonSet version:
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-identity-collector
  labels:
    app: node-identity-collector
spec:
  selector:

What Changed?

  • The kind: DaemonSet line makes per-node coverage the controller's desired state.
  • The missing replicas field lets the DaemonSet calculate the required Pod count from eligible nodes.
  • The existing selector still matches the app: node-identity-collector label in the Pod template.
  • Save daemonset.yaml.

✔️ I Have the Complete Manifest

Your saved manifest now defines the collector as a DaemonSet. Continue to deploy it.

ⓧ I Want to Check the Full File

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-identity-collector
  labels:
    app: node-identity-collector
spec:
  selector:
    matchLabels:
      app: node-identity-collector
  template:
    metadata:
      labels:
        app: node-identity-collector
    spec:
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
        - key: node-role.kubernetes.io/master
          operator: Exists
          effect: NoSchedule
      containers:
        - name: collector
          image: busybox:1.38.0
          command: ["sh", "-c"]
          args:
            - |
              echo "Node identity: ${NODE_NAME}"
              while true; do sleep 3600; done
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName

What Should Match?

The selector matches the Pod-template label exactly. The control-plane tolerations make both playground nodes eligible for a collector.

Each container receives its node name from spec.nodeName. Its first log line therefore identifies the node where that specific collector runs.

Manifest Does Not Match?

  • Confirm that kind: DaemonSet replaces kind: Deployment.
  • Confirm that the replicas: 1 line has been removed.
  • Check that YAML indentation matches the full-file reference.

Help me compare my DaemonSet manifest with the expected YAML.

Run the per-node collector

The manifest now describes node coverage instead of a fixed replica count. Applying it asks Kubernetes to place one collector on every eligible node.

  • Switch back to the Killercoda browser terminal.
  • Create the DaemonSet from the saved manifest by running:
kubectl apply -f daemonset.yaml

What Does This Command Do?

The command sends daemonset.yaml to the Kubernetes API. The DaemonSet controller then creates a collector Pod for each eligible node.

You should see confirmation that the DaemonSet was created. The cluster now starts scheduling its collectors.

DaemonSet Not Created?

  • Confirm that daemonset.yaml was saved before you applied it.
  • Compare the indentation around selector with the full-file reference.
  • Verify that the Deployment deletion completed before applying the DaemonSet.

Help me troubleshoot a DaemonSet manifest that does not apply.

Before you check placement, how many collector Pods do you expect to see? Your prediction is the benchmark for the next result.

  • Inspect the collector placement by running:
kubectl get pods -o wide

What Does the Wide View Show?

The wide Pod view includes the assigned node for each collector. This turns the DaemonSet's scheduling decision into visible evidence.

You should see two collector Pods with Running status. Each row should name a different playground node.

Only One Collector Running?

  • Wait a few seconds before checking the Pod list again.
  • Confirm that both cluster nodes were Ready when you began this step.
  • Check that both control-plane tolerations remain in daemonset.yaml.

Help me find why my DaemonSet covers only one node.

Before you read the logs, what node identities should the two collectors report? The prefixes make each result traceable to its source Pod.

  • Read the output from both collectors by running:
kubectl logs -l app=node-identity-collector --all-containers=true --prefix

What Do the Log Prefixes Prove?

  • The label selector gathers logs from every collector Pod.
  • The --all-containers=true flag includes the collector container from each matching Pod.
  • The --prefix flag identifies the source Pod before each log line.

You should see two prefixed Node identity: lines. Each line should report one of the two playground nodes.

That is the coverage gap closed. Your DaemonSet now maintains one node-aware collector on each eligible node.

Missing One Log Line?

  • Confirm that both collector Pods show Running status before reading their logs.
  • Check that both Pods carry the app: node-identity-collector label.
  • Verify that the container command still prints Node identity: ${NODE_NAME} before sleeping.

Help me troubleshoot missing DaemonSet collector logs.

Your collector now covers both nodes and proves its placement through node-specific logs. Next, you will delete one managed Pod and watch the DaemonSet restore that coverage.

Prove the DaemonSet Self-Heals

Your DaemonSet now gives both playground nodes a collector. That proves Kubernetes can place one managed Pod on every eligible node.

Per-node placement is useful when the cluster can restore it after a failure. In this step, you will delete one managed Pod to reveal the controller's reconciliation loop.

In this step, get ready to:
  • Capture a collector Pod's current placement.
  • Delete that Pod while watching the cluster update.
  • Confirm the DaemonSet restores one collector on each node.
Capture the current placement

Self-healing needs a baseline. The current Pod name plus its assigned node gives you a before state for the deletion test.

  • Display the current collector placement by running this command:
kubectl get pods -o wide

What does the wide view prove?

  • The wide output includes the node assigned to each Pod.
  • This baseline lets you compare the original Pod with its replacement.

You'll see two collector Pods in Running state on different nodes.

  • Record one collector Pod name here: <pod-name>.
  • Record the assigned node from the same row here: your collector node name.
Watch the reconciliation loop

A DaemonSet controller continuously compares its desired state with the cluster's current state. Deleting a managed Pod creates a gap for that loop to reconcile.

  • Start a live Pod view in the browser terminal by running this command:
kubectl get pods --watch

How does the watch work?

The --watch flag keeps the request open after the initial list. It streams each Pod change so you can follow the deletion and replacement.

  • Keep the watch running in the browser terminal.
  • Switch back to the Theia IDE from earlier.
  • Create a second terminal session with the terminal panel's new-session control.

Before you delete the Pod, what do you expect the live list to do after its row disappears? Hold that prediction as you run the deletion.

  • Delete the recorded Pod from the second terminal by running this command:
kubectl delete pod [[POD_NAME="<pod-name>"]]

What does the deletion affect?

This command removes only the selected Pod. The DaemonSet remains active with the same desired state.

  • Return your attention to the watch terminal.

You'll see the recorded Pod leave the list. A new collector row appears and reaches Running.

  • Stop the watch by pressing Ctrl+C after the replacement reaches Running.

Replacement not appearing?

  • Confirm the deleted Pod name matches the value you recorded.
  • Make sure the second terminal belongs to the same running playground session.
  • Confirm both playground nodes still report a ready state.

Help me troubleshoot why my DaemonSet is not replacing the deleted collector Pod.

Verify restored per-node coverage

The live stream proved the controller reacted. The final check tests whether per-node coverage is fully restored.

Before you run the final check, do you expect to see one collector or two? Keep your prediction in mind as you inspect their node placement.

  • Check the final collector placement by running this command:
kubectl get pods -o wide

What does this check prove?

The wide view confirms the number of running collectors. It also confirms whether those collectors cover different nodes.

You'll see two collector Pods in Running state on different nodes. The replacement runs on your collector node name.

That is the proof: the DaemonSet restored one collector on every eligible node without any change to daemonset.yaml.

Coverage not restored?

  • Give the replacement time to finish starting before repeating the final check.
  • Confirm the replacement uses the node recorded before the deletion.
  • Check that both collector rows reach the running state.

Help me restore one DaemonSet collector on each node.

Secret mission

Secret Mission: Target Only Labeled Nodes

Your collector currently runs on every eligible node. In this extension, you will label one node for SSD workloads. Then you will restrict the DaemonSet to that selected node.

Clean Up Your Resources

Clean Up Your Resources

Choose whether to keep your browser playground active, pause the collector, or remove the project resources. Your Killercoda FREE session has a maximum duration of 1 hour, so this Kubernetes project has no ongoing cost.

Resources you used:

  • One Killercoda Kubernetes 1.36 Two Node playground containing two cluster nodes.
  • One DaemonSet named node-identity-collector.
  • One node label set to disktype=ssd.
  • Three YAML files in Theia: deployment.yaml, daemonset.yaml, and daemonset-targeted.yaml.

Keep everything running

No action is needed. Choose this option if you want to keep testing node labels or collector placement during the current playground session.

The targeted collector remains on the node labeled disktype=ssd. The FREE session cannot run beyond its 1 hour limit.

Pause - I'll come back to this later

Remove the active collector while keeping the node label and YAML files available. You can resume within the same playground session by applying daemonset-targeted.yaml again.

  • Switch back to the browser terminal from earlier.
  • Pause the collector by deleting its current manifest resource with this command:
kubectl delete -f daemonset-targeted.yaml

What does this cleanup do?

The command finds the resource described by daemonset-targeted.yaml. It removes the node-identity-collector DaemonSet.

Kubernetes stops the managed collector Pod. Your node label and YAML files remain available.

  • Confirm that the collector Pod is gone by running:
kubectl get pods -o wide

What should you see?

The wide Pod listing should contain no running collector Pod. This confirms that the DaemonSet has been removed.

Your collector is now paused. The manifest still gives you a quick path back to selective node coverage during this session.

Collector still running?

Check that daemonset-targeted.yaml still appears in Theia. Confirm that your terminal remains connected to the same playground.

Ask for help with the cleanup: Help me diagnose why my DaemonSet cleanup did not complete.

Delete - I don't want to use this again

Deleting these resources is safe because the playground is temporary. This option removes the active workload and its three project files.

  • Switch back to the browser terminal from earlier.
  • Delete the active DaemonSet by running:
kubectl delete -f daemonset-targeted.yaml

What does this cleanup do?

The command removes the node-identity-collector DaemonSet. Kubernetes also removes its managed collector Pod.

  • Verify that no collector Pod remains by running:
kubectl get pods -o wide

Check the cluster cleanup

The output should contain no running collector Pod. This proves that the project workload has left the cluster.

The workload is gone. You can now remove the files that described it.

DaemonSet still present?

Confirm that the command uses daemonset-targeted.yaml. The active DaemonSet was last updated from that file.

Get targeted cleanup help: Help me remove the remaining collector workload.

  • Remove all three YAML files from the current terminal directory by running:
rm deployment.yaml daemonset.yaml daemonset-targeted.yaml

What does this file cleanup do?

The rm command deletes the three project manifests from the temporary browser environment. It leaves unrelated files untouched.

  • Confirm that the YAML files are gone by running:
ls

Check the file cleanup

The directory listing should no longer include deployment.yaml, daemonset.yaml, or daemonset-targeted.yaml.

That clears the project files from Theia. Your remaining cluster state is limited to the temporary playground session.

Files still listed?

Check the spelling of each filename against the names shown in Theia. Confirm that your terminal is in the directory containing the three manifests.

Get help locating the files: Help me finish the YAML file cleanup.

  • Close the Killercoda browser tab after confirming the cleanup.
  • Allow the FREE session to reach its 1 hour maximum if it has not ended already.

The remaining disktype=ssd label exists only inside the temporary playground. The session creates no ongoing charge.

Nice Work!

Nice Work!

Excellent work! Your Kubernetes node identity collector now reports the node where it runs. You proved that a DaemonSet restores node-level coverage after a managed Pod disappears.

You've learned how to:

  • Build a BusyBox 1.38.0 collector that prints its node identity from the Kubernetes Downward API.
  • Expose the coverage gap in a one-replica Deployment. Replace it with a DaemonSet that maintains one collector Pod on each eligible node.
  • Prove self-healing by deleting a managed Pod. Watch the DaemonSet restore per-node coverage automatically.
  • Secret Mission: Apply the disktype=ssd node label to one node. Use nodeSelector to keep the collector on that selected node.

Ready to quiz yourself?