Build a Kubernetes Job Platform

Build a resilient batch dashboard on a three-node K3s cluster.

Introduction

30 Second Summary

Infrastructure demos often disappear behind terminal output. A stronger demo makes progress visible while a machine failure tests the system in front of your audience.

In this project, you will build BatchForge as a responsive dashboard on three virtual machines running Kubernetes. The browser tracks each batch task from submission to result during live failure demos.

What You'll Build

You will watch a submitted task reach Succeeded on a live dashboard that stays connected through a VM failure.

By the end of this project, you'll have:

An interactive dashboard that shows the cluster connection state plus live Job metrics. Its activity feed updates without a page refresh.
A batch workflow that turns submitted text into Kubernetes Jobs. You can follow each task from Pending to Succeeded. The completed card shows the worker node plus attempt count plus processing duration. Result cards display word counts plus a SHA-256 digest.
A resilient K3s platform spread across three independent virtual machines. A NodePort gives every VM the same browser entry point. After you power off one VM, the dashboard stays connected.
Secret Mission: Add a PodDisruptionBudget that preserves one healthy BatchForge replica during planned node maintenance. Prove the policy by draining a node while the dashboard stays connected.

Are there any prerequisites?

The lab needs an Ubuntu or Ubuntu-based PC with at least 16 GB RAM plus about 40 GB of free disk space plus enabled CPU virtualization.

Internet access plus sudo privileges plus Git plus VS Code plus Docker experience complete the prerequisites.

Before We Start

This is your moment to commit to BatchForge as a visual platform-engineering portfolio project. Choose the reliability behavior you most want to demonstrate before the hands-on work begins.

Prepare the Virtualization Host

BatchForge depends on hardware-backed virtual machines for its three-node cluster. The host also needs a working container runtime so you can build the image those nodes use.

This step proves that KVM, libvirt, and Docker work before cluster automation begins. You will also prepare the project repository and the SSH identity used to reach each VM.

In this step, get ready to:

Install the virtualization tools and verify KVM acceleration.
Install Docker Engine and prove that your account can run containers.
Prepare the BatchForge repository and SSH identity.

Install and verify virtualization tooling

KVM provides hardware acceleration for the Ubuntu guests. Libvirt manages those guests through a persistent virtual network.

Refresh Ubuntu's package information and install the virtualization bundle by running these commands:

sudo apt update
sudo apt install -y cpu-checker virt-manager

What do these commands install?

The package update refreshes the software information used by Ubuntu.
The cpu-checker package provides the KVM verification command.
The virt-manager bundle installs the QEMU and libvirt components used by the VM automation.

The package download can take a few minutes on a fresh host. Ubuntu prints progress throughout the installation.

Check whether the host can use hardware acceleration by running:

kvm-ok

What does this check prove?

The command checks whether the processor and Linux kernel can support accelerated virtual machines. BatchForge needs that acceleration to run three guests without relying on slow software emulation.

You should see KVM acceleration can be used. That result confirms the virtualization layer is ready.

KVM acceleration unavailable?

Restart the host and confirm CPU virtualization is enabled in its firmware settings. The setting may mention hardware virtualization without using the KVM name.

Run the check again after Ubuntu starts. Help me diagnose why kvm-ok cannot use hardware acceleration on my Ubuntu host.

Libvirt's default NAT network gives every VM a private address while preserving internet access. Autostart keeps that network available after the host reboots.

Start the default libvirt network and configure it to start with the host by running:

sudo virsh net-start default || true
sudo virsh net-autostart default

What do these network commands do?

The first command starts the default NAT network when it is currently inactive.
The || true suffix lets the setup continue when the network is already active.
The second command configures the network to return automatically after a host restart.

Confirm that libvirt can display its domain table by running:

sudo virsh list --all

What should the domain table show?

The table can be empty because the BatchForge VMs do not exist yet. A visible header confirms that virsh can communicate with libvirt.

Install and verify Docker Engine

Docker builds the portable image containing the frontend, API, and worker. Your host may already have Docker Engine, so the version check determines which setup path you need.

Check whether the Docker command is available by running:

docker --version

What does this check tell you?

A version response confirms that the Docker command is installed. A command-not-found response means this host still needs Docker Engine.

✔️ Docker reports a version

Docker Engine is already installed. Continue to the shared container test below.

ⓧ Command not found

Docker provides a convenience installer for development environments. You will download it first so you can inspect the actions it plans to perform.

Download the installer and display its planned changes by running:

curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh ./get-docker.sh --dry-run

What do these commands do?

The first command downloads the Docker development installer as get-docker.sh.
The dry run displays the installation actions without changing the host.

Install Docker Engine with the inspected script by running:

sudo sh get-docker.sh

What does the installer change?

The installer configures Docker's software source and installs Docker Engine for this development lab. Production hosts need a managed installation and upgrade process.

Docker installation not completing?

Confirm that the host has internet access and that the earlier package update completed. A network interruption can stop the installer while it downloads packages.

Review the final terminal lines before retrying. Help me troubleshoot the Docker convenience installer on Ubuntu.

The Docker group permits commands without repeating sudo. Membership grants root-level privileges, so reserve it for this trusted personal lab host.

Add your current account to the Docker group by running:

sudo usermod -aG docker $USER

What does this permission change do?

The command adds your current account to the Docker group. Existing terminal sessions keep their previous group membership until you refresh the session.

Activate the new group membership in the current terminal by running:

newgrp docker

What does this command do?

The command starts a shell with the updated Docker group membership. This lets the next container test run without sudo.

Before you run the container, predict whether Docker can download an image and start it through your current account.

Verify the complete Docker setup by running:

docker run hello-world

What does this test prove?

Docker downloads a small test image and starts a container from it. Its success message proves that the daemon, network, image store, and account permissions work together.

You should see Docker's success message in the terminal. That is the container build path for BatchForge working end to end.

Container test failing?

If the output mentions permission access, confirm that you ran newgrp docker after changing group membership. You can also sign out of Ubuntu and sign back in before retrying.

If the image download fails, confirm that the host can reach the internet. Help me troubleshoot docker run hello-world on Ubuntu.

Prepare the repository and SSH identity

The batchforge-lab repository keeps the dashboard, Kubernetes resources, and VM automation together. A dedicated SSH key lets cloud-init authorize your account inside every VM.

Move to your Desktop by running this command:

cd ~/Desktop

Why use the Desktop?

A fixed starting location makes the repository easy to find from the terminal and VS Code. Every later script assumes you are working inside the same batchforge-lab folder.

Create the repository folder and initialize Git by running these commands:

mkdir batchforge-lab
cd batchforge-lab
git init

What do these commands create?

The first command creates batchforge-lab on your Desktop.
The second command moves the terminal into that folder.
The final command creates the hidden Git metadata used to track the project.

The terminal confirms that it initialized an empty Git repository. The folder is now ready for the project source.

Press Ctrl+Shift+P in VS Code to open the Command Palette.
Select File: Open Folder.
Select the batchforge-lab folder on your Desktop.
Create the frontend, k8s, and scripts directories in the Explorer sidebar.
Create Dockerfile, app.py, and worker.py inside batchforge-lab.
Create index.html, styles.css, and app.js inside frontend.
Create app.yaml inside k8s.
Create provision-vms.sh, bootstrap-k3s.sh, distribute-image.sh, and destroy-lab.sh inside scripts.

How the repository is organized

The top-level Python files implement the API and batch worker.
The frontend directory contains the browser experience.
The k8s directory stores the desired cluster resources.
The scripts directory contains VM, cluster, image, and cleanup automation.

Copy the corresponding Complete Code into every file you created.
Save all open files in VS Code.
Confirm that every listed file appears in the Explorer sidebar under batchforge-lab.

✔️ Awesome, I've got everything!

Great. Double-check that every source file is saved inside batchforge-lab.

ⓧ I'd like to double check the full code

Compare each file with the Complete Code supplied for BatchForge. The filenames and directory locations must match exactly.

Confirm that Dockerfile uses the Python 3.14.7 slim image and starts /app/app.py.
Confirm that app.py contains the Kubernetes service-account paths, Job API helpers, request handler, and threaded HTTP server.
Confirm that worker.py reads JOB_TEXT and JOB_DELAY before printing the JSON result.
Confirm that frontend/index.html contains the branded dashboard, metrics, submission form, architecture panel, activity feed, and script references.
Confirm that frontend/styles.css contains the dark responsive layout, status styles, result cards, focus states, animations, and mobile breakpoints.
Confirm that frontend/app.js contains the form handling, Job rendering, metric updates, connection state, and two-second polling.
Confirm that k8s/app.yaml defines the namespace, ServiceAccount, namespace-scoped RBAC, NodePort Service on port 30080, and two-replica Deployment.
Confirm that the four files in scripts match the VM provisioning, K3s bootstrap, image distribution, and cleanup automation from the Complete Code.

The VM provisioning script expects the public half of an Ed25519 key at ~/.ssh/id_ed25519.pub. Keep an existing matching key pair when those files are already present.

Use Ubuntu's file manager to check whether ~/.ssh/id_ed25519 and ~/.ssh/id_ed25519.pub already exist.
Keep the existing pair when both files are present.
Generate the pair when the public key is missing by running:

ssh-keygen -t ed25519

What does this command create?

The command creates a private key at ~/.ssh/id_ed25519 and a public key at ~/.ssh/id_ed25519.pub. Cloud-init copies the public key into the VM account while the private key stays on your host.

Press Enter to accept the default key path when prompted.
Complete the remaining key-generation prompts.

Existing key warning?

Do not overwrite an existing private key. Cancel the prompt and confirm that the matching id_ed25519.pub file is present beside it.

An overwritten key can break access to systems that trust the original public key. Help me safely inspect my existing Ed25519 SSH key pair.

Return to the terminal inside batchforge-lab.
Make every automation script executable by running:

chmod +x scripts/*.sh

Why make the scripts executable?

Executable permissions let you launch each automation file directly from the repository. The next step starts with ./scripts/provision-vms.sh.

Before the final check, predict whether all three host layers can now answer successfully from the same terminal.

Run the end-of-step checks with these commands:

kvm-ok
sudo virsh list --all
docker run hello-world

What should you see?

The KVM check reports KVM acceleration can be used.
The libvirt command displays a domain table, even when that table contains no VMs.
The Docker container prints its success message.

You have now proved the complete host path from hardware virtualization to container execution. The repository and SSH identity are ready for VM provisioning.

One final check still failing?

Treat each result as a separate layer. Revisit the nearby troubleshooting guidance for KVM, libvirt, or Docker before continuing.

Keep the exact failing output available. Help me diagnose which BatchForge host prerequisite is failing.

Your virtualization host is ready. Next, you will turn it into three independent Ubuntu servers and form the K3s control plane.

Provision the K3s Cluster

Your Ubuntu host now has working KVM acceleration. Its libvirt network is ready for the lab.

A single Kubernetes virtual machine cannot demonstrate control-plane quorum. Three independent servers let you prove VM-level fault isolation before BatchForge reaches the cluster.

In this step, get ready to:

Provision three Ubuntu server VMs with recorded DHCP addresses.
Form an embedded-etcd K3s control plane across the servers.
Verify every node reports Ready with control-plane and etcd roles.

Create three virtual machines

Each VM uses a QCOW2 overlay that preserves the shared Ubuntu image. Cloud-init adds the devops user with your existing public key.

The existing provisioning script expects your terminal to be at the top level of the batchforge-lab repository.

The first run can take several minutes while the matching Ubuntu 24.04 cloud image downloads.

Return to the terminal for your existing batchforge-lab repository.
Create the three VMs by running the existing provisioning automation:

./scripts/provision-vms.sh

What does this script create?

The script detects whether your host uses AMD64 or ARM64.
It creates batchforge-1, batchforge-2, and batchforge-3 with 2 vCPUs per VM.
Each VM receives 2 GB of RAM plus a 12 GB overlay disk.
The script writes the discovered addresses to .lab/nodes.env as NODE1_IP, NODE2_IP, and NODE3_IP.

The script ends by printing the three recorded addresses. Those values connect the remaining automation to the correct VMs.

Review the visible VM boundary by running this command:

sudo virsh list --all

What should you see?

You should see batchforge-1, batchforge-2, and batchforge-3 in the libvirt domain table. Each VM should report a running state.

Missing a VM from the table?

Confirm scripts/provision-vms.sh remains executable.
Confirm ~/.ssh/id_ed25519.pub still exists.
Run the provisioning automation again after correcting the failed prerequisite.

Need help tracing the failed VM? Help me troubleshoot the BatchForge VM provisioning script.

Bootstrap embedded-etcd high availability

An embedded-etcd control plane stores cluster state across the server nodes. Three servers preserve quorum after one server loss.

Why K3s for this lab?

K3s packages the Kubernetes control plane with containerd. It also provides the command-line utilities used by the lab.

This keeps the project focused on high-availability behavior. A larger bootstrap process would add setup that does not strengthen the demonstration.

The bootstrap contacts all three VMs, so the installation can take several minutes while each server downloads K3s.

Keep the server token private

The server token can authorize new cluster members. The script stores it in .lab/server-token with mode 600.

Review each VM's SSH host fingerprint when prompted.
Build the control plane by running the existing bootstrap automation:

./scripts/bootstrap-k3s.sh

What does this script do?

The script reads the VM addresses from .lab/nodes.env.
It creates the shared token in .lab/server-token when the file does not exist.
It pins every server to K3s v1.36.4+k3s1.
It initializes batchforge-1 with --cluster-init.
The remaining servers join through https://$NODE1_IP:6443.

Use the final node table to choose the outcome that matches your installation.

✔️ I see v1.36.4+k3s1 on three nodes

All three rows report Ready with K3s v1.36.4+k3s1. The bootstrap reached every VM through your existing SSH key.

ⓧ I see an older version

The existing automation pins the same K3s release on every server. Reapplying it updates the installation to the project version.

Reapply the pinned K3s installation by running the bootstrap automation again:

./scripts/bootstrap-k3s.sh

Why rerun this script?

The script supplies v1.36.4+k3s1 through the documented K3s version setting. Its final node table lets you confirm the result.

The final table should show the pinned version on all three nodes.

Still seeing an older version?

Confirm the terminal is inside the existing batchforge-lab repository. Help me diagnose the K3s version mismatch.

ⓧ K3s command not found

A missing K3s command means the installation did not complete on the server reached by the final check.

Retry the installation on all three servers by running the bootstrap automation:

./scripts/bootstrap-k3s.sh

What happens on this retry?

The script waits until each VM can use its required download tool. It then installs the pinned K3s release across the cluster.

The final node table confirms whether every server completed its installation.

K3s still unavailable?

Allow the VM's cloud-init setup to finish before retrying.
Check that the affected VM remains running in libvirt.

Need help isolating the affected server? Help me troubleshoot the missing K3s installation.

Inspect the control plane

The final check queries the cluster through batchforge-1. Wide output exposes the node names plus their internal addresses.

Why use three servers?

Embedded etcd uses quorum to accept control-plane changes. Three is the smallest odd server count that keeps quorum after one server fails.

Each server also creates a separate VM failure domain. That separation makes later resilience tests visible.

Before you run this check, how many Ready nodes do you expect to see?

Inspect the cluster from batchforge-1 by running these commands:

source .lab/nodes.env
ssh devops@$NODE1_IP 'sudo k3s kubectl get nodes -o wide'

What do these commands check?

The first command loads NODE1_IP from .lab/nodes.env.
The second command connects to batchforge-1 as the devops user.
The bundled kubectl queries the cluster with -o wide output.

You'll see batchforge-1, batchforge-2, and batchforge-3 in the Ready state. Each row shows control-plane and etcd roles.

That is the cluster boundary proven: the embedded-etcd control plane is healthy across three independent VMs. The wide output also shows a different internal address for each node.

Is a node missing or not Ready?

Confirm all three VMs remain running in the earlier libvirt table.
Confirm the addresses in .lab/nodes.env match the current VM leases.
Retry the bootstrap automation after the affected VM finishes starting.

Need help reading the node table? Help me diagnose the unhealthy K3s node.

Your three-server K3s foundation is ready. Next, you will turn BatchForge into a visual dashboard backed by its Job API.

Build the Dashboard and Job API

Your three-server K3s control plane is healthy. A backend that only appears in terminal output is difficult to demonstrate to an interviewer or teammate.

The existing responsive frontend gives the platform a visual story. Its Python worker supports that story through a Kubernetes-aware REST API.

In this step, you package those existing sources into a Docker image. You also prove the local limitation before distributing the image to each server.

In this step, get ready to:

Build the existing BatchForge sources into a local image.
Test the dashboard presentation and task submission from a local container.
Distribute the tested image into every K3s node image store.

Build the presentation image

The Dockerfile from earlier defines one runtime for the frontend. It also includes the API plus worker.

Building that definition produces localhost/batchforge:1.0.0 for the local showcase test.

Build the BatchForge image from the existing batchforge-lab terminal by running this command:

docker build -t localhost/batchforge:1.0.0 .

What does this command do?

The build reads the existing Dockerfile from the current repository.
The python:3.14.7-slim base provides the application runtime.
The copy instructions place app.py plus worker.py inside the image.
The frontend copy includes index.html plus its stylesheet and browser application.
The tag gives the completed image the name localhost/batchforge:1.0.0.

You now have one portable BatchForge image for the local demo.

Image build not completing?

Confirm the terminal is inside your existing batchforge-lab repository.
Confirm Dockerfile appears beside app.py in the VS Code Explorer sidebar.
Confirm the Docker service is available on your Ubuntu host.

If the build still stops, help me diagnose this BatchForge image build.

Run the local showcase

The local container proves that the dashboard works as a complete presentation layer. The process remains in the foreground until you stop it.

Start the local BatchForge container by running this command:

docker run --rm -p 127.0.0.1:8080:8080 localhost/batchforge:1.0.0

What does this command do?

The --rm flag removes the temporary container after it stops.
The port mapping makes the application available only through the Ubuntu host's loopback address.
The container starts app.py on port 8080.

The terminal displays BatchForge listening on 0.0.0.0:8080 while the container is running.

Open your web browser from Ubuntu's applications menu.
Enter http://127.0.0.1:8080 in the browser address bar.
Press Enter to load BatchForge.

You'll see the responsive hero beside the live connection badge. You'll also see the metric cards plus submission workspace.

The architecture panel explains the virtualized platform. The activity feed is ready for browser polling without a page refresh.

Dashboard not loading?

Return to the terminal that started the container.
Confirm the BatchForge process is still running.
Check that the browser address is exactly http://127.0.0.1:8080.

If the page remains unavailable, help me troubleshoot this local BatchForge container.

Before you submit the default text, do you think this local container can create a Kubernetes Job?

Keep the default text in the Text payload field.
Keep the processing time at 10 seconds.
Click Submit Kubernetes Job.

You'll see Kubernetes service account is unavailable. Deploy BatchForge to the cluster. under the form. The connection badge reads Cluster unavailable.

Why does the Job request fail?

A Kubernetes ServiceAccount projects authentication files into a Pod. The local container has no mount at /var/run/secrets/kubernetes.io/serviceaccount.

The cluster_namespace() function returns no namespace. The kube_request() function surfaces the visible error before a Job request leaves the application.

Switch back to the terminal running the local container.
Press Ctrl+C to stop it.

The terminal returns to its prompt. The temporary local container is now removed.

Distribute the image to every server

The tested image must exist in each node's containerd image store. The prepared manifest uses imagePullPolicy: Never when BatchForge runs in the cluster.

The existing distribution script creates one archive. It places that archive where each K3s server can import it.

Before you run the distribution, how many server image listings do you expect to see?

Create the image archive and distribute it by running this command:

./scripts/distribute-image.sh

What does this script do?

The script saves localhost/batchforge:1.0.0 as .lab/batchforge.tar.
It copies the archive to each server over your existing SSH connection.
It places the archive in the K3s image import directory on every node.
It checks each node's image store for localhost/batchforge:1.0.0.

You'll see a matching image line from batchforge-1. You'll see the same confirmation from batchforge-2.

You'll also see the matching image on batchforge-3. Your application image is now ready on every server.

Missing an image confirmation?

Check that all three BatchForge VMs are still running in libvirt.
Confirm your existing SSH key still connects as devops to the affected server.
Confirm .lab/nodes.env still contains the three discovered node addresses.

If one node still misses the image, help me diagnose the BatchForge image distribution.

BatchForge now has a tested dashboard image in every server's image store. Next, you'll deploy it with in-cluster credentials and process a real Job.

Deploy the Single Replica

Your three K3s servers now hold the same BatchForge image. The dashboard is ready to move from a local demonstration into the cluster.

A Kubernetes Pod receives mounted ServiceAccount credentials. Namespace-scoped RBAC gives those credentials permission to create Jobs. This step starts with one API replica so you can test whether one failure domain provides enough availability.

In this step, get ready to:

Deploy a single-replica Deployment with constrained cluster permissions.
Submit a real Kubernetes Job from the dashboard.
Expose the availability gap by powering off the replica's VM.

Deploy the first in-cluster version

The prepared k8s/app.yaml describes the final resilient state with two replicas. A temporary copy lets you test a single replica while preserving that desired state for the next step.

Create .lab/app-single.yaml with one replica by running these commands from the batchforge-lab folder:

source .lab/nodes.env
sed 's/replicas: 2/replicas: 1/' k8s/app.yaml > .lab/app-single.yaml

What do these commands do?

The first command loads NODE1_IP, NODE2_IP, and NODE3_IP from .lab/nodes.env.
The second command copies the prepared manifest into .lab/app-single.yaml.
The replacement changes only replicas: 2 to replicas: 1.

Confirm .lab/app-single.yaml appears inside the .lab folder in the VS Code Explorer sidebar.

✔️ Awesome, I've got everything!

Great. Your temporary manifest preserves the prepared resources while setting the API replica count to one.

ⓧ I'd like to double check the full code

apiVersion: v1
kind: Namespace
metadata:
  name: batchforge
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: batchforge-api
  namespace: batchforge
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: batchforge-api
  namespace: batchforge
rules:
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["create", "get", "list"]
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: batchforge-api
  namespace: batchforge
subjects:
  - kind: ServiceAccount
    name: batchforge-api
    namespace: batchforge
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: batchforge-api
---
apiVersion: v1
kind: Service
metadata:
  name: batchforge-api
  namespace: batchforge
spec:
  type: NodePort
  selector:
    app: batchforge-api
  ports:
    - name: http
      port: 80
      targetPort: http
      nodePort: 30080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: batchforge-api
  namespace: batchforge
spec:
  replicas: 1
  selector:
    matchLabels:
      app: batchforge-api
  template:
    metadata:
      labels:
        app: batchforge-api
    spec:
      serviceAccountName: batchforge-api
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: batchforge-api
      containers:
        - name: api
          image: localhost/batchforge:1.0.0
          imagePullPolicy: Never
          env:
            - name: WORKER_IMAGE
              value: localhost/batchforge:1.0.0
          ports:
            - name: http
              containerPort: 8080
          readinessProbe:
            httpGet:
              path: /healthz
              port: http
            initialDelaySeconds: 2
            periodSeconds: 3
          livenessProbe:
            httpGet:
              path: /healthz
              port: http
            initialDelaySeconds: 5
            periodSeconds: 10

What should differ?

The temporary manifest matches k8s/app.yaml except for replicas: 1. The original file keeps replicas: 2 for the resilient deployment.

The manifest now needs to reach a server that can apply it through the cluster's bundled command-line tools.

Copy .lab/app-single.yaml to /tmp/app-single.yaml on batchforge-1 by running this command:

scp .lab/app-single.yaml devops@$NODE1_IP:/tmp/app-single.yaml

What does this command do?

The secure copy sends the temporary manifest through your existing SSH access. The destination path keeps the deployment file separate from the server's operating system configuration.

The transfer progress reaches 100 percent when /tmp/app-single.yaml is available on batchforge-1.

Manifest transfer not completing?

Confirm that NODE1_IP was loaded from .lab/nodes.env. Check that batchforge-1 is still running.

Use the same SSH identity that already connects as devops. Help me troubleshoot this manifest transfer.

The manifest creates a NodePort Service on every VM. Its selector sends traffic only to ready API Pods.

Connect to batchforge-1 as devops with the SSH access from earlier.
Apply the temporary manifest by running these commands:

sudo k3s kubectl apply -f /tmp/app-single.yaml
sudo k3s kubectl rollout status deployment/batchforge-api -n batchforge --timeout=120s

What happens during deployment?

The first command creates the namespace resources from /tmp/app-single.yaml.
The second command waits for the batchforge-api rollout to complete.
The readiness probe keeps the Pod out of Service traffic until /healthz responds successfully.

The terminal confirms that the rollout completed. BatchForge now runs with cluster-mounted credentials.

Confirm that one API Pod is ready by running:

sudo k3s kubectl get pods -n batchforge -l app=batchforge-api -o wide

What should you see?

The wide output shows one batchforge-api Pod. Its ready count confirms that the Service can send traffic to it.

That is the first in-cluster version running. The single Pod is now the only endpoint behind the Service.

Rollout not becoming ready?

Check that the manifest still uses localhost/batchforge:1.0.0. Confirm that the image was imported on the node selected for the Pod.

Make sure the container still exposes the named http port used by both probes. Help me diagnose this BatchForge rollout.

Process a real Kubernetes Job

The API can now authenticate through its mounted credentials. Its REST API creates a worker Job without exposing cluster-admin permissions.

Enter http://$NODE1_IP:30080 in your browser address bar.
Confirm that the connection badge reads Cluster connected.

Before you submit the workload, do you expect the same service-account error from the local test?

Set the Simulated processing time slider to 10 seconds.
Click Submit Kubernetes Job.

The card moves through PENDING, RUNNING, and SUCCEEDED. The live metrics update without a page refresh.

Once the Job finishes, its card shows the worker node. It also shows the attempt count plus the processing delay.

The result cards display the word count. They also display the unique-word count plus the SHA-256 digest.

That local limitation is gone: BatchForge can now create real cluster workloads from the browser.

Before checking the permission directly, predict whether the dedicated account can create Jobs in batchforge.

Return to the SSH session on batchforge-1.
Check the API account's Job permission by running:

sudo k3s kubectl auth can-i create jobs --namespace batchforge --as system:serviceaccount:batchforge:batchforge-api

What does this permission check prove?

The command evaluates the exact identity used by the API Pod. A yes response proves the namespace Role permits Job creation.

The permission is limited to the batchforge namespace. That boundary keeps the application away from cluster-wide administration.

You see yes in the terminal. The dashboard's completed Job provides visual proof that the permission works in practice.

Trigger the designed outage

The dashboard currently depends on one ready API Pod. A libvirt VM boundary lets you test that dependency as a real host failure.

Identify the API Pod's hosting node by running this command on batchforge-1 or another ready server:

sudo k3s kubectl get pods -n batchforge -l app=batchforge-api -o wide

Which values matter?

The Pod name identifies the stale workload object if its node disappears. The NODE column identifies the VM that currently holds the only API replica.

Record the Pod name here: name of the single API Pod.
Record the hosting VM here: VM hosting the API Pod.
Record a different VM's IP from .lab/nodes.env here: IP address of a node that is not hosting the API Pod.

A health loop makes the Service response visible once per second. Keeping it on a different node separates the Service test from the VM you are about to stop.

Start the health loop in a second terminal on the Ubuntu host by running:

while true; do
  curl -sS http://[[LIVE_NODE_IP="IP address of a node that is not hosting the API Pod"]]:30080/healthz
  sleep 1
done

What does the loop show?

Each request enters through a VM that does not host the API Pod. The Service still forwards successful requests to the one ready endpoint.

You see {"status": "ok"} repeat while the endpoint remains available.

Before you power off the replica's VM, do you think the Service has another ready endpoint to use?

Switch to a terminal on the Ubuntu host.
Power off the API Pod's VM immediately by running:

sudo virsh destroy [[API_POD_NODE="VM hosting the API Pod"]]

The outage is intentional

The command immediately terminates the VM that held the only API Pod. The Service now has no ready endpoint.

The health loop begins reporting failed requests. The dashboard changes its connection state because browser polling can no longer reach the API.

Return to the open BatchForge browser tab.

The connection badge degrades from its connected state. This visible interruption proves that one replica shares one failure domain.

Force deletion sounds severe. The original VM is powered off, so the stale API process cannot keep serving traffic.

Switch to an SSH session on either surviving K3s server.
Force-delete the stale API Pod by running:

sudo k3s kubectl delete pod [[API_POD_NAME="name of the single API Pod"]] -n batchforge --grace-period=0 --force

Why force-delete this Pod?

The unreachable node cannot confirm normal Pod termination. Force deletion removes the stale object from the API immediately.

The Deployment controller then creates a replacement for the missing replica. The powered-off VM prevents a duplicate API process from remaining active.

Wait for the replacement replica by running these commands:

sudo k3s kubectl rollout status deployment/batchforge-api -n batchforge --timeout=120s
sudo k3s kubectl get pods -n batchforge -l app=batchforge-api -o wide

What should recovery show?

The rollout waits for the replacement Pod to pass its readiness probe. The wide output shows the new Pod on a surviving node.

The dashboard reconnects once the replacement becomes ready. The controller has restored the declared count of one.

Switch back to the Ubuntu host terminal.
Restart the failed VM by running:

sudo virsh start [[API_POD_NODE="VM hosting the API Pod"]]

What does this restore?

The command boots the original K3s server again. Its local image plus cluster data remain on the persistent virtual disk.

The server needs a short period to boot before it can report Ready again.

Before the final cluster check, do you expect the restarted VM to rejoin the other two servers?

Return to an SSH session on a surviving server.
Check the final node state by running:

sudo k3s kubectl get nodes -o wide

What should the cluster show?

The output shows batchforge-1, batchforge-2, and batchforge-3 as Ready.

The replacement API Pod remains ready under the one-replica Deployment. All three application images remain available on their nodes.

Stop the health loop with Ctrl+C.
Refresh the BatchForge browser tab.

The dashboard reports Cluster connected again. Your completed Job cards remain visible after the infrastructure recovery.

Cluster or dashboard still recovering?

Confirm that the restarted VM appears in the node list. Wait for its status to become Ready before judging the final cluster state.

Check that one batchforge-api Pod is ready on a surviving server. Help me diagnose this single-replica recovery.

You have now demonstrated the single-replica failure mode from the browser. Next, you will spread two API replicas across separate VMs so one host failure no longer interrupts the dashboard.

Add Resilient API Replicas

Your single-replica test exposed the failure domain behind the dashboard outage. One lost VM removed the only ready API endpoint.

Two API replicas remove that shared failure domain from the Kubernetes Deployment.

Readiness probes keep the NodePort Service on healthy Pods. Topology spread separates those Pods by VM hostname.

In this step, get ready to:

Deploy two hostname-spread API replicas.
Prove the dashboard survives one replica-hosting VM loss.
Watch the Job controller replace a deleted worker Pod.

Apply the resilient desired state

The existing k8s/app.yaml defines two API replicas. It also includes health probes plus hostname spreading.

The ScheduleAnyway policy prefers separate hostnames. It still allows the Deployment to recover when fewer nodes are available.

Replace the temporary cluster state with k8s/app.yaml by running these commands from the Ubuntu host terminal in your batchforge-lab folder:

scp k8s/app.yaml "devops@$NODE1_IP:/tmp/app.yaml"
ssh "devops@$NODE1_IP" 'sudo k3s kubectl apply -f /tmp/app.yaml'
ssh "devops@$NODE1_IP" 'sudo k3s kubectl rollout status deployment/batchforge-api -n batchforge --timeout=120s'

What do these commands do?

The scp command copies the desired manifest to /tmp/app.yaml on the first server.
The first ssh command asks K3s to reconcile the resources in that manifest.
The final command waits for the two-replica rollout to finish within 120s.

Good progress. The rollout confirmation means the temporary one-replica state has been superseded by the resilient desired state.

Rollout not completing?

Confirm that all three VMs are running. Check that localhost/batchforge:1.0.0 remains available on every node.

A pending replica can also indicate that a node has not returned to Ready. Help me diagnose this BatchForge rollout.

Before you inspect the Pods, predict whether the two replicas will occupy the same VM hostname.

Inspect the API replica placement by running this command from the Ubuntu host terminal:

ssh "devops@$NODE1_IP" 'sudo k3s kubectl get pods -n batchforge -l app=batchforge-api -o wide'

What does this check show?

The label selector limits the output to the API Pods.
The -o wide option includes each Pod's assigned node.
The readiness column shows whether each replica can receive Service traffic.

You should see two ready API Pods. Their node values should name two different BatchForge VMs.

Replicas sharing one hostname?

Confirm the cluster is using k8s/app.yaml instead of .lab/app-single.yaml. Check that all three nodes report Ready.

Inspect the applied Deployment if the placement still looks wrong. Help me check the topology spread configuration.

✔️ Awesome, I've got everything!

Great. Your cluster now uses the complete two-replica manifest from k8s/app.yaml.

ⓧ I'd like to double check the full code

apiVersion: v1
kind: Namespace
metadata:
  name: batchforge
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: batchforge-api
  namespace: batchforge
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: batchforge-api
  namespace: batchforge
rules:
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["create", "get", "list"]
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: batchforge-api
  namespace: batchforge
subjects:
  - kind: ServiceAccount
    name: batchforge-api
    namespace: batchforge
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: batchforge-api
---
apiVersion: v1
kind: Service
metadata:
  name: batchforge-api
  namespace: batchforge
spec:
  type: NodePort
  selector:
    app: batchforge-api
  ports:
    - name: http
      port: 80
      targetPort: http
      nodePort: 30080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: batchforge-api
  namespace: batchforge
spec:
  replicas: 2
  selector:
    matchLabels:
      app: batchforge-api
  template:
    metadata:
      labels:
        app: batchforge-api
    spec:
      serviceAccountName: batchforge-api
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: batchforge-api
      containers:
        - name: api
          image: localhost/batchforge:1.0.0
          imagePullPolicy: Never
          env:
            - name: WORKER_IMAGE
              value: localhost/batchforge:1.0.0
          ports:
            - name: http
              containerPort: 8080
          readinessProbe:
            httpGet:
              path: /healthz
              port: http
            initialDelaySeconds: 2
            periodSeconds: 3
          livenessProbe:
            httpGet:
              path: /healthz
              port: http
            initialDelaySeconds: 5
            periodSeconds: 10

What should match?

The Deployment requests two replicas with matching app=batchforge-api labels.
The topology constraint spreads those replicas across kubernetes.io/hostname values.
The readiness probe checks /healthz every three seconds after its initial delay.
The liveness probe checks the same endpoint every ten seconds after its initial delay.

Repeat the VM failure

The earlier outage left the Service without a ready endpoint. Two spread replicas let one VM disappear while the surviving endpoint keeps handling browser polling.

A libvirt VM remains the failure boundary for this test. The browser plus a separate health loop make continuity visible from two angles.

Identify one replica-hosting VM from the node column in your previous output.
Record that VM name here: <one-api-replica-node>.
Choose the address of a different Ready VM from .lab/nodes.env.
Record that surviving address here: <third-live-node-ip>.
Switch back to the BatchForge dashboard from earlier.
Use a separate Ubuntu host terminal for the health loop.
Start polling the surviving VM once per second by running this command:

watch -n 1 "curl -sS -o /dev/null -w '%{http_code}\n' http://[[HEALTH_NODE_IP="<third-live-node-ip>"]]:30080/healthz"

What does this health loop prove?

The watch command repeats the request once per second.
The curl command discards the response body.
The output shows the HTTP status returned through the surviving node's port 30080.

The health terminal should refresh with 200. The dashboard should still report Cluster connected.

Stopping a VM is deliberate here. The remaining replicas and control-plane servers stay active.

Before you stop the replica-hosting VM, predict whether either live check will lose its successful state.

Destroy the selected replica-hosting VM from another Ubuntu host terminal by running this command:

sudo virsh destroy [[API_REPLICA_NODE="<one-api-replica-node>"]]

What does this command do?

The command immediately terminates the selected guest VM. Its API Pod stops serving traffic with the host.

The dashboard remains Connected through the API replica on another VM. The health loop continues showing 200 responses.

Did the dashboard disconnect?

Confirm that the browser address uses a VM that is still running. Check that the selected health address belongs to a surviving node.

Inspect the API Pods from a surviving server if both checks fail. Help me diagnose this replica failover.

Restart the stopped VM by running this command from the Ubuntu host terminal:

sudo virsh start [[API_REPLICA_NODE="<one-api-replica-node>"]]

What happens after restart?

The guest boots with its existing virtual disk. Its K3s server reconnects to the cluster using the state already stored on that VM.

The node may remain NotReady briefly while K3s reconnects. Give the cluster time to restore its membership before checking placement.

Before you check recovery, predict whether all three nodes will return to Ready with two API replicas available.

Verify the restored nodes plus API placement by running these commands from the Ubuntu host terminal:

ssh "devops@$NODE1_IP" 'sudo k3s kubectl get nodes -o wide'
ssh "devops@$NODE1_IP" 'sudo k3s kubectl get pods -n batchforge -l app=batchforge-api -o wide'

What does recovery look like?

The first command lists the three K3s server nodes.
The second command confirms that the Deployment still has two ready API Pods.
The node column confirms that the replicas remain separated across hostnames.

You should see batchforge-1, batchforge-2, plus batchforge-3 as Ready. You should also see two ready API Pods on different nodes.

Node still not Ready?

Confirm that the VM appears as running in libvirt. Rerun the node check after the K3s service has rejoined.

If one API Pod remains pending, inspect its scheduling details from a surviving server. Help me investigate this node recovery.

Stop the health loop with Ctrl+C after the nodes and replicas recover.

Visualize Job-controller reconciliation

A Job controller keeps the run-to-completion workload aligned with its desired state. Deleting the active worker gives the controller a visible reason to create another attempt.

Switch back to the connected BatchForge dashboard.
Set Simulated processing time to 60 seconds.
Click Submit Kubernetes Job.
Record the created Job name here: <job-name>.
Track the worker Pods from another Ubuntu host terminal by running this command:

ssh "devops@$NODE1_IP" 'sudo k3s kubectl get pods -n batchforge -l batchforge-job=[[JOB_NAME="<job-name>"]] -w'

What does this watch show?

The label selector limits the stream to Pods owned by your new Job.
The -w flag keeps the command open for later Pod changes.
The first active worker appears before you trigger the disruption.

You should see one worker Pod move into its active state while the dashboard card reports the Job as Running.

Record the active worker Pod name here: <worker-pod>.

Before you delete the worker, predict whether the original Job will fail or create another attempt.

Delete the active worker from a separate Ubuntu host terminal by running this command:

ssh "devops@$NODE1_IP" 'sudo k3s kubectl delete pod [[WORKER_POD="<worker-pod>"]] -n batchforge'

What does this disruption test?

The command removes the active worker Pod while its Job still requires successful completion. The Job controller responds by creating a replacement Pod.

The watch output shows the original Pod leave. It then shows the replacement attempt.

Replacement Pod not appearing?

Confirm that you deleted the active worker Pod for the current Job. Check that the Job card remains in Running state during the replacement.

Keep the watch command open long enough to receive the controller update. Help me diagnose this Job replacement.

Watch the first terminal until the replacement Pod reaches Succeeded.
Stop the Pod watch with Ctrl+C.
Return to the BatchForge dashboard.
Keep the dashboard open until the original Job card reports Succeeded.

The card's attempt count should be higher than one. Its result cards should show the word count plus the unique-word count plus the SHA-256 digest.

That's the resilience story complete. BatchForge now stays reachable through a VM failure and repairs interrupted batch execution through controller reconciliation.

Secret mission

Protect Planned Maintenance with a Disruption Budget

Add an availability policy for planned maintenance. You will drain an API-replica node while BatchForge stays connected through the remaining healthy replica.

Clean Up Your Resources

Choose whether to keep the virtual lab running, pause it for later, or delete its infrastructure. The lab costs $0 because everything runs on your Ubuntu host. Active VMs still consume local CPU and memory.

Resources you used:

Three persistent libvirt VMs named batchforge-1, batchforge-2, and batchforge-3.
Three 12 GB QCOW2 overlay disks plus the shared Ubuntu cloud-image backing file.
A three-server K3s cluster with embedded-etcd state and the batchforge namespace.
Local lab artifacts in .lab, including the image archive and private server token.

Keep everything running

No action is needed while you are still demonstrating BatchForge. The three VMs keep the dashboard and API available through the existing NodePort.

Leave all three BatchForge VMs active.
Access the dashboard through port 30080 on any node IP.
Retain k8s/app.yaml as the cluster desired state.
Retain k8s/pdb.yaml as the planned-maintenance policy.

Pause - I'll come back to this later

Pausing the lab releases the CPU and memory used by the VMs. Their disks preserve the K3s cluster for your return.

Shut down the three BatchForge VMs by running these commands:

sudo virsh shutdown batchforge-1
sudo virsh shutdown batchforge-2
sudo virsh shutdown batchforge-3

What Do These Commands Do?

Each command asks libvirt to shut down one VM cleanly. The virtual disks and K3s state stay on disk.

Confirm the three VMs are inactive by running:

sudo virsh list --all

What Should You See?

The domain table should list all three BatchForge VMs as inactive. This confirms that the lab has stopped consuming CPU and memory.

Your lab is now paused with its cluster state preserved.

Resume one paused VM at a time by replacing <vm-name> with its listed VM name in this command:

sudo virsh start <vm-name>

How Does Resuming Work?

Run the command once for each BatchForge VM when you return. K3s uses the preserved disks to restore the existing cluster.

Delete - I don't want to use this again

Deleting the virtual lab permanently removes its VMs and cluster state. Your source repository and local Docker image remain on the Ubuntu host for a future rebuild.

Delete the virtual lab from your existing batchforge-lab repository by running:

./scripts/destroy-lab.sh

What Does the Deletion Script Remove?

The script force-stops and undefines batchforge-1, batchforge-2, and batchforge-3.
It deletes the three QCOW2 overlay disks and the shared backing image.
It removes .lab with the image archive and private server token.
Its final line reports BatchForge lab deleted.

Confirm the BatchForge domains are gone by running:

sudo virsh list --all

How Do You Know It Is Deleted?

The domain table should no longer include the three BatchForge VMs. Their K3s cluster and namespaced resources disappeared with them.

That's the virtual lab cleanup complete. Your project code remains available if you decide to provision it again.

Nice Work!

You did it! BatchForge now presents live Kubernetes Jobs through a polished dashboard backed by a resilient three-server K3s cluster.

You've learned how to:

Build a responsive dashboard that tracks each batch Job from submission to result.
Provision a high-availability cluster across three independent Ubuntu VMs with embedded etcd quorum.
Prove controller reconciliation by preserving API access through a VM failure. Watch an interrupted Job gain a replacement Pod before succeeding.
Secret Mission: Add a PodDisruptionBudget that preserves one healthy BatchForge replica during planned node maintenance.

Ready to quiz yourself?