Build a Self-Healing K3s Workload

Build a self-healing K3s app on AWS Free Tier with Terraform and Ansible.

Introduction

30 Second Summary

When a web service loses its only running copy, users face an outage until someone intervenes. Reliable systems need a way to rebuild failed work automatically.

In this project, you will automate a self-healing Kubernetes application on AWS. Terraform will provision the host while Ansible installs K3s for a recovery demo.

What You'll Build

Your browser will show the healthy application again after a Deployment replaces two Pods you deliberately delete.

By the end of this project, you'll have:

  • A reproducible AWS environment you can create on demand from Terraform.
  • A browser-visible application served from the host through a public URL.
  • A failure and recovery demo that shows a naked Pod staying deleted before a Deployment restores two replicas.
  • Secret Mission: Add CPU and memory requests plus limits to the resilient workload for verification in Kubernetes.

Are there any prerequisites?

You need an AWS account with credentials allowed to create every resource in the supplied Terraform configuration. Your Windows PC should already have Windows Subsystem for Linux 2 plus the required tools.

Before We Start

This step is your moment to commit to the system you are building before the hands-on work begins. Terraform will provision the AWS network and Ubuntu EC2 host. Ansible will configure the host and install K3s. Kubernetes will keep the application aligned with its declared desired state.

Validate Your AWS Automation Toolchain

This project crosses four control planes. A version mismatch can send later failures toward the wrong cause after AWS resources begin billing.

Terraform provisions the cloud infrastructure. Ansible configures the host.

Kubernetes reconciles the application workload. This step confirms that WSL2 can connect those layers with the intended identity.

In this step, get ready to:
  • Validate the automation tools available in WSL.
  • Confirm the active AWS identity.
  • Prepare Terraform with restricted SSH access.
Verify tools and AWS identity

The AWS CLI uses your configured credentials when Terraform requests cloud resources. Confirming the resolved identity prevents the lab from landing in an unexpected account.

Credential checks can feel sensitive. The identity lookup returns account metadata without displaying your access keys.

  • Run the version checks plus the AWS identity lookup in your WSL terminal by using these commands:
terraform version
ansible --version
aws --version
aws sts get-caller-identity

What should these checks show?

  • The Terraform result should show version 1.17.0.
  • The Ansible result should show Ansible Core 2.21.5.
  • The AWS CLI result confirms that the command responds.
  • The identity response should contain an Account value.
  • The identity response should contain an Arn value.
  • Compare the Account value with the AWS account intended for this lab.
  • Confirm that the Arn value represents the intended AWS identity.

You now know which local tools are running. You also know which AWS identity receives the infrastructure in the next step.

Seeing an unexpected tool or identity?

A different version can mean that WSL resolved another executable on its command path. An unexpected identity means that the AWS CLI loaded a different credential configuration.

Help me diagnose the WSL versions or AWS identity.

Generate the disposable SSH key

An SSH key pair gives Ansible access to the Ubuntu host without storing a password in the project. Terraform sends only the public half to AWS.

The private key grants access to the host. You will exclude both key files from source control before generating them.

  • Press the Windows key to open the Windows search bar.
  • Type Visual Studio Code into the search field.
  • Press Enter to open Visual Studio Code.
  • Select the project folder containing terraform/ through the folder picker.
  • Create .gitignore at the top level of that project folder.
  • Protect the generated files by replacing the contents of .gitignore with:
.terraform/
*.tfstate
*.tfstate.*
terraform/terraform.tfvars
my_key
my_key.pub
ansible/inventory.ini

What do these rules protect?

  • The Terraform rules exclude downloaded providers.
  • The state rules exclude local infrastructure state.
  • The terraform/terraform.tfvars rule excludes your public network input.
  • The key rules exclude both halves of the disposable SSH key pair.
  • The inventory rule excludes the generated EC2 host address.
  • Save .gitignore.
  • Confirm that all seven ignore patterns remain visible in the saved file.

Your local credentials plus generated infrastructure data now have source-control boundaries.

Missing an ignore pattern?

Confirm that .gitignore sits beside the terraform/ folder. Check every filename against the reference above.

Help me fix the project ignore rules.

The key command asks twice for an optional passphrase. Press Enter at each prompt so Ansible can use this disposable lab key without interactive input.

  • Generate the RSA key pair from the directory containing terraform/ by running this command:
ssh-keygen -t rsa -f my_key

What does this command create?

  • The my_key file is the private key that Ansible uses for SSH.
  • The my_key.pub file is the public key that Terraform registers with AWS.
  • The empty passphrase supports unattended connections during this disposable lab.
  • Confirm that my_key appears in the Visual Studio Code file sidebar.
  • Confirm that my_key.pub appears beside it.
  • Check the source control panel for pending changes.

Neither key file should appear among the pending source control changes. Your private key stays local.

Missing a key file?

Confirm that you ran the command from the directory containing terraform/. Check that you submitted an empty value at both passphrase prompts.

Help me troubleshoot the missing SSH key files.

Restrict SSH and validate Terraform

A CIDR value defines which source addresses may connect through a network rule. The /32 suffix limits SSH access to one public IPv4 address.

  • Retrieve your current public IPv4 address by running this command:
curl https://checkip.amazonaws.com

What does this command return?

The service returns the public IPv4 address seen for your current connection. Terraform uses that address as the permitted source for SSH.

  • Record the returned address here: your public IPv4 address.

Public address missing?

Confirm that WSL has internet access. Retry the command after your network connection is available.

Help me retrieve my public IPv4 address from WSL.

Terraform reads local input values from terraform/terraform.tfvars. The ignore rule keeps this network-specific value outside source control.

  • Select the terraform/ folder in the Visual Studio Code file sidebar.
  • Create terraform/terraform.tfvars inside that folder.

You should see terraform/terraform.tfvars beside the tracked Terraform files.

  • Set allowed_ssh_cidr to a quoted value containing your public IPv4 address followed immediately by /32.
  • Save terraform/terraform.tfvars.
  • Confirm that terraform/versions.tf exists.
  • Confirm that terraform/variables.tf exists.
  • Confirm that terraform/main.tf exists.
  • Confirm that terraform/outputs.tf exists.
  • Compare the tracked project files with the complete reference below.

✔️ Awesome, I've got everything!

Your tracked files match the intended configuration. The local input file contains your restricted SSH source.

  • Confirm that each tracked file is saved.

ⓧ I'd like to double check the full code

Use these references to correct the tracked project files before initialization.

  • Create any missing named file in its shown location.
  • Replace any mismatched contents with the corresponding reference.
.terraform/
*.tfstate
*.tfstate.*
terraform/terraform.tfvars
my_key
my_key.pub
ansible/inventory.ini

This file excludes provider downloads plus generated project data from source control.

terraform {
  required_version = "= 1.17.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "= 6.70.0"
    }
    local = {
      source  = "hashicorp/local"
      version = "= 2.9.1"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

This file pins Terraform plus both required providers. It also connects the AWS provider to the selected region.

variable "aws_region" {
  description = "AWS Region for the lab resources."
  type        = string
  default     = "us-east-1"
}

variable "allowed_ssh_cidr" {
  description = "The learner's public IPv4 address in /32 CIDR notation."
  type        = string
}

These variables define the AWS region plus the permitted SSH source.

data "aws_ssm_parameter" "ubuntu_ami" {
  name = "/aws/service/canonical/ubuntu/server/26.04/stable/current/amd64/hvm/ebs-gp3/ami-id"
}

resource "aws_vpc" "lab" {
  cidr_block           = "10.42.0.0/16"
  enable_dns_support   = true
  enable_dns_hostnames = true

  tags = {
    Name = "self-healing-k3s-lab"
  }
}

resource "aws_subnet" "public" {
  vpc_id                  = aws_vpc.lab.id
  cidr_block              = "10.42.1.0/24"
  map_public_ip_on_launch = true

  tags = {
    Name = "self-healing-k3s-public"
  }
}

resource "aws_internet_gateway" "lab" {
  vpc_id = aws_vpc.lab.id

  tags = {
    Name = "self-healing-k3s-igw"
  }
}

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.lab.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.lab.id
  }

  tags = {
    Name = "self-healing-k3s-public"
  }
}

resource "aws_route_table_association" "public" {
  subnet_id      = aws_subnet.public.id
  route_table_id = aws_route_table.public.id
}

resource "aws_security_group" "k3s" {
  name_prefix = "self-healing-k3s-"
  description = "Access for the self-healing K3s lab"
  vpc_id      = aws_vpc.lab.id

  tags = {
    Name = "self-healing-k3s"
  }
}

resource "aws_vpc_security_group_ingress_rule" "ssh" {
  security_group_id = aws_security_group.k3s.id
  description       = "SSH from the learner public IP"
  cidr_ipv4         = var.allowed_ssh_cidr
  from_port         = 22
  ip_protocol       = "tcp"
  to_port           = 22
}

resource "aws_vpc_security_group_ingress_rule" "app" {
  security_group_id = aws_security_group.k3s.id
  description       = "Public access to the Kubernetes NodePort"
  cidr_ipv4         = "0.0.0.0/0"
  from_port         = 30080
  ip_protocol       = "tcp"
  to_port           = 30080
}

resource "aws_vpc_security_group_egress_rule" "all_ipv4" {
  security_group_id = aws_security_group.k3s.id
  cidr_ipv4         = "0.0.0.0/0"
  ip_protocol       = "-1"
}

resource "aws_key_pair" "lab" {
  key_name   = "self-healing-k3s-lab"
  public_key = file("${path.module}/../my_key.pub")
}

resource "aws_instance" "k3s" {
  ami                         = data.aws_ssm_parameter.ubuntu_ami.value
  instance_type               = "t3.small"
  subnet_id                   = aws_subnet.public.id
  vpc_security_group_ids      = [aws_security_group.k3s.id]
  key_name                    = aws_key_pair.lab.key_name
  associate_public_ip_address = true

  metadata_options {
    http_endpoint = "enabled"
    http_tokens   = "required"
  }

  root_block_device {
    delete_on_termination = true
    encrypted             = true
    volume_type           = "gp3"
  }

  tags = {
    Name = "self-healing-k3s"
  }
}

resource "local_file" "ansible_inventory" {
  filename        = "${path.module}/../ansible/inventory.ini"
  file_permission = "0644"
  content         = <<-EOT
    [k3s]
    ${aws_instance.k3s.public_ip}
  EOT
}

This file declares the AWS network plus the Ubuntu EC2 host. It also writes the host address into the Ansible inventory.

output "public_ip" {
  description = "Public IPv4 address of the K3s host."
  value       = aws_instance.k3s.public_ip
}

output "app_url" {
  description = "Browser URL for the NodePort application."
  value       = "http://${aws_instance.k3s.public_ip}:30080"
}

These outputs expose the host address plus the browser URL after provisioning.

  • Save every corrected file.

Terraform initialization downloads the pinned provider plugins. It prepares the configuration for validation.

  • Initialize the Terraform configuration from the directory containing terraform/ by running this command:
terraform -chdir=terraform init

What does initialization prepare?

Terraform downloads AWS Provider 6.70.0 plus Local Provider 2.9.1. It stores their working files inside terraform/.terraform/.

  • Confirm that terraform/.terraform/ now exists in the Visual Studio Code file sidebar.

The provider directory confirms that initialization completed.

Initialization did not complete?

Provider downloads require a working network connection. A mismatch in the tracked Terraform files can also stop initialization.

Help me troubleshoot Terraform initialization.

Before you run the final check, predict whether Terraform can parse every declaration plus resolve both providers.

  • Validate the initialized Terraform configuration by running this command:
terraform -chdir=terraform validate

What should you see?

Terraform should report Success! The configuration is valid. This proves that the configuration parses successfully with its required providers.

That is the preparation locked in. Your AWS identity is confirmed plus Terraform accepts the configuration that provisions the lab.

Terraform validation failed?

Use the reported filename plus line number to locate the mismatch. Compare that file with the complete reference above.

Help me troubleshoot the validation failure.

Your toolchain is ready to control the lab safely. Next up, Terraform turns these declarations into a reachable K3s host.

Provision the K3s Host with Terraform

Your Terraform toolchain is validated. The local configuration is ready to become live infrastructure.

A configuration file alone cannot host K3s. Terraform now creates the AWS network around an EC2 host.

Terraform also passes the host address to Ansible through a generated inventory. That handoff keeps infrastructure provisioning separate from host configuration.

In this step, get ready to:
  • Review how the declared VPC protects the future K3s host.
  • Preview the AWS changes before creating the infrastructure.
  • Verify that Terraform gives Ansible the correct public IPv4 address.
Review the AWS architecture

The configuration divides the platform into network access, compute capacity, and an inventory handoff. Reviewing those boundaries now makes the upcoming plan easier to judge.

  • Return to terraform/main.tf in Visual Studio Code.
  • Locate aws_vpc.lab to identify the isolated network boundary.
  • Trace the public network path through aws_subnet.public, aws_internet_gateway.lab, aws_route_table.public, and aws_route_table_association.public.
  • Inspect aws_security_group.k3s plus its ingress and egress rules.
  • Inspect aws_key_pair.lab to confirm that AWS receives the disposable public key.
  • Inspect aws_instance.k3s to identify the host settings.
  • Locate local_file.ansible_inventory to see how Terraform hands the host address to Ansible.

What does the configuration declare?

  • The aws_vpc.lab resource uses 10.42.0.0/16. DNS support plus DNS hostnames are enabled.
  • The public subnet uses 10.42.1.0/24. Its route sends internet traffic through the internet gateway.
  • The security group restricts SSH on TCP port 22 to var.allowed_ssh_cidr.
  • The security group exposes the NodePort on TCP port 30080 to 0.0.0.0/0. All IPv4 egress is allowed.
  • The EC2 instance uses t3.small with the self-healing-k3s-lab key pair. IMDSv2 is required.
  • The encrypted gp3 root volume is deleted when the instance is terminated. The generated inventory places the public IPv4 address under [k3s].
  • Return to terraform/outputs.tf in Visual Studio Code.
  • Identify public_ip as the host address that Ansible uses.
  • Identify app_url as the browser address for the future workload.
Plan and apply the infrastructure

A Terraform plan compares the declared configuration with the current infrastructure. It gives you a safe review point before AWS receives any changes.

Before you run the plan, do you expect Terraform to create resources, update existing resources, or remove resources?

  • Preview the intended AWS resources by running this command:
terraform -chdir=terraform plan

What does this command do?

The -chdir=terraform option tells Terraform to run from the terraform directory. The plan calculates the changes without creating resources.

This preview is your chance to catch an unexpected deletion or replacement. Only the declared lab infrastructure should appear.

  • Confirm that the plan creates the VPC plus its public network path.
  • Confirm that the plan restricts TCP port 22 to var.allowed_ssh_cidr.
  • Confirm that the plan exposes TCP port 30080 for the application.
  • Confirm that the plan creates the key pair plus the t3.small instance.
  • Confirm that the plan creates ansible/inventory.ini through local_file.ansible_inventory.

Plan failing before AWS changes anything?

  • Confirm that your WSL2 session can still use the validated AWS credentials.
  • Check that terraform/terraform.tfvars still contains allowed_ssh_cidr with your current public IPv4 address.
  • Check that my_key.pub still exists beside the terraform directory.

Help me diagnose why this Terraform plan cannot read my AWS credentials or project inputs.

This is the point where potentially billable AWS resources enter your account. You keep control because Terraform pauses for explicit approval.

AWS charges begin here

The public IPv4 address costs $0.005 per hour while it exists. EC2 charges depend on your account eligibility plus usage.

EBS charges also depend on account eligibility plus usage. The cleanup section removes the managed infrastructure when you finish.

Terraform displays the execution plan again before requesting approval. The apply proceeds only after you enter the exact confirmation value.

  • Create the planned AWS infrastructure by running this command:
terraform -chdir=terraform apply

What does this command do?

Terraform sends the approved resource changes to AWS. It also records the resulting resource identities in local Terraform state.

Once the EC2 public IPv4 address exists, the Local Provider writes that value into ansible/inventory.ini.

  • Review the displayed execution plan for unexpected changes.
  • Enter yes when Terraform asks for approval.

Terraform can spend a few minutes waiting for AWS to create the network plus the EC2 instance. A quiet pause during that work is normal.

That's the infrastructure layer live. Your EC2 host now has the network path required for remote configuration.

Apply stopped before completion?

  • Read the failed resource name in the Terraform output to isolate the affected declaration.
  • Confirm that my_key.pub still contains the public half of your disposable key pair.
  • Confirm that your AWS identity has permission to create the resource named in the failure.

Help me troubleshoot the failed Terraform apply for this AWS K3s host.

Verify the Terraform and Ansible handoff

Terraform outputs expose the values needed outside the infrastructure configuration. The generated inventory gives Ansible the same host address without manual copying.

  • Read the public address plus application URL by running this command:
terraform -chdir=terraform output

What do these outputs show?

The public_ip output identifies the EC2 host. The app_url output combines that address with TCP port 30080.

Reading outputs does not change the infrastructure. It exposes selected values from Terraform state.

  • Record public_ip as your EC2 public IPv4 address.
  • Record app_url as http://your-public-ip:30080.
  • Return to ansible/inventory.ini in Visual Studio Code.
  • Confirm that the first line contains [k3s].
  • Confirm that the next line contains your EC2 public IPv4 address.

Before you run the final check, do you expect its value to match the address in ansible/inventory.ini?

  • Print only the Terraform public IPv4 output by running this command:
terraform -chdir=terraform output -raw public_ip

What does the raw output prove?

The -raw option prints the string value without extra formatting. This makes a direct comparison with the generated inventory straightforward.

You should see your EC2 public IPv4 address. The same address should appear directly below [k3s] in ansible/inventory.ini.

Inventory address does not match?

  • Confirm that you opened the generated ansible/inventory.ini file from this project.
  • Return to the apply chunk above if the inventory file is missing or stale.

Help me compare the Terraform public IP with the generated Ansible inventory.

Your AWS host is running, and Ansible knows exactly where to find it. Next, you turn that Ubuntu host into a K3s node.

Configure K3s with Ansible

Terraform has given you a reachable Ubuntu EC2 host. That host needs K3s before it can run a Kubernetes workload.

Ansible now applies the repeatable operating system configuration that creates the cluster. The playbook also uses idempotency to preserve an existing installation during later runs.

In this step, get ready to:
  • Review the pinned K3s installation tasks in the Ansible playbook.
  • Configure the Ubuntu host as a single-node K3s cluster.
  • Rerun the playbook to prove that K3s is not reinstalled.
Review the K3s installation playbook

The playbook owns the host configuration layer. Its tasks install the required package, fetch the installer, pin the cluster version, and wait for a healthy node.

  • Return to the Visual Studio Code window from earlier.
  • Select ansible/install-k3s.yml in the Explorer sidebar.
  • Locate the Install pinned curl task.
  • Trace the remaining tasks from the package installation to the node readiness check.

How does the playbook build the cluster?

  • The ansible.builtin.apt task installs curl=8.18.0-1ubuntu2.8 from the Ubuntu package repository.
  • The ansible.builtin.get_url task downloads the official installer to /tmp/install-k3s.sh.
  • The '0755' mode makes the downloaded installer executable.
  • The INSTALL_K3S_VERSION environment value pins the installation to v1.37.1+k3s1.
  • The creates guard preserves an existing /usr/local/bin/k3s binary during later runs.
  • The final task retries the node check until its output contains Ready.

✔️ Awesome, I've got everything!

You have identified the package pin, installer location, K3s version pin, idempotency guard, and readiness check.

ⓧ I'd like to double check the full code

---
- name: Install a pinned single-node K3s cluster
  hosts: k3s
  become: true
  gather_facts: false

  tasks:
    - name: Install pinned curl
      ansible.builtin.apt:
        name: curl=8.18.0-1ubuntu2.8
        state: present
        update_cache: true

    - name: Download the official K3s installer
      ansible.builtin.get_url:
        url: https://get.k3s.io
        dest: /tmp/install-k3s.sh
        mode: '0755'

    - name: Install the pinned K3s release once
      ansible.builtin.command:
        cmd: /tmp/install-k3s.sh
        creates: /usr/local/bin/k3s
      environment:
        INSTALL_K3S_VERSION: "v1.37.1+k3s1"

    - name: Wait for the K3s node to report Ready
      ansible.builtin.command:
        cmd: /usr/local/bin/k3s kubectl get nodes
      register: k3s_nodes
      until: "'Ready' in k3s_nodes.stdout"
      retries: 24
      delay: 5
      changed_when: false

What should you check?

Compare each task in your file with this reference. Pay close attention to indentation because YAML uses spacing to define structure.

Install K3s and confirm node health

The first playbook run connects to the host from ansible/inventory.ini. It uses the ubuntu account with your disposable my_key private key.

The first run can take several minutes while Ubuntu refreshes its package information and K3s starts. A quiet wait during those tasks is expected.

Before you run the playbook, which final state do you expect the node check to report?

  • Install the pinned K3s cluster by running this command from the project root:
ansible-playbook -v -i ansible/inventory.ini -u ubuntu --private-key my_key ansible/install-k3s.yml

What does this command do?

  • The -i ansible/inventory.ini option targets the public host written by Terraform.
  • The -u ubuntu option selects the default account for the Ubuntu EC2 image.
  • The --private-key my_key option authenticates with the matching private key.
  • The -v option shows each playbook task as Ansible processes it.

✔️ I see the pinned version and Ready

The final node table shows Ready in the status column. Its version column shows v1.37.1+k3s1.

Your EC2 host is now a working single-node K3s cluster.

ⓧ I see an older version

An older version means /usr/local/bin/k3s already existed on a reused host. The creates guard preserves that binary.

Replacing a reused lab host can feel drastic. Terraform rebuilds it from the configuration you already validated.

  • Return to Step 2 to replace the reused EC2 host with the existing Terraform configuration.
  • Confirm that ansible/inventory.ini contains the replacement public IP.
  • Install the pinned cluster on the replacement host by running:
ansible-playbook -v -i ansible/inventory.ini -u ubuntu --private-key my_key ansible/install-k3s.yml

Still seeing the older version?

This command reapplies the same pinned host configuration to the address in ansible/inventory.ini. Confirm that the inventory contains the new host address before retrying.

Help me diagnose why the replacement K3s host still reports an older version.

ⓧ K3s is not installed

A failed playbook task leaves the host before the final readiness check. The task name in the Ansible output identifies which part needs attention.

  • Read the task name immediately above the failure details.
  • Confirm that the security group still permits SSH from your current public IPv4 address.
  • Retry the full host configuration by running:
ansible-playbook -v -i ansible/inventory.ini -u ubuntu --private-key my_key ansible/install-k3s.yml

Playbook still failing?

The retry reconnects to the inventory host and resumes the desired configuration. Previously completed tasks remain in their correct state.

Help me troubleshoot the failed K3s installation task.

That is the host configuration layer working: Ansible has turned your Ubuntu instance into a K3s node that accepts Kubernetes commands.

Prove the playbook is idempotent

A repeatable playbook can run again without rebuilding resources that already match the declared configuration. The creates: /usr/local/bin/k3s guard provides that behavior for the installation task.

Before the second run, do you expect the K3s installation task to change the host again?

  • Run the same playbook again to test its idempotent behavior:
ansible-playbook -v -i ansible/inventory.ini -u ubuntu --private-key my_key ansible/install-k3s.yml

What does the second run prove?

Ansible evaluates every task against the host again. The existing /usr/local/bin/k3s binary causes the installation command to be skipped.

The final node check still runs because it verifies current cluster health. You should see the node report Ready again.

The recap shows that the Install the pinned K3s release once task did not reinstall K3s. Your cluster remains healthy after the repeat run.

Installation task changed again?

Check that the task still includes creates: /usr/local/bin/k3s beneath ansible.builtin.command. Incorrect indentation can detach the guard from the command task.

Help me find why my K3s installation task runs on every playbook execution.

Your cluster is healthy and repeatable. Next, you will deploy a naked Pod and watch what happens when no controller owns it.

Demonstrate a Naked Pod Failure

Your K3s node is ready to schedule containers. The next question is whether a running application can return after its only workload object disappears.

A naked Pod creates a focused recovery test. You will serve a real page before removing the Pod to observe what Kubernetes does.

In this step, get ready to:
  • Create the fragile workload manifest and its deployment playbook.
  • Load the fragile application through its public NodePort URL.
  • Delete the naked Pod and observe whether the application recovers.
Create and deploy the fragile workload

The application page lives in a ConfigMap. A Pod mounts that content into NGINX.

A NodePort Service carries browser traffic to the Pod. An Ansible playbook applies all three objects to K3s.

  • Switch back to your project in Visual Studio Code.
  • Create a folder named k8s beside the existing terraform and ansible folders.
  • Create fragile-app.yaml inside the new k8s folder.
  • Add the page content to k8s/fragile-app.yaml by pasting this first document:
apiVersion: v1
kind: ConfigMap
metadata:
  name: self-healing-html
data:
  index.html: |
    <!doctype html>
    <html lang="en">
      <head>
        <meta charset="utf-8">
        <title>Fragile Pod</title>
      </head>
      <body>
        <h1>Fragile Pod: running</h1>
        <p>This page disappears when its naked Pod is deleted.</p>
      </body>
    </html>

What does this manifest do?

  • The ConfigMap stores the HTML page separately from the container image.
  • The index.html key becomes the file that NGINX serves later.
  • The page heading creates a clear visual signal that the fragile workload is running.
  • Save k8s/fragile-app.yaml.
  • Confirm the file now contains the ConfigMap named self-healing-html.

ConfigMap document looks misplaced?

Check that fragile-app.yaml sits inside the k8s folder. Make sure apiVersion starts at the far left of the file.

Help me check the ConfigMap document in my fragile Kubernetes manifest.

  • Place your cursor after the ConfigMap document in k8s/fragile-app.yaml.
  • Add the naked Pod as the second document by pasting this code:
---
apiVersion: v1
kind: Pod
metadata:
  name: fragile-web
  labels:
    app: self-healing-demo
spec:
  containers:
    - name: web
      image: nginx:1.30.6-alpine
      ports:
        - containerPort: 80
      volumeMounts:
        - name: html
          mountPath: /usr/share/nginx/html
          readOnly: true
  volumes:
    - name: html
      configMap:
        name: self-healing-html

How does the Pod serve the page?

  • The Pod runs the pinned NGINX container image.
  • The app=self-healing-demo label gives the Service a stable way to select the Pod.
  • The ConfigMap volume places the fragile page in the directory served by NGINX.
  • Save k8s/fragile-app.yaml.
  • Confirm the file now contains a second document with kind: Pod and name: fragile-web.

Pod document not lining up?

Check that the Pod document begins with ---. Verify that containers and volumes have the same indentation beneath spec.

Help me inspect the Pod document and its ConfigMap volume indentation.

  • Place your cursor after the Pod document in k8s/fragile-app.yaml.
  • Add the public Service as the third document by pasting this code:
---
apiVersion: v1
kind: Service
metadata:
  name: self-healing-demo
spec:
  type: NodePort
  selector:
    app: self-healing-demo
  ports:
    - port: 80
      targetPort: 80
      nodePort: 30080

How does traffic reach the Pod?

  • The Service selects workloads carrying the app=self-healing-demo label.
  • Port 30080 accepts traffic through the EC2 host.
  • The Service forwards that traffic to port 80 on the selected Pod.
  • Save k8s/fragile-app.yaml.
  • Confirm the file ends with the NodePort value 30080.

Service document not selecting the Pod?

Compare the Service selector with the Pod label. Both values must use app: self-healing-demo.

Help me compare the Service selector with the Pod label in my manifest.

The manifest defines the remote objects. The deployment playbook copies that manifest to the K3s host before applying it.

  • Create deploy-fragile.yml inside the existing ansible folder.
  • Build the deployment workflow by pasting this playbook into ansible/deploy-fragile.yml:
---
- name: Deploy the fragile naked Pod
  hosts: k3s
  become: true
  gather_facts: false

  tasks:
    - name: Copy the fragile manifest
      ansible.builtin.copy:
        src: ../k8s/fragile-app.yaml
        dest: /tmp/fragile-app.yaml
        mode: '0644'

    - name: Apply the fragile manifest
      ansible.builtin.command:
        cmd: /usr/local/bin/k3s kubectl apply -f /tmp/fragile-app.yaml

    - name: Show the fragile Pod
      ansible.builtin.command:
        cmd: /usr/local/bin/k3s kubectl get pod fragile-web
      changed_when: false

What does this playbook do?

  • The copy task transfers the local manifest to the remote Ubuntu host.
  • The apply task sends all three manifest documents to the K3s API.
  • The final task retrieves fragile-web so the playbook output proves the Pod exists.
  • Save ansible/deploy-fragile.yml.
  • Deploy the fragile application from your project root by running:
ansible-playbook -v -i ansible/inventory.ini -u ubuntu --private-key my_key ansible/deploy-fragile.yml

You should see the manifest copied before the three Kubernetes objects are applied. The final task should list fragile-web as a running Pod.

Fragile deployment did not complete?

Confirm that the terminal is at the project folder containing ansible. Check that my_key remains in that same folder.

Help me troubleshoot the fragile workload deployment playbook.

✔️ Awesome, I've got everything!

Great. Save both files before you continue to the browser check.

ⓧ I'd like to double check the full code

apiVersion: v1
kind: ConfigMap
metadata:
  name: self-healing-html
data:
  index.html: |
    <!doctype html>
    <html lang="en">
      <head>
        <meta charset="utf-8">
        <title>Fragile Pod</title>
      </head>
      <body>
        <h1>Fragile Pod: running</h1>
        <p>This page disappears when its naked Pod is deleted.</p>
      </body>
    </html>
---
apiVersion: v1
kind: Pod
metadata:
  name: fragile-web
  labels:
    app: self-healing-demo
spec:
  containers:
    - name: web
      image: nginx:1.30.6-alpine
      ports:
        - containerPort: 80
      volumeMounts:
        - name: html
          mountPath: /usr/share/nginx/html
          readOnly: true
  volumes:
    - name: html
      configMap:
        name: self-healing-html
---
apiVersion: v1
kind: Service
metadata:
  name: self-healing-demo
spec:
  type: NodePort
  selector:
    app: self-healing-demo
  ports:
    - port: 80
      targetPort: 80
      nodePort: 30080
---
- name: Deploy the fragile naked Pod
  hosts: k3s
  become: true
  gather_facts: false

  tasks:
    - name: Copy the fragile manifest
      ansible.builtin.copy:
        src: ../k8s/fragile-app.yaml
        dest: /tmp/fragile-app.yaml
        mode: '0644'

    - name: Apply the fragile manifest
      ansible.builtin.command:
        cmd: /usr/local/bin/k3s kubectl apply -f /tmp/fragile-app.yaml

    - name: Show the fragile Pod
      ansible.builtin.command:
        cmd: /usr/local/bin/k3s kubectl get pod fragile-web
      changed_when: false
View the fragile application

The Terraform output named app_url combines the EC2 public address with the NodePort. Opening that URL tests the entire path from your browser to the Pod.

  • Read the application URL from Terraform by running:
terraform -chdir=terraform output -raw app_url
  • Record the returned URL here: your app URL.
  • Paste your app URL into your browser address bar.
  • Load the application by pressing Enter.

You should see the heading Fragile Pod: running. You should also see a sentence explaining that the page disappears when its naked Pod is deleted.

That is your first end-to-end workload running on K3s. Browser traffic now reaches the NGINX container through the public NodePort Service.

Page not loading?

Check that the deployment playbook listed fragile-web successfully. Confirm that your browser URL ends with port 30080.

Help me trace why the fragile page is unavailable through its NodePort URL.

Delete the naked Pod and test recovery

A controlled deletion reveals which Kubernetes object owns recovery. The break playbook removes only fragile-web from the cluster.

  • Create break-fragile.yml inside the existing ansible folder.
  • Define the controlled deletion by pasting this playbook into ansible/break-fragile.yml:
---
- name: Delete the naked Pod
  hosts: k3s
  become: true
  gather_facts: false

  tasks:
    - name: Delete fragile-web
      ansible.builtin.command:
        cmd: /usr/local/bin/k3s kubectl delete pod fragile-web

What does this playbook target?

The playbook deletes the Pod named fragile-web. The ConfigMap and NodePort Service remain in the cluster.

  • Save ansible/break-fragile.yml.
  • Confirm the playbook contains one task named Delete fragile-web.

Deletion playbook looks different?

Make sure the command targets fragile-web exactly. Verify that the task remains under the tasks key.

Help me check the structure of my Pod deletion playbook.

✔️ Awesome, I've got everything!

Your deletion playbook is ready for the recovery test.

ⓧ I'd like to double check the full code

---
- name: Delete the naked Pod
  hosts: k3s
  become: true
  gather_facts: false

  tasks:
    - name: Delete fragile-web
      ansible.builtin.command:
        cmd: /usr/local/bin/k3s kubectl delete pod fragile-web

Before you run this, do you think Kubernetes will restore the page after the Pod disappears?

  • Delete the naked Pod from your project root by running:
ansible-playbook -v -i ansible/inventory.ini -u ubuntu --private-key my_key ansible/break-fragile.yml

What did the break playbook do?

The successful play recap confirms that fragile-web was deleted. The Service still points at workloads carrying the matching label.

  • Return to the browser tab showing your app URL.
  • Refresh the page.

The page no longer loads. The Pod stayed deleted because no controller owns fragile-web or recreates it.

Why did the application stay unavailable?

The Service still exists. It has no matching Pod available to receive traffic.

A naked Pod carries no desired replica count for Kubernetes to maintain. Its deletion therefore becomes a lasting outage.

Page still visible?

Wait a few seconds before refreshing again. Confirm that the deletion playbook completed successfully for fragile-web.

Help me verify whether the fragile Pod was deleted while the browser still shows its page.

You have now demonstrated the exact weakness of a naked Pod. Next up, you will give Kubernetes a desired replica count so it can replace deleted Pods automatically.

Deploy a Self-Healing Application

The fragile Pod served your application until you deleted it. Kubernetes had no controller responsible for bringing it back.

You will now apply a two-replica Deployment through Ansible. Readiness probes control when each Pod receives traffic. Liveness probes check whether each container remains healthy.

In this step, get ready to:
  • Deploy a two-replica workload with readiness probes and liveness probes.
  • Confirm the healthy application through the existing NodePort Service.
  • Delete the managed Pods to prove that the Deployment restores them.
Introduce the Deployment controller

The Deployment declares that two matching Pods should exist. Its controller continually compares that desired state with the Pods currently running on your K3s node.

  • Switch back to the WSL terminal in the project root from earlier.
  • Apply the resilient manifest with the existing Ansible inventory by running this command:
ansible-playbook -v -i ansible/inventory.ini -u ubuntu --private-key my_key ansible/deploy-resilient.yml

What does this playbook do?

  • The playbook copies k8s/resilient-app.yaml to the K3s host.
  • It applies the ConfigMap, Deployment, and existing Service definitions.
  • It waits for the Deployment rollout to finish.
  • It lists the Pods labeled app=self-healing-demo.

The play recap should finish without a failed task. The final listing should show two Pods with the Running status.

Deployment rollout not completing?

Check that both ansible/inventory.ini and my_key still exist in the project root. Confirm that the K3s node from the earlier step remains reachable.

Compare the Pod details with the resilient manifest if the rollout keeps waiting. Help me diagnose why the self-healing Deployment rollout is not completing.

The resilient path depends on three files. Use this reference if you want to compare every workload field and automation task.

✔️ Awesome, I've got everything!

Your resilient manifest and both Ansible playbooks match the self-healing path.

ⓧ I'd like to double check the full code

Check k8s/resilient-app.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: self-healing-html
data:
  index.html: |
    <!doctype html>
    <html lang="en">
      <head>
        <meta charset="utf-8">
        <title>Self-Healing Deployment</title>
      </head>
      <body>
        <h1>Self-Healing Deployment: healthy</h1>
        <p>Kubernetes keeps two replicas running and routes traffic only to ready Pods.</p>
      </body>
    </html>
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: self-healing-demo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: self-healing-demo
  template:
    metadata:
      labels:
        app: self-healing-demo
    spec:
      containers:
        - name: web
          image: nginx:1.30.6-alpine
          ports:
            - containerPort: 80
          readinessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 2
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 5
            periodSeconds: 10
          volumeMounts:
            - name: html
              mountPath: /usr/share/nginx/html
              readOnly: true
      volumes:
        - name: html
          configMap:
            name: self-healing-html
---
apiVersion: v1
kind: Service
metadata:
  name: self-healing-demo
spec:
  type: NodePort
  selector:
    app: self-healing-demo
  ports:
    - port: 80
      targetPort: 80
      nodePort: 30080

How does the manifest create resilience?

The ConfigMap supplies the healthy page. The Deployment maintains two Pods with the same application label.

The readiness probe controls traffic eligibility. The liveness probe checks container health. The Service keeps the application available on port 30080.

Check ansible/deploy-resilient.yml
---
- name: Deploy the self-healing workload
  hosts: k3s
  become: true
  gather_facts: false

  tasks:
    - name: Copy the resilient manifest
      ansible.builtin.copy:
        src: ../k8s/resilient-app.yaml
        dest: /tmp/resilient-app.yaml
        mode: '0644'

    - name: Apply the resilient manifest
      ansible.builtin.command:
        cmd: /usr/local/bin/k3s kubectl apply -f /tmp/resilient-app.yaml

    - name: Wait for the Deployment rollout
      ansible.builtin.command:
        cmd: /usr/local/bin/k3s kubectl rollout status deployment/self-healing-demo --timeout=120s
      changed_when: false

    - name: Show the running Pods
      ansible.builtin.command:
        cmd: /usr/local/bin/k3s kubectl get pods -l app=self-healing-demo
      changed_when: false

How does deployment automation work?

The playbook transfers the manifest before applying it with the embedded K3s command. It waits for the rollout before listing the matching Pods.

Check ansible/test-healing.yml
---
- name: Prove that the Deployment replaces deleted Pods
  hosts: k3s
  become: true
  gather_facts: false

  tasks:
    - name: Delete all Pods owned by the Deployment
      ansible.builtin.command:
        cmd: /usr/local/bin/k3s kubectl delete pods -l app=self-healing-demo

    - name: Wait until two replacement Pods are running
      ansible.builtin.command:
        cmd: /usr/local/bin/k3s kubectl get pods -l app=self-healing-demo --field-selector=status.phase=Running -o name
      register: running_pods
      until: running_pods.stdout_lines | length == 2
      retries: 24
      delay: 5
      changed_when: false

How does the healing test work?

The playbook deletes every Pod matching the Deployment label. It repeats its check until two replacement Pods reach the running phase.

Confirm the healthy application

A readiness probe keeps a Pod out of the Service endpoints until its web server responds. A liveness probe detects an unhealthy container so Kubernetes can restart it.

  • Switch back to the browser tab that showed the fragile application earlier.
  • Refresh the application page.

You should see Self-Healing Deployment: healthy. That healthy page confirms that the existing NodePort Service is routing traffic to the new ready Pods.

Why use both probes?

Readiness controls whether a Pod receives requests. Liveness checks whether its container should be restarted.

Together they protect traffic flow while the Deployment protects the replica count.

Healthy page not loading?

Confirm that the browser still uses the app_url value from Terraform. Check that the URL includes port 30080.

A page that still shows the fragile message usually means the resilient ConfigMap was not applied. Help me troubleshoot why the self-healing application is not loading through the NodePort Service.

Prove the Deployment self-heals

Desired-state reconciliation keeps the declared replica count true over time. Deleting the managed Pods creates a gap that the Deployment controller must close.

Before you run the healing test, which part of the declared state do you expect Kubernetes to restore?

  • Delete the managed Pods through the healing test by running this command:
ansible-playbook -v -i ansible/inventory.ini -u ubuntu --private-key my_key ansible/test-healing.yml

What does the healing test prove?

  • The first task deletes every Pod with the app=self-healing-demo label.
  • The Deployment detects that its actual replica count has fallen below two.
  • The final task waits until two replacement Pods reach the running phase.

You should see two replacement Pod names in the final task output. Their presence proves that the Deployment restored its declared replica count.

  • Refresh the application page in your existing browser tab.

You should still see Self-Healing Deployment: healthy. You have now watched Kubernetes replace deleted Pods while the application returned to its healthy state.

Replacement Pods not reaching Running?

Check whether the playbook can still reach the host through ansible/inventory.ini. A changed public IP can make the generated inventory stale.

Inspect the Deployment manifest for the replica count and matching labels. Help me diagnose why the Deployment is not creating two replacement Pods.

Your application now has a controller that maintains two replicas after failure. The fragile outage has become a repeatable self-healing demonstration.

Secret mission

Add Resource Guardrails

Your Deployment already restores deleted Pods. Now add CPU and memory guardrails so every replacement Pod receives a defined share of the K3s node.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project uses billable AWS resources, so charges can continue while the lab is running.

Cost warning

Billing deserves a quick decision here. The Pause and Delete options below keep it under control.

The public IPv4 address costs $0.005 per hour. EC2 charges depend on your account eligibility, credits, region, and usage.

EBS charges also depend on your account eligibility, credits, region, and usage. Stopping the EC2 instance leaves persistent resources in place.

Resources you used:

  • Terraform-managed AWS network: one VPC with CIDR 10.42.0.0/16. The VPC contains one public subnet with CIDR 10.42.1.0/24.
  • AWS routing resources: one internet gateway. One route table connects the public subnet to that gateway.
  • AWS access controls: one security group. It contains one restricted SSH ingress rule. It contains one public NodePort ingress rule. It contains one all-IPv4 egress rule.
  • AWS compute resources: one EC2 key pair. One t3.small Ubuntu instance runs with an encrypted gp3 root volume. The instance uses one public IPv4 address.
  • Local Ansible inventory: ansible/inventory.ini contains the EC2 host address.
  • K3s runtime: one single-node cluster runs on the EC2 instance.
  • Kubernetes workload objects: the ConfigMap serves the application page. The Deployment manages two guarded Pods. The NodePort Service exposes the application on TCP port 30080.

Keep everything running

No action is needed while you continue using the demo. The AWS host stays available for more failure tests.

  • Keep the local Terraform state so Terraform can continue managing the stack.
  • Leave the EC2 instance running so the browser demo remains available.
  • Keep ansible/inventory.ini so Ansible can continue targeting the K3s host.
  • Budget for applicable public IPv4, EC2, and EBS charges while the stack remains active.

Pause - I'll come back to this later

Pausing shuts down the running EC2 instance while preserving the lab for later. Persistent storage can still incur applicable charges.

  • Stop the self-healing-k3s EC2 instance through AWS.
  • Retain the local Terraform state so Terraform still knows about the paused stack.
  • Retain the encrypted gp3 root volume so the K3s installation stays available.
  • Recheck the instance address after resuming it.
  • Confirm Ansible can reach the K3s inventory host before running another playbook.

Delete - I don't want to use this again

Remove the Terraform-managed stack when you no longer need the demo. This ends applicable usage from the AWS resources created for the lab.

  • Return to the WSL terminal from earlier.
  • Destroy the Terraform-managed stack by running this command:
terraform -chdir=terraform destroy

What does this command remove?

Terraform reads its local state to identify the AWS resources created by this project.

Destroying the EC2 instance also removes its root volume, K3s cluster, and Kubernetes workload objects. Terraform also removes the generated Ansible inventory.

  • Review the destroy plan to confirm it targets the lab resources.
  • Approve the plan when Terraform prompts you.
  • Keep the terminal open until Terraform finishes.
  • Confirm Terraform reports that its managed resources were destroyed.

Cleanup complete. Terraform has removed the AWS resources it managed for this lab.

Nice Work!

Nice Work!

You did it! You built a browser-visible application on an AWS-hosted K3s cluster. When its Pods are deleted, a two-replica Deployment restores them to the declared state.

What you learned:

  • Provisioned a reproducible AWS network with an EC2 host by using Terraform. Passed the public address into Ansible through a generated inventory.
  • Installed a pinned K3s cluster with an idempotent Ansible playbook. Confirmed the single node reported Ready before deploying workloads.
  • Exposed the NGINX page through a NodePort Service. Ran a controlled failure that left the naked Pod deleted. Proved Kubernetes desired-state reconciliation by watching the Deployment restore two Pods. Used readiness probes to control traffic eligibility. Used liveness probes to control container restarts.
  • Secret Mission: Added CPU and memory requests and limits to the resilient Deployment. Verified the resource guardrails on the recreated Pods.

Ready to quiz yourself?