Set Up a Kubernetes Cluster on AWS

Bootstrap a 3-node Kubernetes cluster from scratch using kubeadm on AWS EC2.

Introduction

30 Second Summary

Every time you use a streaming service, order food online, or scroll social media, your request is handled by containers running on Kubernetes clusters. Most engineers interact with these clusters after someone else has already built them.

In this project, you will build a 3-node Kubernetes cluster from scratch on AWS EC2 using kubeadm. You will install the container runtime, bootstrap the control plane, join worker nodes, and deploy a test application to prove it all works.

What You'll Build

A production-style 3-node Kubernetes cluster running on AWS EC2 instances, configured from the ground up using kubeadm with containerd as the container runtime and Calico for pod networking.

By the end of this project, you'll have:

  • A control plane node running the API server, etcd, scheduler, and controller manager.
  • Two worker nodes joined to the cluster and ready to run workloads.
  • AWS security groups configured with the exact ports Kubernetes components need to communicate.
  • A verified cluster with a test nginx deployment scheduled across your worker nodes.
  • Secret Mission: Deploy a multi-tier application with frontend and backend services, and observe how Kubernetes distributes pods across your worker nodes.

Are there any prerequisites?

You will need an AWS account with EC2 access and the AWS CLI installed and configured on your local machine. Familiarity with SSH and basic Linux terminal commands is helpful.

Launch EC2 Instances and Configure Security Groups

Every Kubernetes cluster needs infrastructure to run on. In this project, that means three EC2 instances in a shared network, with carefully configured firewall rules so the Kubernetes components can talk to each other.

Kubernetes components communicate on specific TCP ports. If those ports are blocked, the cluster will never form.

In this step, get ready to:
  • Create a security group with all the Kubernetes-required ports open between nodes.
  • Launch 3 t2.medium EC2 instances running Ubuntu 24.04.
  • Set unique hostnames on each instance for easy identification.
Create a security group for your Kubernetes cluster

Kubernetes components use specific TCP ports to communicate. The API server listens on port 6443, etcd uses 2379-2380, and kubelets use 10250. Without a security group that permits traffic on these ports between your nodes, the cluster cannot initialize.

  • Open your terminal application. Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or PowerShell (Windows) and press Enter to open it.
  • Create the security group by running this command:
# Create a security group in the default VPC for your Kubernetes cluster
SG_ID=$(aws ec2 create-security-group \
  --group-name kubernetes-cluster-sg \
  --description "Security group for Kubernetes cluster nodes" \
  --query 'GroupId' \
  --output text)

echo $SG_ID

What does this command do?

This creates a new security group called kubernetes-cluster-sg in your default VPC. The --query 'GroupId' flag extracts just the security group ID from the response and stores it in the SG_ID variable for use in the next commands.

You will see output like sg-0abc1234def56789 printed in your terminal. Copy this value somewhere safe. You will need it later.

  • Add an SSH rule so you can connect to your instances from your local machine by running:
# Allow SSH access from your current IP address
MY_IP=$(curl -s https://checkip.amazonaws.com)

aws ec2 authorize-security-group-ingress \
  --group-id $SG_ID \
  --protocol tcp \
  --port 22 \
  --cidr ${MY_IP}/32
  • Add the Kubernetes control plane and worker node port rules by running this command:
# Allow Kubernetes ports between nodes in the same security group
# API server
aws ec2 authorize-security-group-ingress \
  --group-id $SG_ID --protocol tcp --port 6443 --source-group $SG_ID

# etcd
aws ec2 authorize-security-group-ingress \
  --group-id $SG_ID --protocol tcp --port 2379-2380 --source-group $SG_ID

# Kubelet API
aws ec2 authorize-security-group-ingress \
  --group-id $SG_ID --protocol tcp --port 10250 --source-group $SG_ID

# kube-proxy
aws ec2 authorize-security-group-ingress \
  --group-id $SG_ID --protocol tcp --port 10256 --source-group $SG_ID

# kube-controller-manager
aws ec2 authorize-security-group-ingress \
  --group-id $SG_ID --protocol tcp --port 10257 --source-group $SG_ID

# kube-scheduler
aws ec2 authorize-security-group-ingress \
  --group-id $SG_ID --protocol tcp --port 10259 --source-group $SG_ID

# NodePort services (accessible from anywhere)
aws ec2 authorize-security-group-ingress \
  --group-id $SG_ID --protocol tcp --port 30000-32767 --cidr 0.0.0.0/0

Why these specific ports?

Each Kubernetes component listens on a dedicated port. The API server (6443) is the cluster's front door. etcd (2379-2380) stores all cluster state. The kubelet (10250) on each node reports status back to the control plane.

The --source-group $SG_ID flag creates a self-referencing rule. Any instance in this security group can talk to any other instance in the same group on that port. The NodePort range (30000-32767) is open to all IPs so you can access services from your browser later.

  • Verify your security group rules are in place by running:
aws ec2 describe-security-groups --group-ids $SG_ID \
  --query 'SecurityGroups[0].IpPermissions[*].[IpProtocol,FromPort,ToPort]' \
  --output table

You should see a table listing each TCP rule with the port ranges you just configured.

Getting a 'GroupId' error?

If you see an error about an invalid group ID, your SG_ID variable may be empty. Run echo $SG_ID to check. If it is blank, the create-security-group command failed. Check that your AWS CLI is configured to the correct region and that you have EC2 permissions.

Help me troubleshoot my security group creation

Launch 3 t2.medium EC2 instances

You need three instances: one control plane node and two worker nodes. All three must be t2.medium (2 vCPUs, 4 GB RAM) because kubeadm requires a minimum of 2 CPUs on the control plane.

  • Create a key pair for SSH access by running:
# Create a key pair and save it locally
aws ec2 create-key-pair \
  --key-name k8s-cluster-key \
  --query 'KeyMaterial' \
  --output text > k8s-cluster-key.pem

# Restrict permissions so SSH will accept the key
chmod 400 k8s-cluster-key.pem

What is a key pair?

A key pair is how you authenticate when connecting to EC2 instances via SSH. The private key (k8s-cluster-key.pem) stays on your machine. AWS stores the public half and installs it on your instances at launch. Without this file, you cannot SSH into your nodes.

  • Look up the latest Ubuntu 24.04 AMI ID for your region by running:
# Get the latest Ubuntu 24.04 AMI ID from Canonical's SSM parameter
AMI_ID=$(aws ssm get-parameters \
  --names /aws/service/canonical/ubuntu/server/24.04/stable/current/amd64/hvm/ebs-gp3/ami-id \
  --query 'Parameters[0].Value' \
  --output text)

echo $AMI_ID

What is an AMI?

An AMI (Amazon Machine Image) is a template that contains the operating system and software your instance boots with. Canonical publishes official Ubuntu AMI IDs through AWS Systems Manager parameters, so you always get the latest patched image for your region.

  • Launch all three instances by running this command:
# Launch 3 instances: control-plane, worker-1, worker-2
INSTANCE_IDS=$(aws ec2 run-instances \
  --image-id $AMI_ID \
  --instance-type t2.medium \
  --key-name k8s-cluster-key \
  --security-group-ids $SG_ID \
  --count 3 \
  --query 'Instances[*].InstanceId' \
  --output text)

echo $INSTANCE_IDS
  • Tag each instance with a descriptive name by running these commands:
# Split the instance IDs into individual variables
CP_ID=$(echo $INSTANCE_IDS | awk '{print $1}')
W1_ID=$(echo $INSTANCE_IDS | awk '{print $2}')
W2_ID=$(echo $INSTANCE_IDS | awk '{print $3}')

# Tag each instance with its role name
aws ec2 create-tags --resources $CP_ID --tags Key=Name,Value=control-plane
aws ec2 create-tags --resources $W1_ID --tags Key=Name,Value=worker-1
aws ec2 create-tags --resources $W2_ID --tags Key=Name,Value=worker-2
  • Wait for all instances to reach the running state by running:
# Wait until all three instances are running
aws ec2 wait instance-running --instance-ids $CP_ID $W1_ID $W2_ID
echo "All instances are running."
  • Get the public IP addresses of your instances by running:
# Retrieve public IPs for SSH access
aws ec2 describe-instances \
  --instance-ids $CP_ID $W1_ID $W2_ID \
  --query 'Reservations[*].Instances[*].[Tags[?Key==`Name`].Value | [0], PublicIpAddress]' \
  --output table

You should see a table showing the name and public IP of each instance. Write down all three IPs. You will use them to SSH into each node.

No public IP showing for an instance?

If PublicIpAddress is blank, your default VPC subnet may not auto-assign public IPs. You can assign an Elastic IP, or re-launch the instances with --associate-public-ip-address added to the run-instances command.

Help me fix my missing public IP addresses

Set unique hostnames on each instance

Kubernetes uses hostnames to identify nodes in the cluster. Each node must have a unique hostname, so you will SSH into each instance and set it using hostnamectl.

  • SSH into the control-plane instance by running:
ssh -i k8s-cluster-key.pem ubuntu@[[CP_IP="your control-plane public IP"]]

What if SSH asks about a fingerprint?

The first time you connect, SSH will ask you to verify the host fingerprint. Type yes and press Enter to continue. This is normal for first-time connections to a new server.

  • Set the hostname for the control-plane node by running:
# Set the hostname to control-plane
sudo hostnamectl set-hostname control-plane

# Verify it worked
hostname

You should see control-plane printed in the terminal output.

  • Exit the SSH session by running:
exit
  • SSH into worker-1 by running:
ssh -i k8s-cluster-key.pem ubuntu@[[W1_IP="your worker-1 public IP"]]
  • Set the hostname for worker-1 by running:
# Set the hostname to worker-1
sudo hostnamectl set-hostname worker-1
hostname

You should see worker-1 printed.

  • Exit and SSH into worker-2 by running:
exit
ssh -i k8s-cluster-key.pem ubuntu@[[W2_IP="your worker-2 public IP"]]
  • Set the hostname for worker-2 by running:
# Set the hostname to worker-2
sudo hostnamectl set-hostname worker-2
hostname

You should see worker-2 printed.

  • Exit this session by running:
exit

Connection timed out when SSH-ing?

If SSH hangs or times out, your security group SSH rule may not include your current IP. Run curl https://checkip.amazonaws.com and compare the result to the CIDR in your security group's port 22 rule. If your IP has changed (common with home internet), update the rule.

Help me troubleshoot my SSH connection timeout

Your AWS infrastructure is ready. You have three EC2 instances networked together with exactly the right firewall rules for Kubernetes. Next up, you will install the container runtime on all three nodes.

Install containerd and Configure Prerequisites

Your EC2 instances are running and accessible via SSH. Before you can bootstrap a Kubernetes cluster, every node needs a container runtime and specific Linux networking settings.

In this step, you will prepare all three nodes with the kernel modules, network settings, and containerd runtime that Kubernetes requires to function.

In this step, get ready to:
  • Load kernel modules and configure sysctl settings for Kubernetes networking.
  • Install and configure containerd as the container runtime.
  • Disable swap on all nodes.
Load kernel modules and configure sysctl for Kubernetes networking

Kubernetes uses Linux bridge networking to route traffic between pods. Two kernel modules and three sysctl settings must be enabled on every node for this to work.

  • SSH into your control-plane node.
  • Create the kernel modules configuration file by running this command:
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
  • Load both modules immediately by running these commands:
sudo modprobe overlay
sudo modprobe br_netfilter

What do these modules do?

The overlay module enables overlay filesystems, which containerd uses for container image layers. The br_netfilter module allows iptables to process bridged network traffic.

  • Create the sysctl configuration file for Kubernetes networking by running this command:
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
  • Apply the sysctl settings without rebooting by running:
sudo sysctl --system

What do these sysctl settings do?

The three settings tell Linux to apply firewall rules to bridged IPv4 and IPv6 traffic, and to forward IP packets between network interfaces. Without these, pods on different nodes cannot communicate with each other.

  • Verify the modules are loaded by running:
lsmod | grep br_netfilter
lsmod | grep overlay

You should see output showing both br_netfilter and overlay listed with their size and usage count.

Not seeing any output?

If lsmod returns nothing, the modprobe commands may not have run with sudo. Re-run them with sudo prepended.

Help me troubleshoot why my kernel modules are not loading on my Ubuntu EC2 instance.

  • Now repeat all of the above commands on your worker-1 and worker-2 nodes. SSH into each one and run the same set of commands.

Keep multiple terminal tabs open

Every command in this step must be run on all 3 nodes unless stated otherwise. Open multiple terminal tabs or windows to keep SSH sessions active to each node simultaneously.

Install containerd from the Docker repository

Containerd is the container runtime that Kubernetes uses to pull images and run containers. You will install it from the official Docker apt repository, which provides the containerd.io package.

  • On your control-plane node, install prerequisite packages and add Docker's GPG key by running these commands:
# Install prerequisite packages
sudo apt-get update
sudo apt-get install -y ca-certificates curl

# Add Docker's official GPG key
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
  • Add the Docker repository to your apt sources by running:
# Add the repository to apt sources
sudo tee /etc/apt/sources.list.d/docker.sources <<EOF
Types: deb
URIs: https://download.docker.com/linux/ubuntu
Suites: $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}")
Components: stable
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/docker.asc
EOF

sudo apt-get update

What does this do?

These commands add Docker's official package repository to your system. The GPG key verifies that packages you download are genuinely from Docker. The docker.sources file tells apt where to find the containerd package for your Ubuntu version.

  • Install containerd by running:
sudo apt-get install -y containerd.io
  • Generate the default containerd configuration file by running:
sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml
  • Enable the systemd cgroup driver by updating the configuration:
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml

Why enable the systemd cgroup driver?

Kubernetes and containerd both need to agree on how to manage control groups (cgroups) for containers. Since Ubuntu 24.04 uses systemd as its init system, setting SystemdCgroup = true ensures containerd delegates cgroup management to systemd.

This prevents resource tracking conflicts between kubelet and the container runtime.

  • Restart containerd to apply the new configuration by running:
sudo systemctl restart containerd
  • Verify containerd is running by checking its status:
sudo systemctl status containerd

You should see active (running) in the output. Press q to exit the status view.

Containerd not running?

If the status shows failed, there may be a syntax issue in the config file. Regenerate the default config and re-apply the sed command to fix it.

Help me fix my containerd configuration on Ubuntu 24.04.

  • Repeat the entire containerd installation process on your worker-1 and worker-2 nodes.
Disable swap on all nodes

Kubelet (the Kubernetes node agent) refuses to start if swap is enabled. Swap interferes with Kubernetes memory management and pod scheduling guarantees.

  • Disable swap immediately on your control-plane node by running:
sudo swapoff -a
  • Prevent swap from re-enabling on reboot by commenting out any swap entries in /etc/fstab:
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

What do these commands do?

The first command immediately turns off swap. The second command edits /etc/fstab to comment out any line containing swap, ensuring swap stays off after a reboot.

  • Verify swap is disabled by running:
free -h

The Swap row should show all zeros (0B total, 0B used, 0B free).

  • Repeat the swap disable commands on worker-1 and worker-2.

Swap still showing values?

Some Ubuntu AMIs use a swap file at /swapfile instead of a swap partition. Run cat /etc/fstab to check. If you see an uncommented swap line, manually edit the file with sudo nano /etc/fstab and add # to the beginning of that line.

Help me fully disable swap on my Ubuntu EC2 instance.

All three nodes now have the networking prerequisites, containerd runtime, and swap disabled. Next up, you will install kubeadm, kubelet, and kubectl to start building the actual cluster.

Install kubeadm, kubelet, and kubectl

Your nodes now have containerd ready to run containers. The next piece of the puzzle is installing the Kubernetes tooling that orchestrates everything.

In this step, you will add the official Kubernetes package repository and install the three core binaries that every cluster node needs: kubeadm (bootstraps the cluster), kubelet (runs pods on each node), and kubectl (your command-line interface to the cluster).

In this step, get ready to:
  • Add the Kubernetes apt repository for v1.35 on all 3 nodes.
  • Install kubeadm, kubelet, and kubectl on all 3 nodes.
  • Pin package versions and verify the installation.
Add the Kubernetes apt repository

Before you can install Kubernetes packages, you need to tell Ubuntu where to find them. The official packages live at pkgs.k8s.io, with a separate repository for each minor version.

  • Switch to your control-plane SSH session.
  • Install the prerequisite packages for the apt repository by running this command:
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gpg
  • Download the signing key for the Kubernetes package repository by running:
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.35/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
  • Add the Kubernetes v1.35 apt repository by running:
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.35/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
  • Update the package index and install the three Kubernetes packages by running:
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl

What do these three packages do?

kubeadm is the bootstrapping tool. It handles initializing the control plane and generating join tokens for worker nodes.

kubelet is the agent that runs on every node. It watches for pod assignments from the API server and tells containerd to start or stop containers.

kubectl is the CLI you use to interact with the cluster. You send commands to the API server through kubectl.

  • Repeat the exact same set of commands on your worker-1 and worker-2 SSH sessions.

Speed up multi-node installs

You can run the commands on worker-1 and worker-2 in parallel since they are independent of each other. Open both SSH sessions side by side and paste the same commands into each.

Seeing a GPG error or package not found?

Make sure the /etc/apt/keyrings directory exists. On some systems you need to create it first before downloading the signing key:

sudo mkdir -p -m 755 /etc/apt/keyrings

If you see a 404 error on the repository URL, double check you typed v1.35 in both the key URL and the repository line.

help me troubleshoot a GPG or apt repository error when adding the Kubernetes repo on Ubuntu

Pin the package versions

Kubernetes upgrades require a careful, coordinated process. Pinning (holding) packages prevents apt from accidentally upgrading them when you run a general system update.

  • On your control-plane node, pin the three packages by running:
sudo apt-mark hold kubelet kubeadm kubectl

What does apt-mark hold do?

The hold flag tells apt to skip these packages during system-wide upgrades. This ensures your cluster stays on a consistent version until you explicitly choose to upgrade.

  • Enable the kubelet service so it starts automatically by running:
sudo systemctl enable --now kubelet
  • Repeat both commands on worker-1 and worker-2.

kubelet keeps restarting?

This is expected. The kubelet enters a crash loop until kubeadm tells it what to do. You will resolve this in the next step when you initialize the control plane.

help me understand why kubelet is crash-looping before kubeadm init

Verify the installations
  • On your control-plane node, check the kubeadm version by running:
kubeadm version

You should see output that includes v1.35 in the version string.

  • Check that the kubelet service is active by running:
sudo systemctl status kubelet

You should see the service listed as active or in a restart loop. Both are normal at this stage because kubelet has no cluster configuration yet.

  • Run the same two verification commands on worker-1 and worker-2 to confirm all nodes have the packages installed.

kubeadm version shows command not found?

This means the install did not complete successfully. Verify the repository was added correctly by running:

cat /etc/apt/sources.list.d/kubernetes.list

The output should contain v1.35 in the URL. If it does, re-run the install:

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl

help me fix kubeadm command not found after trying to install the Kubernetes packages

All three nodes now have the Kubernetes tooling installed and locked to the same version. Next up, you will use kubeadm to initialize the control plane and bring your cluster to life.

Initialize the Control Plane

Your three nodes are prepped with containerd, kubeadm, and kubelet. Now it is time to turn one of them into the brains of the cluster.

The control plane runs the Kubernetes API server, scheduler, controller manager, and etcd database. You will use kubeadm init to bootstrap all of these components on your control-plane node in a single command.

In this step, get ready to:
  • Run kubeadm init on the control-plane node to bootstrap the cluster.
  • Configure kubectl so you can manage the cluster from your user account.
  • Verify the control plane components are running.
Run kubeadm init on the control-plane node

The kubeadm init command does all the heavy lifting of setting up a Kubernetes control plane. It generates TLS certificates, creates static pod manifests, starts the API server, and outputs a join command that worker nodes will use later.

  • Switch back to your SSH session on the control-plane node.
  • Initialize the control plane by running this command:
sudo kubeadm init --pod-network-cidr=192.168.0.0/16

What does this command do?

The --pod-network-cidr=192.168.0.0/16 flag tells Kubernetes which IP range to assign to pods. Calico (the CNI plugin you will install in the next step) expects this exact CIDR by default.

Behind the scenes, kubeadm runs preflight checks, generates certificates, writes static pod manifests for etcd, kube-apiserver, kube-scheduler, and kube-controller-manager, then waits for them to start.

This command may take a few minutes while it pulls container images and starts the control plane components.

When it finishes, you should see a success message starting with Your Kubernetes control-plane has initialized successfully!

Save the join command

At the bottom of the output, you will see a kubeadm join command with a --token and --discovery-token-ca-cert-hash flag. Copy this entire command and save it somewhere safe. You will need it to join your worker nodes in a later step.

The token expires after 24 hours. If you need to regenerate it later, run sudo kubeadm token create --print-join-command on the control-plane node.

Seeing a timeout or preflight error?

If kubeadm reports a preflight error about swap, confirm you ran sudo swapoff -a in the previous step. Swap must be completely disabled for kubelet to start.

If kubeadm hangs at the "waiting for the control plane" stage, check that containerd is running with systemctl status containerd and that your security group allows TCP port 6443 between cluster nodes.

For kubelet-specific errors, check the logs with journalctl -xeu kubelet.

Help me troubleshoot my kubeadm init failure

Configure kubectl access

After kubeadm init completes, kubectl does not know how to reach the API server yet. You need to copy the admin kubeconfig file to your user's home directory so kubectl can authenticate.

  • Set up kubectl for your ubuntu user by running these commands:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

What do these commands do?

The admin.conf file is generated by kubeadm init and contains the cluster's CA certificate, API server address, and admin credentials. Copying it to ~/.kube/config makes kubectl use it by default.

The chown command ensures your non-root user owns the file, so you do not need sudo every time you run kubectl.

  • Verify kubectl can reach the API server by running:
kubectl cluster-info

You should see output showing the Kubernetes control plane and CoreDNS addresses.

Connection refused on localhost:8080?

This error means kubectl is not finding your config file. Make sure you ran the mkdir, cp, and chown commands above as your regular user, not as root.

Check that the file exists with ls -la ~/.kube/config. If it does not exist, re-run the three commands.

Help me fix kubectl connection refused

Verify the control plane is running

Now that kubectl is configured, check that your control-plane node is registered and that the system pods are running.

  • Check the node status by running:
kubectl get nodes

You should see your control-plane node listed with a status of NotReady. This is expected. The node will remain NotReady until you install a CNI network plugin in the next step.

  • Check the system pods by running:
kubectl get pods -n kube-system

What should I see?

You should see pods for etcd, kube-apiserver, kube-controller-manager, kube-scheduler, and kube-proxy in a Running state. These are the core control plane components.

The coredns pods will show as Pending. CoreDNS will not start until a network plugin (CNI) is installed. This is completely normal at this stage.

Seeing pods in CrashLoopBackOff?

If any control plane pod is crash-looping, check its logs with kubectl logs <pod-name> -n kube-system. Common causes include containerd not running or a cgroup driver mismatch.

Verify containerd is healthy with systemctl status containerd and confirm SystemdCgroup = true is set in /etc/containerd/config.toml.

Help me debug my kube-system pods

Your control plane is bootstrapped and running. Next up, you will install Calico to give your cluster a pod network, which will move the control-plane node to Ready status.

Install the Calico CNI Plugin

Your control plane is running, but it shows NotReady because there is no networking layer yet.

Kubernetes delegates all pod-to-pod networking to a Container Network Interface (CNI) plugin. Without one, pods cannot communicate and system components like CoreDNS refuse to start. In this step, you will install Calico to provide pod networking and make the control-plane node fully operational.

In this step, get ready to:
  • Install the Calico operator and custom resource definitions.
  • Wait for all Calico pods to become Ready.
  • Confirm the control-plane node status changes to Ready.
Install the Calico operator and custom resources

Calico uses the Tigera Operator to manage its lifecycle. You will apply three manifests: the CRDs that define Calico's API types, the operator itself, and a custom resource that tells the operator how to configure networking.

  • In your SSH session on the control-plane node, install the Calico custom resource definitions by running this command:
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.32.0/manifests/v1_crd_projectcalico_org.yaml

Why kubectl create instead of kubectl apply?

The Calico CRD bundle is very large. kubectl apply can exceed the request size limit for annotations. kubectl create bypasses this issue because it sends the resource directly without storing a last-applied annotation.

  • Install the Tigera Operator by running:
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.32.0/manifests/tigera-operator.yaml
  • Install the Calico custom resources to configure networking by running:
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.32.0/manifests/custom-resources.yaml

What do these manifests do?

The CRD manifest registers Calico's custom API types (like IPPool and FelixConfiguration) with the Kubernetes API server.

The operator manifest deploys a controller that watches for Calico custom resources and reconciles the actual networking state. The custom resources manifest tells the operator to create an IPPool using 192.168.0.0/16 as the pod CIDR. This matches the --pod-network-cidr you passed to kubeadm init earlier.

Seeing connection errors?

If you see a connection refused or timeout error, make sure you are running these commands on the control-plane node where kubectl is configured.

Also verify your security group allows outbound HTTPS traffic so the node can reach GitHub to download the manifests.

help me troubleshoot Calico manifest download errors on my EC2 instance

Wait for all Calico pods to become Ready

The operator now deploys Calico components into the calico-system namespace. This takes 1-2 minutes as container images are pulled and pods start.

  • Watch the Calico pods come online by running:
watch kubectl get pods -n calico-system

What should I see?

Wait until every pod in the output shows Running status and all containers in the READY column show the full count (e.g. 1/1). You will see pods like calico-node, calico-typha, and calico-kube-controllers.

Once all pods are Running and Ready, press Ctrl+C to stop the watch command.

Pods stuck in Pending or CrashLoopBackOff?

If pods stay in Pending, the operator may still be starting. Wait an extra minute and check again.

If you see CrashLoopBackOff, run kubectl logs -n calico-system <pod-name> to check the error. A common cause is the pod CIDR not matching between kubeadm init and the Calico custom resource.

help me debug Calico pods that are not starting properly

Verify the control-plane node is now Ready

With Calico running, the control-plane node should transition from NotReady to Ready. This confirms pod networking is operational.

  • Check the node status by running:
kubectl get nodes
  • You should see the control-plane node with status Ready.

What does Ready mean?

A node reports Ready when the kubelet confirms that the container runtime is healthy AND a CNI plugin is configured. Before Calico, the kubelet had no networking plugin to report, so it stayed NotReady.

Still seeing NotReady?

Return to the previous substep and confirm all Calico pods are fully Running. The node can take an extra 30 seconds to transition after all Calico pods are ready.

If it persists, run kubectl describe node control-plane and look for the Conditions section. The NetworkReady condition shows whether the CNI initialized successfully.

help me figure out why my control-plane node is still NotReady after Calico install

Your control-plane node is now fully operational with pod networking enabled. Next up, you will join the two worker nodes to complete your 3-node cluster.

Join the Worker Nodes

Your control plane is up and running with Calico handling pod networking. The last piece of the puzzle is connecting your two worker nodes to the cluster.

Worker nodes are where your actual application workloads run. The control plane schedules pods onto workers based on available resources. In this step, you'll run the kubeadm join command on each worker to register them with the cluster, then verify everything works by deploying a test pod.

In this step, get ready to:
  • Join worker-1 to the cluster using the saved kubeadm join command.
  • Join worker-2 to the cluster using the same command.
  • Verify all 3 nodes are Ready and deploy a test nginx pod.
Join worker-1 to the cluster

Each worker node needs to authenticate with the control plane and register itself. The kubeadm join command uses the bootstrap token and CA certificate hash you saved from kubeadm init to establish this trust securely.

  • Open a new terminal window and SSH into your worker-1 instance by running:
ssh -i [[PEMPATH="path to your .pem file"]] ubuntu@[[WORKER1_IP="worker-1 public IP address"]]
  • Run the kubeadm join command you saved from the kubeadm init output in Step 4. It follows this format:
sudo kubeadm join --token <token> <control-plane-host>:<control-plane-port> --discovery-token-ca-cert-hash sha256:<hash>

What does this command do?

The --token flag provides a temporary credential that lets the worker authenticate with the API server.

The --discovery-token-ca-cert-hash flag verifies that the control plane's CA certificate is legitimate. This prevents a man-in-the-middle attack during the join process.

  • Wait for the join process to complete. You should see output ending with a message confirming the node has joined the cluster and telling you to run kubectl get nodes on the control plane.

Seeing a token error?

Bootstrap tokens expire after 24 hours. If your token has expired, switch back to the control-plane SSH session and generate a new join command by running:

sudo kubeadm token create --print-join-command

Copy the fresh command and run it on the worker node instead.

help me fix a kubeadm join token error

Join worker-2 to the cluster
  • Open another terminal window and SSH into your worker-2 instance by running:
ssh -i [[PEMPATH="path to your .pem file"]] ubuntu@[[WORKER2_IP="worker-2 public IP address"]]
  • Run the same kubeadm join command you used for worker-1:
sudo kubeadm join --token <token> <control-plane-host>:<control-plane-port> --discovery-token-ca-cert-hash sha256:<hash>
  • Wait for the same success output confirming the node has joined the cluster.
Verify all nodes are Ready and deploy a test pod

Both workers have joined the cluster. Now you'll confirm all three nodes report Ready status and prove the cluster works by scheduling a real workload.

  • Switch back to your control-plane SSH session.
  • Check that all 3 nodes are Ready by running:
kubectl get nodes

You should see all three nodes listed with Ready status. The worker nodes will show <none> under the ROLES column.

Why do workers show no role?

Worker nodes display <none> under ROLES because kubeadm only labels the control-plane node with a role by default. The workers are still fully functional cluster members that receive scheduled pods.

Seeing NotReady on a worker?

Give it 30-60 seconds. The Calico agent needs time to initialize on the new node.

If it stays NotReady, check the kubelet logs on the affected worker by running sudo journalctl -u kubelet -f from that worker's SSH session. Also verify that your security group allows TCP port 10250 traffic between nodes.

help me debug a NotReady worker node

  • Deploy a test nginx pod to confirm the cluster can schedule workloads by running:
kubectl run nginx --image=nginx
  • Check that the pod is running and see which node it was scheduled to by running:
kubectl get pods -o wide

You should see the nginx pod with a Running status. The NODE column shows which worker the Kubernetes scheduler placed it on.

Why is the pod on a worker, not the control plane?

By default, kubeadm applies a taint to the control-plane node that prevents regular pods from being scheduled there. This keeps the control plane dedicated to cluster management. Your workloads land on worker nodes automatically.

Pod stuck in Pending or ContainerCreating?

If the pod shows Pending, it means no node is available for scheduling. Confirm all nodes show Ready with kubectl get nodes.

If it shows ContainerCreating for more than a minute, the worker may still be pulling the nginx image. Wait another 30 seconds and run kubectl get pods -o wide again.

help me debug a stuck pod on my cluster

Your 3-node Kubernetes cluster is fully operational. You have a control plane managing the cluster and two worker nodes ready to run any workload you throw at them. Next up in the Secret Mission, you'll deploy a multi-tier application to see how Kubernetes distributes pods across your workers.

Secret mission

Deploy a Multi-Tier Application Across Your Cluster

Your cluster is running, but it's empty. In a real production environment, you'd have multiple services talking to each other across your worker nodes. Deploy a frontend and backend to see Kubernetes scheduling in action.

Clean Up Your Resources

Clean Up Your Resources

Your 3 EC2 instances are running at a combined cost of approximately $0.14 per hour ($3.34 per day). Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Cost warning

Each t2.medium instance costs $0.0464/hour. With 3 instances running, you accumulate about $3.34/day in charges.

If you are done experimenting, terminate all instances now to avoid unexpected charges on your AWS bill.

Resources you used:

  • 3x EC2 instances (control-plane, worker-1, worker-2) running t2.medium with Ubuntu 24.04.
  • 1x Security Group (kubernetes-cluster-sg).

Keep everything running

No action needed. Choose this if you are still actively experimenting with your cluster today.

  • Your 3 EC2 instances will continue running and accruing charges at ~$0.14/hour total.
  • Your Kubernetes cluster remains fully operational. You can SSH back in and run kubectl commands at any time.
  • Remember to come back and terminate the instances when you are finished.

Pause - I'll come back to this later

Stop the EC2 instances to pause compute billing. EBS storage charges still apply at a few pennies per day. Your Kubernetes cluster state is preserved on disk, but Kubernetes components may need reconfiguration when you restart.

  • Stop all 3 instances by running this command from your local terminal (replace the instance IDs with your own):
aws ec2 stop-instances --instance-ids <instance-id-1> <instance-id-2> <instance-id-3>
  • You can find your instance IDs in the AWS Console under EC2 > Instances.
  • When you return, start the instances from the AWS Console and SSH back in. Note that public IP addresses change on restart unless you use Elastic IPs.
  • Kubernetes services like kubelet and containerd should auto-start, but you may need to verify cluster health with kubectl get nodes after restarting.

Delete - I don't want to use this again

Terminate all instances and delete the security group to completely remove every resource this project created.

  • Terminate all 3 EC2 instances by running this command from your local terminal (replace the instance IDs with your own):
aws ec2 terminate-instances --instance-ids <instance-id-1> <instance-id-2> <instance-id-3>
  • You can find your instance IDs in the AWS Console under EC2 > Instances.
  • Wait for all instances to reach the terminated state. This usually takes a minute or two.
  • Once the instances are terminated, delete the security group by running:
aws ec2 delete-security-group --group-name kubernetes-cluster-sg

Why wait before deleting the security group?

A security group cannot be deleted while it is still attached to running or shutting-down instances. Waiting for the instances to fully terminate ensures the deletion succeeds.

  • Verify everything is gone by checking the EC2 > Instances page in the AWS Console. You should see all 3 instances in the terminated state.

That's a Wrap!

That's a Wrap!

Nice work! You just built a real, multi-node Kubernetes cluster from scratch on AWS, the way infrastructure teams do it in production.

You've learned how to:

  • Bootstrapped a control plane using kubeadm init, configuring containerd as the container runtime and AWS security groups with the exact ports Kubernetes components need to communicate.
  • Installed Calico CNI to enable pod-to-pod networking across nodes, transforming the cluster from NotReady to fully operational.
  • Joined 2 worker nodes using kubeadm join with TLS bootstrap authentication and verified multi-node scheduling by deploying workloads.
  • Deployed a multi-tier application with frontend and backend Services, scaled replicas, and observed Kubernetes distribute pods across your worker nodes.

Ready to quiz yourself?