Distributed Load Testing with Kubernetes

Deploy a distributed Locust load-testing platform on a local Kubernetes cluster.

Introduction

30 Second Summary

Every time you shop online during a flash sale or stream a live event, thousands of people are hitting the same servers at once. Companies need to know their systems can handle that rush before it happens, and they find out by simulating the traffic with load-testing tools.

In this project, you will build a distributed load-testing platform on your own machine using Kubernetes. You will deploy a target web application, launch a swarm of test workers against it, and watch Kubernetes automatically spin up more workers when CPU demand spikes.

What You'll Build

Picture opening a browser dashboard, pressing Start, and watching a live chart of requests-per-second climb as Kubernetes detects the strain and spins up new worker containers on the fly.

By the end of this project, you'll have:

  • A local Kubernetes cluster you control with real kubectl and AWS CLI commands, powered by Floci EKS running entirely on your machine.
  • A distributed Locust load-testing system where you press Start in a browser dashboard and watch a master node coordinate worker pods to flood a target web server with HTTP traffic.
  • A Horizontal Pod Autoscaler that detects rising CPU usage and automatically spins up new workers. You will watch the system break through the throughput ceiling you hit with a fixed worker count.
  • Secret Mission: Deploy a deliberately slow application and observe how your load-testing system detects and reports degraded performance.

Are there any prerequisites?

This project runs entirely on your local machine using Docker Desktop on Windows. No prior Kubernetes experience is required.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

Set Up Your Tools and Local Kubernetes Cluster

You have committed to building a distributed load-testing platform on Kubernetes. Before you can deploy anything, you need the command-line tools that talk to a Kubernetes cluster and a cluster to talk to.

In this step, you will install AWS CLI v2 and kubectl, then spin up a local Kubernetes cluster using Floci. Floci emulates AWS EKS on your machine, so you get a real Kubernetes experience without an AWS account or any cost.

In this step, get ready to:
  • Install AWS CLI v2 and kubectl on Windows.
  • Start Floci and create a local EKS cluster.
  • Explore the cluster to verify everything is running.
Install AWS CLI v2 and kubectl

Two command-line tools power your interaction with Kubernetes. AWS CLI v2 lets you create and manage EKS clusters. kubectl lets you deploy and inspect workloads inside those clusters.

  • Press the Windows key to open your search bar.
  • Type PowerShell and press Enter to open it.

Why PowerShell?

PowerShell is the modern terminal for Windows. All commands in this project use PowerShell syntax. Make sure you are not using the older Command Prompt (cmd.exe).

  • Install AWS CLI v2 by running this command:
irm https://awscli.amazonaws.com/v2/install.ps1 | iex

What does this command do?

irm downloads a PowerShell script from the AWS website. The pipe | sends it to iex, which executes it. This installs AWS CLI v2 system-wide on your machine.

The installation may take a minute or two. Wait until you see the PowerShell prompt return before continuing.

  • Verify the installation by running:
aws --version

✔️ I see a version number

AWS CLI is ready. Move on to installing kubectl below.

ⓧ Command not found

PowerShell has not picked up the new PATH yet. Close PowerShell completely and reopen it, then run aws --version again.

If it still fails, download the installer manually from the AWS CLI install page.

  • Install kubectl by running:
winget install -e --id Kubernetes.kubectl

What does this command do?

winget is the Windows Package Manager. This command downloads and installs the official Kubernetes CLI tool. The -e flag ensures an exact ID match so you get the right package.

  • Close PowerShell completely, then reopen it so the PATH updates.
  • Verify the installation by running:
kubectl version --client

✔️ I see a version number

kubectl is ready. You now have both CLI tools installed.

ⓧ Command not found

Close PowerShell one more time and reopen it. The winget installer updates the system PATH, but the current shell session does not pick up changes until it restarts.

If it still fails after reopening, check the official kubectl Windows install guide for alternative methods.

Seeing a different kubectl version than expected?

Docker Desktop for Windows adds its own version of kubectl to PATH. If kubectl version --client shows a version but it does not match what winget installed, both versions will work for this project. You can proceed with whichever version is active.

help me fix a kubectl version conflict between Docker Desktop and winget on Windows

Start Floci and create an EKS cluster

Floci is a local AWS emulator that runs inside a Docker container. It creates real Kubernetes clusters using k3s, so you can practice with kubectl and the AWS CLI without touching the cloud.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir load-testing-platform
cd load-testing-platform
  • Create the docker-compose.yml file by opening it in Notepad:
notepad docker-compose.yml
  • Notepad will ask if you want to create a new file. Click Yes.
  • Paste the following content into Notepad:
services:
  floci:
    image: floci/floci:latest
    ports:
      - "4566:4566"
    volumes:
      - //var/run/docker.sock:/var/run/docker.sock
      - ./data:/app/data

What does this file do?

This Docker Compose file defines a single service called floci. It pulls the floci/floci:latest image and exposes port 4566, which is the endpoint for all AWS API calls.

The first volume mounts the Docker socket so Floci can start new containers (like k3s clusters) on your behalf. The second volume persists data to a local ./data folder.

  • Save the file with Ctrl+S and close Notepad.
  • Confirm the file was created by running:
ls

You should see docker-compose.yml listed in the output.

✔️ Awesome, I've got everything!

Great. Make sure you have saved docker-compose.yml before continuing.

ⓧ I'd like to double check the full code

Here is the complete docker-compose.yml file. Compare it with yours character by character:

services:
  floci:
    image: floci/floci:latest
    ports:
      - "4566:4566"
    volumes:
      - //var/run/docker.sock:/var/run/docker.sock
      - ./data:/app/data
  • Start Floci by running:
docker compose up -d

You should see Docker pull the Floci image and start the container. This may take a minute on the first run.

Seeing a Docker error?

Make sure Docker Desktop is running. Check that the Docker icon appears in your system tray. If the container fails to start, try running docker compose down and then docker compose up -d again.

help me troubleshoot why docker compose up is failing for Floci on Windows

  • Configure your PowerShell session to point the AWS CLI at Floci by running these commands:
# Point AWS CLI at the local Floci endpoint
$env:AWS_ENDPOINT_URL = "http://localhost:4566"
$env:AWS_DEFAULT_REGION = "us-east-1"
$env:AWS_ACCESS_KEY_ID = "test"
$env:AWS_SECRET_ACCESS_KEY = "test"

What are these environment variables?

These tell the AWS CLI to send all requests to your local Floci emulator at http://localhost:4566 instead of the real AWS cloud. The access key and secret are dummy values. Floci accepts any non-empty credentials.

  • Create your EKS cluster by running:
aws eks create-cluster --name load-test-cluster --role-arn arn:aws:iam::000000000000:role/eks-role --resources-vpc-config subnetIds=subnet-00000001

What does this command do?

This tells Floci to create a local EKS cluster called load-test-cluster. Behind the scenes, Floci starts a k3s container that acts as a real Kubernetes cluster. The role ARN and subnet values are placeholders that Floci accepts without validation.

You should see a JSON response with cluster details. Give it a few seconds to finish spinning up the k3s container.

  • Connect kubectl to your new cluster by updating your kubeconfig:
aws eks update-kubeconfig --name load-test-cluster

You should see a message confirming the kubeconfig was updated.

Before you run the next command, what do you think you will see? Will the cluster have a node, and what state do you expect it to be in?

  • Verify your cluster is running by checking its nodes:
kubectl get nodes

You should see a single k3s node with a status of Ready. This confirms your local Kubernetes cluster is up and kubectl is connected to it.

Node not showing as Ready?

The k3s container may need a few seconds to finish starting. Wait 10-15 seconds and run kubectl get nodes again. If you see no nodes at all, verify that the update-kubeconfig step completed successfully and that your environment variables are still set in the same PowerShell window.

help me troubleshoot why kubectl get nodes is not showing a Ready node after creating a Floci EKS cluster

Explore the cluster

Your cluster is running, but what is actually inside it? Kubernetes comes with several system-level pods that keep the cluster healthy. Let's take a look.

  • View all pods running across every namespace by running:
kubectl get pods -A

What are these system pods?

You should see pods in the kube-system namespace. Three key ones to look for:

  • coredns handles DNS resolution inside the cluster so pods can find each other by name.
  • metrics-server collects CPU and memory usage from every pod. This is what the Horizontal Pod Autoscaler will use later to make scaling decisions.
  • traefik acts as the cluster's built-in ingress controller for routing external traffic.
  • Verify that metrics-server is collecting resource data by running:
kubectl top nodes

You should see a table with CPU and memory usage for your k3s node. This proves that metrics-server is active and feeding data to the cluster.

Seeing 'metrics not available yet'?

metrics-server needs 30-60 seconds after cluster creation to start collecting data. Wait a minute and try kubectl top nodes again. If it still fails, check that the metrics-server pod is running with kubectl get pods -A.

help me troubleshoot why kubectl top nodes is not returning data on my Floci EKS cluster

Your environment is ready. You have a running Kubernetes cluster, connected CLI tools, and a working metrics pipeline. Next up, you will deploy a target web application for your load testers to hammer.

Deploy a Target Web Application

In the previous step, you stood up a local Kubernetes cluster and verified it was healthy. Now you need to put something inside it.

Every load test needs a target. In this step, you will deploy a lightweight nginx web server into your cluster so that the Locust workers (coming in Step 3) have an application to hit with HTTP requests.

In this step, get ready to:
  • Deploy an nginx web server as a Kubernetes Deployment and Service.
  • Verify the application is reachable by port-forwarding it to your browser.
Create a Deployment and Service for nginx

Kubernetes needs two resources to run a web server and make it reachable inside the cluster.

A Deployment tells Kubernetes which container image to run and how many copies to keep alive. A Service gives those copies a stable internal address that other pods can reach.

  • Create a new file called target-app-deployment.yaml in the same folder as your docker-compose.yml.
  • Paste the following Deployment definition into the file:
# Deployment: runs one nginx container as the load-test target
apiVersion: apps/v1
kind: Deployment
metadata:
  name: target-app
spec:
  # Keep one replica running
  replicas: 1
  selector:
    matchLabels:
      app: target-app
  template:
    metadata:
      labels:
        app: target-app
    spec:
      containers:
        - name: nginx
          image: nginx:alpine
          ports:
            - containerPort: 80
          # Set CPU and memory boundaries for the container
          resources:
            requests:
              cpu: "100m"
              memory: "64Mi"
            limits:
              cpu: "500m"
              memory: "128Mi"

What does this Deployment do?

  • replicas: 1 tells Kubernetes to run exactly one copy of this container.
  • image: nginx:alpine pulls the lightweight Alpine-based nginx image from Docker Hub.
  • The resources block sets a CPU floor (100m = 10% of a core) and a ceiling (500m = 50% of a core). Kubernetes uses these numbers to decide where to place the pod and when to throttle it.
  • Below the Deployment definition, add a --- separator and the Service definition by pasting this at the bottom of the same file:
---
# Service: gives the Deployment a stable internal address
apiVersion: v1
kind: Service
metadata:
  name: target-app
spec:
  # Route traffic to pods labelled app: target-app
  selector:
    app: target-app
  ports:
    - port: 80
      targetPort: 80
  type: ClusterIP

What does this Service do?

  • type: ClusterIP creates an internal-only address. Other pods in the cluster can reach this Service, but nothing outside the cluster can.
  • The selector matches the app: target-app label on the Deployment's pods, so traffic sent to the Service lands on the right container.
  • Later, your Locust workers will send requests to http://target-app and Kubernetes will route them here.
  • Save target-app-deployment.yaml.
  • Apply both resources to your cluster by running this command:
kubectl apply -f target-app-deployment.yaml

You should see two lines of output confirming the Deployment and Service were created.

  • Verify the pod is running by running:
kubectl get pods

You should see a pod whose name starts with target-app- with a status of Running.

Pod not showing as Running?

  • If the status shows ContainerCreating, wait a few seconds for the image to download and run kubectl get pods again.
  • If the status shows ErrImagePull, make sure Docker Desktop is running and has internet access.
  • Check your YAML for indentation errors. YAML is whitespace-sensitive, so every level must use exactly two spaces.

Still stuck? Help me debug why my target-app pod is not running in Kubernetes.

✔️ Awesome, I've got everything!

Great. Make sure you have saved target-app-deployment.yaml and that your pod is in Running status.

ⓧ I'd like to double check the full code

Your target-app-deployment.yaml should contain two YAML documents separated by a --- line. Scroll up and confirm both sections are present in this order:

  • The Deployment (starts with apiVersion: apps/v1, kind: Deployment).
  • A --- separator line.
  • The Service (starts with apiVersion: v1, kind: Service).

If anything looks off, copy the chunks from above and re-run kubectl apply -f target-app-deployment.yaml.

Test the target app from your browser

Your nginx pod is running, but the ClusterIP Service only exposes it inside the cluster. To reach it from your browser, you need to use kubectl port-forward to tunnel traffic from your machine into the cluster.

  • Start the port-forward by running this command:
kubectl port-forward svc/target-app 8080:80

What does this command do?

This forwards port 8080 on your local machine to port 80 on the target-app Service. The command keeps running and holds the tunnel open, so your terminal will not return to the prompt.

When you are done testing, press Ctrl+C to stop the port-forward and get your terminal back.

Before you open the browser, what do you think you will see at http://localhost:8080?

  • Open your browser and go to http://localhost:8080.

You should see the nginx default welcome page with the heading "Welcome to nginx!" in your browser. This is the application your Locust workers will hammer with HTTP requests in the next step.

  • After confirming the page loads, return to your terminal and press Ctrl+C to stop the port-forward.

Page not loading in your browser?

  • Make sure the kubectl port-forward command is still running in your terminal. If you pressed Ctrl+C or closed the terminal, the tunnel stopped.
  • Confirm nothing else is using port 8080 on your machine. If something is, stop it or choose a different local port (e.g. 9090:80).
  • Check the URL is exactly http://localhost:8080 with http, not https.

Still stuck? Help me troubleshoot why kubectl port-forward is not working for my target-app Service.

Your target application is live inside the cluster. Next up, you will deploy Locust in distributed mode to start hammering it with traffic.

Deploy Locust in Distributed Mode

Your target application is live and serving responses inside the cluster. Now you need something to hit it with realistic traffic.

A single load generator hits a throughput ceiling fast. Locust solves this with a master/worker architecture where a master node coordinates the test while worker containers do the actual load generation.

In this step, get ready to:
  • Create a Locust test script and load it into Kubernetes as a ConfigMap.
  • Deploy a Locust master and worker pods in distributed mode.
  • Run your first load test and watch traffic flow through the Locust web UI.
Create a ConfigMap with your Locust test script

Locust needs a Python script that defines what HTTP requests the simulated users make. You could bake this script into a custom Docker image, but that means rebuilding the image every time the test changes.

A ConfigMap is a Kubernetes object that stores configuration data as key-value pairs. You can mount a ConfigMap into a container as a file, which means every Locust pod can share the same test script without a custom image.

  • Create a file called locustfile.py in your project folder (the same folder as docker-compose.yml) and paste the following test script:
from locust import HttpUser, task, between


# Define a simulated user that makes HTTP requests
class LoadTestUser(HttpUser):
    # Wait 0.5 to 1.5 seconds between each request
    wait_time = between(0.5, 1.5)

    # Each user repeatedly hits the homepage
    @task
    def get_homepage(self):
        self.client.get("/")

What does this script do?

The LoadTestUser class represents a single virtual user. It extends HttpUser, which gives it an HTTP client to make requests.

The @task decorator marks get_homepage as the action each user performs repeatedly. Each user sends a GET request to /, waits between 0.5 and 1.5 seconds (set by wait_time), then sends another.

  • Save locustfile.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved locustfile.py in the same folder as your other project files.

ⓧ I'd like to double check the full code

from locust import HttpUser, task, between


class LoadTestUser(HttpUser):
    wait_time = between(0.5, 1.5)

    @task
    def get_homepage(self):
        self.client.get("/")

Now you need to load this script into Kubernetes so the Locust pods can access it.

  • Create a ConfigMap from the file by running this command:
kubectl create configmap locust-script --from-file=locustfile.py
  • Verify the ConfigMap was created by running:
kubectl get configmaps

You should see locust-script listed in the output.

Don't see the ConfigMap?

  • Make sure you are running the command from the same directory where locustfile.py is saved. The --from-file flag looks for the file relative to your current directory.
  • If you see an error about the file not being found, run ls to confirm locustfile.py is in your current folder.
  • Help me troubleshoot my ConfigMap creation.
Deploy the Locust master and workers

In Locust's distributed mode, the master runs the web UI and coordinates workers. It does not generate any load itself. The workers connect to the master and run the actual simulated users.

Why the master/worker pattern?

A single process can only push so many HTTP requests before its CPU maxes out. By splitting load generation across multiple worker containers, each worker handles a slice of the total users. The master collects results from all workers and shows them in one dashboard.

This is the same pattern used in production load testing at scale. Kubernetes makes it easy because you can spin up more worker pods with a single replica count change.

  • Create a file called locust-master.yaml and paste the Deployment definition:
# Locust master - coordinates workers and hosts the web UI
apiVersion: apps/v1
kind: Deployment
metadata:
  name: locust-master
spec:
  replicas: 1
  selector:
    matchLabels:
      app: locust
      role: master
  template:
    metadata:
      labels:
        app: locust
        role: master
    spec:
      containers:
        - name: locust
          image: locustio/locust:latest
          args:
            - "-f"
            - "/mnt/locust/locustfile.py"
            - "--master"
          ports:
            - containerPort: 8089
            - containerPort: 5557
  • Directly below the ports section (still inside the container spec), add the volume mount, resource limits, and then the Service definition after the --- separator:
          # Mount the test script ConfigMap and set resource limits
          volumeMounts:
            - name: locust-script
              mountPath: /mnt/locust
          resources:
            requests:
              cpu: "100m"
              memory: "128Mi"
            limits:
              cpu: "500m"
              memory: "256Mi"
      volumes:
        - name: locust-script
          configMap:
            name: locust-script
---
# Service to expose the master's web UI and worker port
apiVersion: v1
kind: Service
metadata:
  name: locust-master
spec:
  selector:
    app: locust
    role: master
  ports:
    - name: web
      port: 8089
      targetPort: 8089
    - name: worker
      port: 5557
      targetPort: 5557
  type: ClusterIP

What does this file define?

The Deployment creates one master pod running locustio/locust:latest with the --master flag. It mounts the locust-script ConfigMap at /mnt/locust so the container can read locustfile.py from that path.

The Service exposes two ports: 8089 for the web dashboard and 5557 for worker communication. Workers use the Service name locust-master as a hostname to find the master.

  • Save locust-master.yaml.

✔️ Awesome, I've got everything!

Great. Scroll up and confirm your file has the Deployment definition (starting with apiVersion: apps/v1), followed by the volume mount and resource section, and then the Service definition (after ---). All three sections should be in locust-master.yaml.

ⓧ I'd like to double check the full code

Your locust-master.yaml file is too long to show in one block. Scroll up and confirm these sections appear in order:

  • The Deployment definition (kind: Deployment, container args with --master, ports 8089 and 5557).
  • The volume mount (volumeMounts, mountPath: /mnt/locust) and resource limits (CPU 100m/500m).
  • The Service definition after the --- separator (kind: Service, ports 8089 and 5557).
  • Confirm the file exists by running:
ls

You should see locust-master.yaml alongside your other project files.

Now create the worker Deployment. Workers do the actual load generation and connect to the master.

  • Create a file called locust-worker.yaml and paste the top portion of the Deployment:
# Locust workers - generate the actual HTTP load
apiVersion: apps/v1
kind: Deployment
metadata:
  name: locust-worker
spec:
  replicas: 2
  selector:
    matchLabels:
      app: locust
      role: worker
  template:
    metadata:
      labels:
        app: locust
        role: worker
    spec:
      containers:
        - name: locust
          image: locustio/locust:latest
          args:
            - "-f"
            - "/mnt/locust/locustfile.py"
            - "--worker"
            - "--master-host"
            - "locust-master"
  • Directly below the args section, add the volume mount and resource limits:
          # Mount the same test script and set CPU limits
          volumeMounts:
            - name: locust-script
              mountPath: /mnt/locust
          resources:
            requests:
              cpu: "200m"
              memory: "128Mi"
            limits:
              cpu: "400m"
              memory: "256Mi"
      volumes:
        - name: locust-script
          configMap:
            name: locust-script

How are workers different from the master?

Workers use the --worker and --master-host locust-master flags instead of --master. This tells each worker to connect to the master Service and wait for test instructions.

Notice the CPU request is 200m with a limit of 400m. These are intentionally tight limits. You will see these workers hit their CPU ceiling in the next step, which sets up the autoscaling problem.

  • Save locust-worker.yaml.

✔️ Awesome, I've got everything!

Great. Scroll up and confirm your file has the Deployment header, the container args (including --worker and --master-host), then the volume mount and resource limits.

ⓧ I'd like to double check the full code

Your locust-worker.yaml is too long to show in one block. Scroll up and confirm these two sections appear in order:

  • The Deployment definition (kind: Deployment, replicas: 2, container args with --worker and --master-host locust-master).
  • The volume mount (volumeMounts, mountPath: /mnt/locust) and resource limits (CPU 200m/400m).
  • Deploy both the master and worker resources by running:
kubectl apply -f locust-master.yaml -f locust-worker.yaml

Kubernetes creates the Deployments and Services defined in both files. Before you check the result, take a moment to predict.

How many pods do you expect from these two Deployments?

  • Check the pods by running:
kubectl get pods

You should see three new pods: one locust-master pod and two locust-worker pods, all in Running status. The target-app pod from the previous step is still there too.

Pods stuck in ContainerCreating or CrashLoopBackOff?

  • If pods are in ContainerCreating for more than a minute, Kubernetes may be pulling the locustio/locust:latest image. Wait a minute and run kubectl get pods again.
  • If a worker pod is in CrashLoopBackOff, check the master pod first. Workers cannot start without a running master. Run kubectl logs followed by the master pod name to see its output.
  • Confirm the ConfigMap name is locust-script by running kubectl get configmaps. If it does not match, workers will fail to mount the test script.
  • Help me debug my Locust pod issues.
Run your first load test

The Locust master is running inside the cluster, but its web UI is only accessible within the cluster network. You need to forward a port from your local machine to reach it.

  • If you still have a port-forward running from the previous step, press Ctrl+C in that terminal to stop it.
  • Forward the Locust web UI port by running:
kubectl port-forward svc/locust-master 8089:8089

What does this command do?

This forwards port 8089 on your local machine to port 8089 on the locust-master Service inside the cluster. The port-forward keeps running and occupies this terminal. You will need a separate terminal for any other commands.

  • Open your browser and go to http://localhost:8089.

You should see the Locust start page with fields for configuring the test.

  • In the Number of users field, enter 50.
  • In the Ramp Up (spawn rate) field, enter 10.
  • In the Host field, enter http://target-app.
  • Click Start.

Why is the host just http://target-app?

Inside the cluster, Kubernetes DNS resolves Service names to cluster IPs. Since both Locust and the target app run in the same namespace, workers can reach the nginx server simply by using the Service name target-app as the hostname.

Watch the live dashboard. You should see requests per second climbing as Locust ramps up to 50 users. The response times should be very low since nginx serves a static page.

Look at the worker count displayed on the dashboard. You should see 2 connected workers processing the requests.

Seeing 0 requests or connection errors?

  • Double check the Host field. It must be http://target-app (including http://). Without the protocol, Locust does not know how to connect.
  • Confirm the target-app pods are still running by opening a new terminal and running kubectl get pods. You should see the target-app pod in Running status.
  • If you see 0 workers connected, the worker pods may not have started. Check their status with kubectl get pods in a separate terminal.
  • Help me debug my Locust load test.

Your distributed load-testing system is live. You have a master coordinating 2 workers, all hammering your target app from inside the cluster. Next up, you will push these workers to their CPU limits and watch throughput plateau.

Hit the Scaling Ceiling

In the last step, you built a distributed load-testing system. A Locust master coordinates 2 worker pods, and you ran your first test with 50 virtual users hitting the target nginx app.

But 50 users is a warm-up. Each worker pod has a hard CPU limit of 400m, and you only have 2 of them. In this step, you will crank up the pressure to 200 virtual users and find out exactly where the system hits its ceiling.

In this step, get ready to:
  • Increase the load to 200 virtual users and observe the throughput plateau.
  • Confirm that the worker pods, not the target application, are the bottleneck.
Increase the load and observe the bottleneck

Your 2 worker pods each have a CPU limit of 400m. That was plenty for 50 virtual users. With 200 users hammering the target app, those same 2 workers will be doing four times the work.

Before you start, think about this: if you quadruple the number of virtual users from 50 to 200 but keep only 2 workers, what do you think happens to requests per second?

  • Return to the Locust web UI at http://localhost:8089.
  • If a test is still running from the previous step, stop it.
  • Start a new test with the number of users set to 200.
  • Set the spawn rate to 50.
  • Confirm the host is still set to http://target-app, then start the test.

Watch the Locust dashboard for about 30 seconds. Requests per second climbs at first as users ramp up, then it flattens out and stops growing.

What is happening?

Even though Locust keeps adding virtual users, the 2 worker pods can only process so many requests. Once they max out their CPU allocation, they physically cannot generate more load.

The requests/second line on the dashboard plateaus because the workers are CPU-bound. More simulated users just queue up, waiting for a worker thread that is already at full capacity.

Now let's see the proof from Kubernetes itself. You need a separate terminal because the port-forward from the previous step is still using one.

  • Press the Windows key, type PowerShell, and press Enter to open a new terminal window.
  • Check the CPU and memory usage of every pod by running this command:
kubectl top pods

You should see both locust-worker pods showing CPU usage close to their 400m limit. The numbers will fluctuate slightly, but both workers are clearly running hot.

Seeing an error from kubectl top?

  • If you see a "Metrics API not available" message, the metrics-server may still be starting. Wait 30 seconds and try again.
  • If the command returns no pods, make sure the Locust test is actively running in the browser before you check.

Ask for help: Why is kubectl top pods not showing my pod metrics?

Confirm the workers are the bottleneck

High CPU on the workers alone does not prove they are the bottleneck. If the target application were also maxed out, the problem could be on the server side instead. You need to compare both.

  • Run kubectl top pods again in your second terminal to get a fresh reading while the test is still running:
kubectl top pods
  • Compare the CPU column for the locust-worker pods versus the target-app pod.

The worker pods are near their 400m CPU limit. The target-app pod is using far less CPU, with plenty of headroom under its 500m limit. The bottleneck is clearly the workers, not the target.

Now switch back to the Locust web UI. Look at the requests/second chart. Even with 200 virtual users active, the throughput has plateaued at the same level it reached a minute ago.

Why does identifying the bottleneck matter?

Pinpointing the bottleneck is the core skill of performance testing. If the target app were maxed out, that would mean the app needs optimization or more replicas. But the target is fine.

The workers are the ones hitting their ceiling. The fix is not to tune the application. The fix is to add more workers so the load-generation side can keep up.

You have proven that your 2 workers are the bottleneck. They are maxed out on CPU, and no amount of extra virtual users can push more traffic through them. In the next step, you will tell Kubernetes to automatically spin up more workers when CPU demand rises.

Add Autoscaling with HPA

In the last step, you proved that two worker pods hit their CPU ceiling and throughput stopped growing. The bottleneck is the workers, not the target app.

Kubernetes can fix this automatically. The Horizontal Pod Autoscaler (HPA) monitors CPU usage across your worker pods and adds more replicas when usage climbs above a threshold you define. In this step, you will configure an HPA and watch it solve the bottleneck in real time.

In this step, get ready to:
  • Create a HorizontalPodAutoscaler targeting the locust-worker Deployment.
  • Re-run the load test and watch new worker pods spin up automatically.
  • Observe workers scaling back down after stopping the test.
Create a HorizontalPodAutoscaler for the worker Deployment

The HPA needs three pieces of information: which Deployment to scale, the minimum and maximum replica counts, and the CPU threshold that triggers scaling.

  • Create a new file named locust-hpa.yaml in the same project folder as your other YAML files and paste the following code:
# Defines an autoscaler that watches locust-worker CPU usage
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: locust-worker
spec:
  # Target the locust-worker Deployment for scaling
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: locust-worker
  # Scale between 2 and 8 worker replicas
  minReplicas: 2
  maxReplicas: 8
  # Scale up when average CPU exceeds 50% of the requested CPU
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50

What does this YAML do?

  • scaleTargetRef points the HPA at the locust-worker Deployment. This is the Deployment whose replica count the HPA controls.
  • minReplicas: 2 and maxReplicas: 8 set the scaling boundaries. The HPA will never drop below 2 workers or exceed 8.
  • averageUtilization: 50 means the HPA adds replicas whenever the average CPU usage across all worker pods exceeds 50% of their requested CPU.
  • Save locust-hpa.yaml.
  • Apply the HPA to your cluster by running this command:
kubectl apply -f locust-hpa.yaml
  • Verify the HPA is active by running:
kubectl get hpa

You should see a row for locust-worker showing the current CPU percentage, the 50% target, and replica counts of 2 (min) to 8 (max).

Seeing <unknown> for CPU?

The metrics-server may need a minute to collect data from the worker pods. Wait 30-60 seconds and run kubectl get hpa again. If it still shows <unknown>, confirm that your worker pods have CPU resource requests defined in locust-worker.yaml. HPA needs resource requests to calculate utilization percentages.

Help me debug why my HPA shows unknown CPU

✔️ Awesome, I've got everything!

Great. Double check you have saved locust-hpa.yaml and applied it to the cluster.

ⓧ I'd like to double check the full code

Here is the complete locust-hpa.yaml file. Compare your file against it:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: locust-worker
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: locust-worker
  minReplicas: 2
  maxReplicas: 8
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50
Re-run the load test and watch autoscaling in action

The HPA is watching. Now you need to give it something to react to by generating the same 200-user load that plateaued in the last step.

  • Return to the Locust web UI at http://localhost:8089.
  • Set the host to http://target-app.
  • Set the number of users to 200 and the spawn rate to 50.

Before you start the test, think about this: last time, throughput plateaued because the two workers were CPU-bound. What do you think will happen differently now that the HPA is watching?

  • Click Start to begin the load test.
  • Open a new terminal window. Watch the HPA react to rising CPU by running:
kubectl get hpa -w

You should see the CPU percentage climb above 50% and the REPLICAS column increase as the HPA spins up new worker pods.

  • Open another terminal window and watch the pods appear in real time by running:
kubectl get pods -w

You should see new locust-worker pods appear with a status of ContainerCreating and then transition to Running. Each new pod connects to the Locust master as an additional worker.

What is happening behind the scenes?

Every 15 seconds, the HPA queries the metrics-server for the average CPU utilization across all locust-worker pods. When the average exceeds 50%, the HPA calculates how many replicas are needed to bring utilization back to the target.

Kubernetes then updates the Deployment's replica count. The Deployment creates new pods, and each new pod automatically connects to the Locust master on port 5557.

  • Switch back to the Locust web UI in your browser.

You should see the number of connected workers increase on the dashboard. Requests per second should climb past the ceiling you saw in the last step.

  • Confirm CPU is now spread across more workers by running this in a terminal:
kubectl top pods

You should see more than 2 locust-worker pods listed, each at a healthier CPU utilization instead of all of them being maxed out.

Still seeing only 2 workers?

The HPA may need a minute to detect the CPU spike and react. Run kubectl get hpa and check the TARGETS column. If it shows a percentage above 50%, scaling is about to happen. If it shows a low percentage, the load test may not be generating enough CPU pressure. Make sure you have 200 users configured.

Help me debug why my HPA is not scaling up workers

Watch scale-down after stopping the test

Autoscaling is not just about scaling up. When load drops, the HPA should scale workers back down to avoid wasting resources.

  • Return to the Locust web UI and click Stop to end the load test.
  • Switch to the terminal where you ran kubectl get hpa -w earlier, or start a new watch by running:
kubectl get hpa -w

What do you think will happen to the replica count now that CPU usage is dropping?

Over the next few minutes, you should see the CPU percentage drop and the REPLICAS column decrease as the HPA removes worker pods that are no longer needed.

Why does scaling down take so long?

The HPA uses a default stabilization window of 300 seconds (5 minutes) before scaling down. This prevents the cluster from thrashing between adding and removing pods during short load dips.

The scale-up direction does not have this delay, which is why new pods appeared quickly when load increased. The asymmetry is intentional. Scaling up fast protects your service. Scaling down slowly protects against false calm.

  • Once the REPLICAS column shows 2 again, confirm the final state of the HPA by pressing Ctrl+C to stop the watch, then running:
kubectl get hpa

You should see the locust-worker HPA with a low CPU percentage, a target of 50%, and replicas back to 2. You have now seen the complete autoscaling lifecycle: scale up under load, then scale back down when the pressure eases.

Workers not scaling down?

Make sure you actually stopped the test in the Locust web UI. If the test is still running, workers will stay busy. Also, the 5-minute stabilization window means you need to wait at least 5 minutes after stopping the test before the HPA begins removing pods.

Help me understand why my HPA is not scaling down

You have built a complete autoscaling system. Your workers scale up to meet demand and scale back down when things quiet down. Up next, the Secret Mission challenges you to test how this system handles a very different kind of workload.

Secret mission

Test Against a Slow Application

Your load-testing platform works great against fast nginx. But real-world applications are not always fast. Can you deploy a deliberately slow application and use your distributed testing system to detect the performance difference?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely on your local machine using Docker containers, so there are no ongoing cloud costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Floci Docker container running a local EKS/k3s cluster (all Kubernetes Deployments, Services, ConfigMaps, and the HPA live inside it).
  • Local project files on your Desktop: docker-compose.yml, target-app-deployment.yaml, locust-master.yaml, locust-worker.yaml, locust-hpa.yaml, locustfile.py, and the data/ directory created by the Docker volume mount.
  • AWS CLI v2 and kubectl installed on your system.

Keep everything running

No action needed. Choose this if you are still actively building or want to experiment further with your Kubernetes cluster.

  • Your Floci container continues running in Docker Desktop, keeping your entire cluster and all Kubernetes resources alive.
  • AWS CLI v2 and kubectl remain installed on your machine and are available for other projects.
  • Your project files stay on your Desktop for reference.

Pause - I'll come back to this later

Shut down the Floci container to free up memory and CPU. Your cluster state is preserved so you can pick up where you left off.

  • Open a terminal and navigate to the folder containing your docker-compose.yml.
  • Stop the Floci container by running:
docker compose stop
  • When you are ready to resume, run docker compose start from the same folder to bring your cluster back.

Delete - I don't want to use this again

Remove the Floci container, all cluster data, and your local project files. This is a full teardown.

  • Stop any running port-forward sessions by pressing Ctrl+C in each terminal that has one active.
  • Open a terminal and navigate to the folder containing your docker-compose.yml on the Desktop.
  • Remove the Floci container and all its volumes by running:
docker compose down -v

This removes the Floci container, the k3s cluster, and every Kubernetes resource inside it (Deployments, Services, ConfigMaps, and the HPA).

  • Delete your local project folder from the Desktop. Open File Explorer, navigate to your Desktop, right-click the project folder, and select Delete.

This removes docker-compose.yml, all the YAML manifests, locustfile.py, and the data/ directory.

  • To uninstall kubectl, open PowerShell and run:
winget uninstall --id Kubernetes.kubectl
  • To uninstall AWS CLI v2, open Windows Settings.
  • Select Apps, then Installed apps.
  • Search for AWS Command Line Interface, click the three dots next to it, and select Uninstall.

Nice Work!

Nice Work!

Nice work! You just built a fully local distributed load-testing platform on Kubernetes, watched it hit a scaling ceiling, and then fixed it with autoscaling.

You've learned how to:

  • Deploy a distributed Locust load-testing system on a local Floci EKS cluster, using Deployments, Services, and ConfigMaps to wire a master node to worker pods that generate HTTP traffic against a target application.
  • Hit a real throughput ceiling with fixed workers, then configure a Horizontal Pod Autoscaler (HPA) that automatically scaled workers from 2 to 8 replicas based on CPU metrics from metrics-server.
  • Monitor autoscaling in real time using kubectl top, kubectl get hpa, and the Locust web dashboard to watch pods spin up, CPU spread across workers, and requests per second climb past the previous ceiling.
  • Secret Mission: Deployed a deliberately slow target application and compared Locust performance metrics against the fast nginx baseline to see how the HPA behaves differently with I/O-bound versus CPU-bound workers.

Ready to quiz yourself?