Build a Distributed Load Testing Platform

Introduction

30 Second Summary

When companies need to know if their servers can handle a surge of users, they run load tests. These tests simulate hundreds or thousands of people hitting a website at the same time to find breaking points before real users do.

In this project, you will build a distributed load testing platform from scratch. Multiple worker processes will generate HTTP traffic against a target server while a central controller aggregates performance metrics in real time.

What You'll Build

You will build a three-component distributed system: a target HTTP server to test against, a controller that orchestrates the load test, and workers that generate traffic and report results back.

By the end of this project, you'll have:

• A target server built with Flask that simulates fast, slow, and random-latency endpoints.
• A controller that uses ZeroMQ to broadcast commands to workers and collect timing data for live latency percentile computation (p50, p95, p99).
• A fully containerized platform using Docker Compose that runs the full stack and scales workers horizontally with a single command.
• Secret Mission: Add a ramp-up load profile that gradually increases virtual users over time to mimic real-world traffic patterns.

Are there any prerequisites?

You will need basic Python knowledge (variables, functions, loops) and Docker Desktop installed on your machine.

Set Up the Project and Build the Target Server

Before your distributed workers can generate load, they need a target to hit. The target server simulates a real backend with endpoints that respond at different speeds.

In this step, you'll install the project's Python libraries and build a Flask server with fast, slow, and random-latency endpoints. This gives you a controllable service to throw traffic at once the workers are ready.

In this step, get ready to:

• Install the project's Python dependencies.
• Build a target HTTP server with configurable latency endpoints.
• Verify the server responds to requests locally.

Install the project dependencies

Your project needs three Python libraries: one for messaging between the controller and workers, one for the target web server, and one for making HTTP requests.

• Add the project's dependencies to requirements.txt by pasting the following code:

pyzmq==27.1.0
flask==3.1.3
requests==2.32.3

What are these libraries?

• pyzmq is the Python binding for ZeroMQ, a high-performance messaging library. The controller and workers will use it to communicate.
• flask is a lightweight web framework. You'll use it to build the target server that your workers will send requests to.
• requests is an HTTP client library. The workers will use it to make HTTP requests to the target server.

• Save requirements.txt.

✔️ Awesome, I've got everything!

Great. Make sure you've saved the file before moving on.

ⓧ I'd like to double check the full code

pyzmq==27.1.0
flask==3.1.3
requests==2.32.3

• Install the dependencies by running this command:

pip install -r requirements.txt

• Verify the installation by running this command:

python -c "import zmq; print(zmq.zmq_version())"

You should see a version number like 4.3.5 printed in your terminal. The exact number may differ depending on your system.

Command not found?

If pip is not recognized, try using pip3 instead. Some systems install Python 3's package manager under that name.

If python is not recognized for the verify step, try python3 instead.

Help me troubleshoot my Python installation.

Build the target server

The target server is the service your workers will bombard with HTTP requests during a load test. It needs endpoints that respond at different speeds so you can see how your metrics capture that variation.

• In target_server.py, paste the following code to create the Flask app and add the first two endpoints:

import time
import random
from flask import Flask, jsonify

# Create the Flask application
app = Flask(__name__)


@app.route("/fast")
def fast():
    # Simulate a fast response (~10ms)
    time.sleep(0.01)
    return jsonify({"endpoint": "fast", "delay_ms": 10})


@app.route("/slow")
def slow():
    # Simulate a slow response (~200ms)
    time.sleep(0.2)
    return jsonify({"endpoint": "slow", "delay_ms": 200})

What does this code do?

• app = Flask(__name__) creates a Flask web application.
• The /fast endpoint uses time.sleep(0.01) to pause for 10 milliseconds before responding. This simulates a quick API call.
• The /slow endpoint pauses for 200 milliseconds. During a load test, you'll see this endpoint pull your p95 and p99 latencies upward.

• Save target_server.py.

You won't see a change yet. The next chunk adds the remaining endpoints and the server runner.

• Below the slow() function in target_server.py, add the remaining endpoints and the server runner:

@app.route("/random")
def random_latency():
    # Simulate unpredictable latency between 10-500ms
    delay = random.uniform(0.01, 0.5)
    time.sleep(delay)
    return jsonify({"endpoint": "random", "delay_ms": round(delay * 1000, 1)})


@app.route("/health")
def health():
    # Simple health check for Docker and monitoring
    return jsonify({"status": "ok"})


if __name__ == "__main__":
    # Listen on all interfaces so Docker containers can reach the server
    app.run(host="0.0.0.0", port=5000)

What does this code do?

• The /random endpoint picks a delay between 10ms and 500ms each time it's called. This is the most realistic endpoint because real services have unpredictable response times.
• The /health endpoint returns a simple {"status": "ok"} response. Docker Compose will use this later to check that the server is ready before starting the other services.
• The if __name__ == "__main__" block starts the Flask server on port 5000, listening on all network interfaces so it's reachable from Docker containers.

• Save target_server.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before running the server.

ⓧ I'd like to double check the full code

import time
import random
from flask import Flask, jsonify

app = Flask(__name__)


@app.route("/fast")
def fast():
    time.sleep(0.01)
    return jsonify({"endpoint": "fast", "delay_ms": 10})


@app.route("/slow")
def slow():
    time.sleep(0.2)
    return jsonify({"endpoint": "slow", "delay_ms": 200})


@app.route("/random")
def random_latency():
    delay = random.uniform(0.01, 0.5)
    time.sleep(delay)
    return jsonify({"endpoint": "random", "delay_ms": round(delay * 1000, 1)})


@app.route("/health")
def health():
    return jsonify({"status": "ok"})


if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

• Start the target server by running this command:

python target_server.py

You should see Flask's startup message in your terminal, indicating the server is running on port 5000. The terminal will stay occupied because the server is a long-running process.

• Open a second terminal window.
• Navigate to your project folder in the new terminal.
• Test the fast endpoint by running this command:

curl http://localhost:5000/fast

You should see a JSON response like {"delay_ms":10,"endpoint":"fast"}. This confirms the target server is running and responding to requests.

Try the other endpoints too

Run curl http://localhost:5000/slow and curl http://localhost:5000/random to see the different response times. Notice how /random returns a different delay_ms value each time.

Connection refused?

Make sure the server is still running in your first terminal. If you see an error or the process stopped, re-run python target_server.py.

If port 5000 is already in use by another application, stop that application first and try again.

Help me troubleshoot my Flask server not responding.

• Once you've confirmed the server works, go back to the first terminal and stop the server by pressing Ctrl+C.

Your target server is ready. Next up, you'll build the controller that orchestrates the entire load test by broadcasting commands to workers and collecting their metrics in real time.

Build the Controller

Your target server is ready with three endpoints that simulate different latency profiles. Now you need something to orchestrate a load test against it.

The controller is the brain of the system. It broadcasts test configurations to workers, collects timing data from every request they make, and crunches that data into live latency percentiles and throughput stats. You will build it using ZeroMQ, the same messaging library that production tools like Locust use for distributed coordination.

In this step, get ready to:

• Set up two ZeroMQ sockets for broadcasting commands and collecting metrics.
• Implement a metrics aggregation loop that computes live p50, p95, and p99 latency percentiles.
• Add test lifecycle management with start/stop commands and a final summary report.

Create the controller with two ZeroMQ sockets

The controller needs two communication channels. One channel broadcasts commands (like "start testing" or "stop") to every worker at once. The other channel collects individual timing samples from all workers into one place.

Why two sockets instead of one?

ZeroMQ's PUB/SUB pattern sends each message to ALL connected subscribers. This is perfect for broadcasting commands because every worker needs to hear "start" and "stop".

ZeroMQ's PUSH/PULL pattern delivers each message to exactly one receiver with fair-queuing. This is ideal for collecting metrics because each sample goes to the controller once, no matter how many workers are sending.

You will also need a helper function to compute latency percentiles from the raw timing data. Percentiles tell you how fast your system is for most users (p50), for unlucky users (p95), and for the worst-case users (p99).

• Open controller.py in your editor. The file is currently empty.
• Add the imports and the percentile helper function by copying this code:

import zmq
import json
import time
import sys
import statistics


# Calculate a specific percentile from pre-sorted data
def compute_percentile(sorted_data, percentile):
    if not sorted_data:
        return 0
    # Use linear interpolation between data points
    k = (len(sorted_data) - 1) * (percentile / 100)
    f = int(k)
    c = f + 1
    if c >= len(sorted_data):
        return sorted_data[f]
    return sorted_data[f] + (k - f) * (sorted_data[c] - sorted_data[f])

What does this code do?

The imports bring in zmq for ZeroMQ messaging, json for serializing messages, time for tracking durations, sys for command-line arguments, and statistics for calculating averages.

The compute_percentile() function uses linear interpolation to find a specific percentile from sorted data. Passing 50 gives the median (p50), 95 gives the value below which 95% of samples fall (p95), and so on.

• Save controller.py.

You won't see any output yet. The next chunk adds the main controller function that uses this helper.

Now add the main controller function with the ZeroMQ socket setup. This function creates the two sockets and waits briefly for workers to connect before starting.

• Add the following code below the compute_percentile() function:

def run_controller(target_url, duration, vus_per_worker):
    # Create a ZeroMQ context to manage all sockets
    context = zmq.Context()

    # PUB socket broadcasts commands to all connected workers
    pub_socket = context.socket(zmq.PUB)
    pub_socket.bind("tcp://*:5555")

    # PULL socket collects metrics sent by workers
    pull_socket = context.socket(zmq.PULL)
    pull_socket.bind("tcp://*:5556")

    print(f"Controller started. Waiting 2 seconds for workers to connect...")
    time.sleep(2)


if __name__ == "__main__":
    target = sys.argv[1] if len(sys.argv) > 1 else "http://localhost:5000/random"
    dur = int(sys.argv[2]) if len(sys.argv) > 2 else 30
    vus = int(sys.argv[3]) if len(sys.argv) > 3 else 5
    run_controller(target, dur, vus)

What does this code do?

• zmq.Context() creates a ZeroMQ context that manages all sockets. Every ZeroMQ program starts with one.
• The PUB socket binds on port 5555 and sends commands to all workers. The PULL socket binds on port 5556 and receives metrics from workers.
• The 2-second sleep mitigates ZeroMQ's "slow joiner" problem. When a PUB socket starts sending before subscribers finish connecting, the subscribers miss those early messages. The brief pause ensures workers are ready.
• The __main__ block parses optional command-line arguments for the target URL, test duration, and virtual users per worker. It defaults to the /random endpoint, 30 seconds, and 5 VUs.

• Save controller.py.
• Verify the socket setup works by running this command:

python controller.py

You should see Controller started. Waiting 2 seconds for workers to connect... printed. After a 2-second pause, the script exits because there is no logic after the sleep yet.

Seeing a ModuleNotFoundError?

• Make sure you installed the project dependencies in Step 1. Run pip install pyzmq==27.1.0 to install ZeroMQ.
• If you are using a virtual environment, confirm it is activated before running the script.

Still stuck? Help me fix this ModuleNotFoundError for zmq in my controller script.

Implement the metrics aggregation loop

The controller needs a loop that continuously polls for incoming metrics from workers, tracks every latency sample, and prints a live stats dashboard every second. This is where the raw data from distributed workers gets turned into meaningful numbers.

• In controller.py, find this line inside run_controller():

    time.sleep(2)

• Add the following metrics aggregation loop directly below that line:

    # Initialize tracking for all collected metrics
    all_latencies = []
    total_requests = 0
    total_errors = 0
    workers_seen = set()
    start_time = time.time()
    last_print = start_time
    interval_latencies = []

    # Set up a poller to check for incoming messages without blocking forever
    poller = zmq.Poller()
    poller.register(pull_socket, zmq.POLLIN)

    # Main loop: collect metrics for the test duration plus a 2-second buffer
    while time.time() - start_time < duration + 2:
        socks = dict(poller.poll(timeout=500))
        if pull_socket in socks:
            message = pull_socket.recv_string()
            sample = json.loads(message)
            workers_seen.add(sample["worker_id"])
            total_requests += 1
            latency = sample["latency_ms"]
            all_latencies.append(latency)
            interval_latencies.append(latency)
            if sample["status_code"] >= 400:
                total_errors += 1

        # Print a live stats line every second
        now = time.time()
        if now - last_print >= 1.0 and interval_latencies:
            elapsed = now - start_time
            rps = total_requests / elapsed if elapsed > 0 else 0
            sorted_interval = sorted(interval_latencies)
            p50 = compute_percentile(sorted_interval, 50)
            p95 = compute_percentile(sorted_interval, 95)
            p99 = compute_percentile(sorted_interval, 99)
            error_rate = (total_errors / total_requests * 100) if total_requests > 0 else 0
            print(
                f"[{elapsed:5.1f}s] "
                f"RPS: {rps:6.1f} | "
                f"p50: {p50:6.1f}ms | "
                f"p95: {p95:6.1f}ms | "
                f"p99: {p99:6.1f}ms | "
                f"Errors: {error_rate:.1f}% | "
                f"Workers: {len(workers_seen)}"
            )
            interval_latencies = []
            last_print = now

What does this code do?

• all_latencies stores every sample for the final report. interval_latencies stores only samples since the last stats print, giving you a rolling window of recent performance.
• zmq.Poller() lets the controller check for incoming messages with a 500ms timeout. Without a poller, the socket would block indefinitely waiting for a message, and the stats-printing code would never execute.
• Each incoming sample contains a worker_id, latency_ms, and status_code. The loop tracks which workers have reported in using the workers_seen set.
• Every second, the loop sorts the interval samples, computes p50/p95/p99 using your compute_percentile() function, and prints a formatted stats line. Then it clears the interval for the next window.

• Save controller.py.

You won't see any stats output yet because no workers are sending metrics. The next substep adds the start/stop lifecycle that completes the controller.

Add test lifecycle management

The controller still needs to tell workers when to start generating load and when to stop. It also needs to print a final summary after the test ends, showing cumulative stats across the entire run.

The start command gets broadcast right after the 2-second wait (before the aggregation loop). The stop command and final report go after the loop finishes.

• In controller.py, find this line inside run_controller():

    time.sleep(2)

• Add the start config broadcast between the time.sleep(2) line and the # Initialize tracking comment you added in the previous substep:

    # Build and broadcast the test configuration to all workers
    config = {
        "command": "start",
        "target_url": target_url,
        "duration": duration,
        "vus": vus_per_worker,
    }
    pub_socket.send_string(json.dumps(config))
    print(f"Sent start command: {vus_per_worker} VUs/worker -> {target_url}")

What does this code do?

The config dictionary packages everything a worker needs to know: the target URL, how long to run, and how many virtual users to spawn. pub_socket.send_string() broadcasts this as a JSON string over the PUB socket, so every connected worker receives a copy.

• Save controller.py.

You won't see the start message in action yet because no workers are listening. The next chunk adds the stop command and final report.

Now add the stop broadcast, final summary, and socket cleanup. These go after the while loop you created in the previous substep.

• Find the end of the while loop (the last line should be last_print = now).
• Add the following code below the while loop, at the same indentation level as the loop itself (inside run_controller() but outside the while):

    # Broadcast stop command to all workers
    stop_msg = json.dumps({"command": "stop"})
    pub_socket.send_string(stop_msg)
    print("\nSent stop command.")

    # Print the final aggregated report
    print("\n=== Final Report ===")
    print(f"Total Requests:  {total_requests}")
    print(f"Total Errors:    {total_errors}")
    if all_latencies:
        sorted_all = sorted(all_latencies)
        print(f"p50 Latency:     {compute_percentile(sorted_all, 50):.1f}ms")
        print(f"p95 Latency:     {compute_percentile(sorted_all, 95):.1f}ms")
        print(f"p99 Latency:     {compute_percentile(sorted_all, 99):.1f}ms")
        print(f"Min Latency:     {min(all_latencies):.1f}ms")
        print(f"Max Latency:     {max(all_latencies):.1f}ms")
        print(f"Mean Latency:    {statistics.mean(all_latencies):.1f}ms")
    elapsed = time.time() - start_time
    print(f"Avg RPS:         {total_requests / elapsed:.1f}")
    print(f"Workers Used:    {len(workers_seen)}")
    print(f"Duration:        {elapsed:.1f}s")

    # Clean up ZeroMQ resources
    pub_socket.close()
    pull_socket.close()
    context.term()

What does this code do?

• The stop message is broadcast over the PUB socket, telling all workers to shut down their virtual users and exit.
• The final report computes percentiles from all_latencies (every sample from the entire test), giving you an overall view instead of the rolling 1-second windows from the live stats.
• context.term() cleans up the ZeroMQ context and releases the bound ports. Skipping this can leave ports occupied and block future runs.

• Save controller.py.
• Test the complete controller by running it with a short 5-second duration:

python controller.py http://localhost:5000/random 5 5

You should see Controller started. Waiting 2 seconds for workers to connect... followed by Sent start command: 5 VUs/worker. After about 7 seconds, you will see the final report with 0 total requests and 0 workers. This is expected because no workers are running yet.

Script hanging or port already in use?

• If you see an error about the address already being in use, a previous run may not have cleaned up properly. Wait a few seconds and try again, or use Ctrl+C to stop any lingering processes.
• Make sure the indentation of the stop/report code is inside run_controller() but outside the while loop. Python relies on indentation to determine scope.

Still stuck? Help me debug my controller.py. It's not printing the final report or it hangs indefinitely.

✔️ Awesome, I've got everything!

Great. Double check you have saved controller.py.

ⓧ I'd like to double check the full code

Here is the complete controller.py file. Compare it with yours and make sure everything matches.

import zmq
import json
import time
import sys
import statistics


def compute_percentile(sorted_data, percentile):
    if not sorted_data:
        return 0
    k = (len(sorted_data) - 1) * (percentile / 100)
    f = int(k)
    c = f + 1
    if c >= len(sorted_data):
        return sorted_data[f]
    return sorted_data[f] + (k - f) * (sorted_data[c] - sorted_data[f])


def run_controller(target_url, duration, vus_per_worker):
    context = zmq.Context()

    pub_socket = context.socket(zmq.PUB)
    pub_socket.bind("tcp://*:5555")

    pull_socket = context.socket(zmq.PULL)
    pull_socket.bind("tcp://*:5556")

    print(f"Controller started. Waiting 2 seconds for workers to connect...")
    time.sleep(2)

    config = {
        "command": "start",
        "target_url": target_url,
        "duration": duration,
        "vus": vus_per_worker,
    }
    pub_socket.send_string(json.dumps(config))
    print(f"Sent start command: {vus_per_worker} VUs/worker -> {target_url}")

    all_latencies = []
    total_requests = 0
    total_errors = 0
    workers_seen = set()
    start_time = time.time()
    last_print = start_time
    interval_latencies = []

    poller = zmq.Poller()
    poller.register(pull_socket, zmq.POLLIN)

    while time.time() - start_time < duration + 2:
        socks = dict(poller.poll(timeout=500))
        if pull_socket in socks:
            message = pull_socket.recv_string()
            sample = json.loads(message)
            workers_seen.add(sample["worker_id"])
            total_requests += 1
            latency = sample["latency_ms"]
            all_latencies.append(latency)
            interval_latencies.append(latency)
            if sample["status_code"] >= 400:
                total_errors += 1

        now = time.time()
        if now - last_print >= 1.0 and interval_latencies:
            elapsed = now - start_time
            rps = total_requests / elapsed if elapsed > 0 else 0
            sorted_interval = sorted(interval_latencies)
            p50 = compute_percentile(sorted_interval, 50)
            p95 = compute_percentile(sorted_interval, 95)
            p99 = compute_percentile(sorted_interval, 99)
            error_rate = (total_errors / total_requests * 100) if total_requests > 0 else 0
            print(
                f"[{elapsed:5.1f}s] "
                f"RPS: {rps:6.1f} | "
                f"p50: {p50:6.1f}ms | "
                f"p95: {p95:6.1f}ms | "
                f"p99: {p99:6.1f}ms | "
                f"Errors: {error_rate:.1f}% | "
                f"Workers: {len(workers_seen)}"
            )
            interval_latencies = []
            last_print = now

    stop_msg = json.dumps({"command": "stop"})
    pub_socket.send_string(stop_msg)
    print("\nSent stop command.")

    print("\n=== Final Report ===")
    print(f"Total Requests:  {total_requests}")
    print(f"Total Errors:    {total_errors}")
    if all_latencies:
        sorted_all = sorted(all_latencies)
        print(f"p50 Latency:     {compute_percentile(sorted_all, 50):.1f}ms")
        print(f"p95 Latency:     {compute_percentile(sorted_all, 95):.1f}ms")
        print(f"p99 Latency:     {compute_percentile(sorted_all, 99):.1f}ms")
        print(f"Min Latency:     {min(all_latencies):.1f}ms")
        print(f"Max Latency:     {max(all_latencies):.1f}ms")
        print(f"Mean Latency:    {statistics.mean(all_latencies):.1f}ms")
    elapsed = time.time() - start_time
    print(f"Avg RPS:         {total_requests / elapsed:.1f}")
    print(f"Workers Used:    {len(workers_seen)}")
    print(f"Duration:        {elapsed:.1f}s")

    pub_socket.close()
    pull_socket.close()
    context.term()


if __name__ == "__main__":
    target = sys.argv[1] if len(sys.argv) > 1 else "http://localhost:5000/random"
    dur = int(sys.argv[2]) if len(sys.argv) > 2 else 30
    vus = int(sys.argv[3]) if len(sys.argv) > 3 else 5
    run_controller(target, dur, vus)

Your controller is ready to orchestrate a load test. It can broadcast start/stop commands and aggregate live metrics. Next up, you will build the worker that actually generates HTTP traffic and feeds timing data back to this controller.

Build the Worker

In the previous step, you built the controller that broadcasts commands and collects metrics. Now you need something to actually generate the load.

The worker is the other half of the distributed system. It receives commands from the controller, spawns virtual users as threads, and streams timing data back for live aggregation.

In this step, get ready to:

• Write a virtual user function that makes HTTP requests and reports timing data.
• Build a worker process that connects to the controller over ZeroMQ.
• Add graceful shutdown so workers stop cleanly on command.

Write the virtual user function

Each virtual user is a Python thread that loops continuously, making HTTP requests to the target server. After every request, it measures how long the response took and pushes that timing data to the controller.

• Open worker.py in your editor.
• Add the imports and the virtual_user() function by pasting this code:

import zmq
import json
import time
import os
import threading
import requests


def virtual_user(worker_id, target_url, push_socket, stop_event, lock):
    # Loop until the stop_event is set by the main thread
    while not stop_event.is_set():
        start = time.time()
        try:
            # Make an HTTP GET request to the target URL
            resp = requests.get(target_url, timeout=5)
            latency_ms = (time.time() - start) * 1000
            status_code = resp.status_code
        except requests.RequestException:
            # Record failed requests with status code 0
            latency_ms = (time.time() - start) * 1000
            status_code = 0

        # Package the timing data as a JSON sample
        sample = json.dumps({
            "worker_id": worker_id,
            "timestamp": time.time(),
            "latency_ms": round(latency_ms, 2),
            "status_code": status_code,
        })
        # Use a lock to safely send on the shared PUSH socket
        with lock:
            push_socket.send_string(sample)

What does this code do?

Each virtual user runs in its own thread, looping until stop_event tells it to quit. On each loop iteration, it makes an HTTP GET request and measures the round-trip time in milliseconds.

The timing sample includes the worker_id so the controller can tell which worker sent it. The lock prevents multiple threads from writing to the same ZeroMQ socket at the same time, which would corrupt messages.

Why threading instead of asyncio?

Virtual users spend most of their time waiting for HTTP responses, not crunching numbers. This makes them I/O-bound, which is exactly where Python threads shine. Each thread can wait on a network response without blocking the others.

Threading is also conceptually simpler than asyncio. Each virtual user is a straightforward loop with no async/await ceremony.

• Save worker.py.

You won't see a change yet. The next chunk adds the worker process that actually calls this function.

Connect to the controller and handle commands

The controller uses PUB to broadcast and PULL to collect. The worker mirrors this with the opposite socket types: SUB to listen for commands and PUSH to send metrics.

• In worker.py, add the run_worker() function and the entry point below the virtual_user() function by pasting this code:

def run_worker(controller_host):
    # Read worker ID from environment or generate one from process ID
    worker_id = os.environ.get("WORKER_ID", f"worker-{os.getpid()}")
    context = zmq.Context()

    # SUB socket to receive commands from the controller
    sub_socket = context.socket(zmq.SUB)
    sub_socket.connect(f"tcp://{controller_host}:5555")
    sub_socket.setsockopt_string(zmq.SUBSCRIBE, "")

    # PUSH socket to send metrics to the controller
    push_socket = context.socket(zmq.PUSH)
    push_socket.connect(f"tcp://{controller_host}:5556")

    print(f"Worker {worker_id} connected to controller at {controller_host}")

    lock = threading.Lock()
    threads = []
    stop_event = threading.Event()

    # Listen for commands from the controller
    while True:
        message = sub_socket.recv_string()
        msg = json.loads(message)

        if msg["command"] == "start":
            target_url = msg["target_url"]
            vus = msg["vus"]
            print(f"Worker {worker_id}: Starting {vus} virtual users -> {target_url}")
            stop_event.clear()
            # Spawn virtual user threads
            for i in range(vus):
                t = threading.Thread(
                    target=virtual_user,
                    args=(worker_id, target_url, push_socket, stop_event, lock),
                    daemon=True,
                )
                t.start()
                threads.append(t)

        elif msg["command"] == "stop":
            print(f"Worker {worker_id}: Stopping...")
            # Signal all virtual users to stop
            stop_event.set()
            for t in threads:
                t.join(timeout=3)
            threads.clear()
            print(f"Worker {worker_id}: Stopped.")
            break

    # Clean up ZeroMQ resources
    sub_socket.close()
    push_socket.close()
    context.term()


if __name__ == "__main__":
    host = os.environ.get("CONTROLLER_HOST", "localhost")
    run_worker(host)

What does this code do?

• The worker reads WORKER_ID and CONTROLLER_HOST from environment variables. This lets you run multiple workers on the same machine with different IDs, and point them at any controller host.
• The SUB socket connects to port 5555 and subscribes to all messages with an empty string filter. This means it receives every command the controller publishes.
• The PUSH socket connects to port 5556 to send timing samples back. This is the return channel for metrics.
• On a "start" command, the worker spawns N threads (one per virtual user). Each thread runs the virtual_user() function you wrote in the previous chunk.
• On a "stop" command, it sets the stop_event flag, which signals all virtual user threads to finish their current request and exit. The join(timeout=3) call waits up to 3 seconds for each thread to shut down.

Why does the SUB socket subscribe with an empty string?

ZeroMQ SUB sockets filter messages by prefix. Setting the subscription to an empty string ("") means "match everything." Without this call, the SUB socket would receive nothing at all.

• Save worker.py.
• Verify the file has no syntax errors by running:

python -c "import worker; print('worker.py OK')"

You should see worker.py OK printed in the terminal.

Seeing a syntax error?

• Make sure the run_worker() function starts after the virtual_user() function. Python reads top to bottom, so virtual_user must be defined before it is referenced as a thread target.
• Check that the if __name__ == "__main__": block is at the very bottom of the file, not indented inside another function.
• Confirm you have a blank line between the end of virtual_user() and the def run_worker line.

Help me debug a syntax error in my worker.py file.

Understand the graceful shutdown flow

The worker needs to shut down cleanly when the controller says the test is over. A clean shutdown means no orphaned threads and no half-sent messages.

How does the stop mechanism work?

The shutdown relies on a threading.Event called stop_event. When the controller broadcasts a "stop" command, the worker calls stop_event.set(). Every virtual user thread checks this flag at the top of its loop with stop_event.is_set() and exits when it becomes True.

The join(timeout=3) call gives each thread up to 3 seconds to finish its current HTTP request before the worker moves on. After all threads are joined, the worker closes its sockets and terminates the ZeroMQ context.

✔️ Awesome, I've got everything!

Great. Double check you've saved worker.py.

ⓧ I'd like to double check the full code

Here is the complete worker.py file. Make sure yours matches exactly.

import zmq
import json
import time
import os
import threading
import requests


def virtual_user(worker_id, target_url, push_socket, stop_event, lock):
    while not stop_event.is_set():
        start = time.time()
        try:
            resp = requests.get(target_url, timeout=5)
            latency_ms = (time.time() - start) * 1000
            status_code = resp.status_code
        except requests.RequestException:
            latency_ms = (time.time() - start) * 1000
            status_code = 0

        sample = json.dumps({
            "worker_id": worker_id,
            "timestamp": time.time(),
            "latency_ms": round(latency_ms, 2),
            "status_code": status_code,
        })
        with lock:
            push_socket.send_string(sample)


def run_worker(controller_host):
    worker_id = os.environ.get("WORKER_ID", f"worker-{os.getpid()}")
    context = zmq.Context()

    sub_socket = context.socket(zmq.SUB)
    sub_socket.connect(f"tcp://{controller_host}:5555")
    sub_socket.setsockopt_string(zmq.SUBSCRIBE, "")

    push_socket = context.socket(zmq.PUSH)
    push_socket.connect(f"tcp://{controller_host}:5556")

    print(f"Worker {worker_id} connected to controller at {controller_host}")

    lock = threading.Lock()
    threads = []
    stop_event = threading.Event()

    while True:
        message = sub_socket.recv_string()
        msg = json.loads(message)

        if msg["command"] == "start":
            target_url = msg["target_url"]
            vus = msg["vus"]
            print(f"Worker {worker_id}: Starting {vus} virtual users -> {target_url}")
            stop_event.clear()
            for i in range(vus):
                t = threading.Thread(
                    target=virtual_user,
                    args=(worker_id, target_url, push_socket, stop_event, lock),
                    daemon=True,
                )
                t.start()
                threads.append(t)

        elif msg["command"] == "stop":
            print(f"Worker {worker_id}: Stopping...")
            stop_event.set()
            for t in threads:
                t.join(timeout=3)
            threads.clear()
            print(f"Worker {worker_id}: Stopped.")
            break

    sub_socket.close()
    push_socket.close()
    context.term()


if __name__ == "__main__":
    host = os.environ.get("CONTROLLER_HOST", "localhost")
    run_worker(host)

Your worker is ready to generate load. Next up, you will run the full distributed system locally and watch live metrics stream across multiple workers.

Run a Local Distributed Test

You've built all three components of the distributed system: the target server, the controller, and the worker. Each one is a standalone Python process that communicates over ZeroMQ sockets.

Now it's time to run a real distributed load test on your own machine. You'll launch four terminal windows, each running a separate process, and watch live metrics stream in as workers hammer the target server with traffic.

In this step, get ready to:

• Start the target server, controller, and two workers in separate terminals.
• Observe live latency percentiles, throughput, and error rates aggregating in real time.
• Verify that both workers are contributing metrics to the test.

Start all components in separate terminals

A distributed system runs each component as its own process. You'll simulate this locally with four terminal windows, one for each process.

• Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
• Type Terminal and press Enter to open it.
• Open three more terminal windows by pressing Cmd+N (macOS) or Ctrl+Shift+N (Windows) in the terminal app.
• In each of the four terminals, navigate to the folder containing your target_server.py, controller.py, and worker.py files.

Label the windows mentally as Terminal 1, Terminal 2, Terminal 3, and Terminal 4. You'll run one component in each.

• In Terminal 1, start the target server by running:

python target_server.py

You should see Flask's startup message indicating the server is running on port 5000.

Seeing an address already in use error?

Port 5000 might be occupied by another process. Stop whatever is using that port, then try running target_server.py again.

Help me figure out what's using port 5000 on my machine.

• In Terminal 3, start the first worker with a unique ID by running:

macOS

WORKER_ID=worker-1 python worker.py

Windows

$env:WORKER_ID="worker-1"; python worker.py

• In Terminal 4, start the second worker by running:

macOS

WORKER_ID=worker-2 python worker.py

Windows

$env:WORKER_ID="worker-2"; python worker.py

Both workers will print a message confirming they connected to the controller. The WORKER_ID environment variable gives each worker a unique name, so the controller can track which workers are reporting metrics.

Why start workers before the controller?

The controller uses ZeroMQ's PUB/SUB pattern to broadcast the "start" command. PUB/SUB has a "slow joiner" problem: if a subscriber connects after the publisher sends a message, it misses that message entirely.

The controller includes a 2-second sleep before sending the start command, giving workers time to subscribe. Starting workers first ensures they're listening before the countdown begins.

• In Terminal 2, start the controller and kick off the load test by running:

python controller.py http://localhost:5000/random 30 5

What do these arguments mean?

• http://localhost:5000/random is the target URL workers will hit. The /random endpoint returns responses with 10-500ms of random latency.
• 30 is the test duration in seconds.
• 5 is the number of virtual users per worker. With 2 workers, that's 10 virtual users total.

The controller prints "Waiting 2 seconds for workers to connect..." and then sends the start command to both workers over PUB/SUB.

Workers not starting their virtual users?

If the workers stay silent after the controller prints "Sent start command", they may have missed the PUB/SUB message. Stop the controller with Ctrl+C, make sure both workers are still running (showing the "connected" message), then start the controller again.

My workers aren't receiving the start command from the controller.

Observe live metrics during the test

Switch to Terminal 2 (the controller). Every second, a new stats line appears with aggregated metrics from both workers.

You should see output that looks like this:

[  3.0s] RPS:  18.5 | p50:  245.2ms | p95:  478.3ms | p99:  495.1ms | Errors: 0.0% | Workers: 2
[  4.0s] RPS:  19.1 | p50:  230.8ms | p95:  470.6ms | p99:  491.2ms | Errors: 0.0% | Workers: 2

What does each metric mean?

• RPS (Requests Per Second) is the overall throughput across all workers.
• p50 is the median latency. Half of all requests were faster than this.
• p95 and p99 are tail latencies. 95% and 99% of requests were faster than these values.
• Workers shows how many distinct worker IDs the controller has seen in the incoming metrics.

Why do production teams care about p95 and p99?

Average latency can hide problems. If 99 out of 100 requests take 10ms but one takes 5 seconds, the average looks fine at 60ms. The p99 latency reveals that 5-second outlier.

Production SLAs are almost always defined in percentiles, not averages. A service might promise "p99 latency under 500ms," meaning 99% of requests complete in under half a second.

• Watch the controller terminal for about 30 seconds until the test completes.

When the test finishes, the controller sends a "stop" command to all workers and prints a final report.

Verify the distributed behavior

The final report in the controller terminal summarizes the entire test run. This is where you confirm the system worked as a real distributed platform.

• Check the Workers Used line in the final report. It should show 2, confirming both worker-1 and worker-2 reported metrics.
• Check the p95 Latency value. Because the /random endpoint sleeps between 10ms and 500ms, the p95 should land somewhere in the 400-500ms range.
• Check the Total Errors line. It should be 0 since all endpoints return valid responses.

What should the final report look like?

The report shows aggregate stats across all workers and the full test duration. Your exact numbers will differ, but the structure looks like this:

=== Final Report ===
Total Requests:  482
Total Errors:    0
p50 Latency:     242.3ms
p95 Latency:     467.8ms
p99 Latency:     493.1ms
Min Latency:     12.4ms
Max Latency:     503.7ms
Mean Latency:    251.6ms
Avg RPS:         15.5
Workers Used:    2
Duration:        32.0s

• Check Terminals 3 and 4 (the workers). Each should print a "Stopping..." and "Stopped." message, confirming they received the stop command and shut down cleanly.
• In Terminal 1, stop the target server by pressing Ctrl+C.

Only seeing 1 worker in the output?

Make sure you set a different WORKER_ID for each worker (worker-1 and worker-2). If both workers use the same ID, the controller counts them as one.

Also check that both worker terminals printed a "Starting virtual users" message after the controller sent the start command. If a worker stayed silent, it missed the start message and needs to be restarted before the controller.

My controller only sees one worker even though I started two.

Your distributed load testing platform works across multiple processes on a single machine. Next, you'll containerize everything with Docker Compose so you can scale workers with a single command instead of opening terminals manually.

Containerize with Docker Compose

Your distributed load test works locally, but running three terminals with manual process management is not how production systems operate.

In this step, you will containerize the entire platform with Docker Compose. Instead of juggling terminals, a single command will spin up the target server, controller, and as many workers as you want.

In this step, get ready to:

• Write a Dockerfile that packages your Python code and dependencies into a container image.
• Define a Docker Compose configuration with three services that automatically wire together.
• Launch the full stack and scale workers to 4 containers with a single command.

Write the Dockerfile

A Dockerfile is a recipe that tells Docker how to build a container image. Every service in your load testing platform (target, controller, worker) will use the same image since they share the same Python dependencies.

• Open Dockerfile in your code editor.
• Paste the following code into the file:

FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

What does this Dockerfile do?

• FROM python:3.12-slim starts from a lightweight Python image that already has pip installed.
• COPY requirements.txt and RUN pip install happen before copying the rest of the code. This means Docker can cache the dependency layer and skip re-installing packages when only your Python files change.
• COPY . . copies all your source files (target_server.py, controller.py, worker.py) into the image.

• Save Dockerfile.
• You won't see a change yet. The next substep defines the Compose services that use this image.

✔️ Awesome, I've got everything!

Great. Double check you have saved your file.

ⓧ I'd like to double check the full code

FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

Define the Compose services

Docker Compose lets you define multiple services in a single file and launch them together. Services in the same compose.yaml can reference each other by service name as a hostname, so your worker can connect to controller:5555 without any manual networking.

• Open compose.yaml in your code editor.
• Paste the following configuration into the file:

services:
  target:
    build: .
    command: python target_server.py
    ports:
      - "5000:5000"
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:5000/health')"]
      interval: 5s
      timeout: 3s
      retries: 5

  controller:
    build: .
    command: python controller.py http://target:5000/random 30 5
    depends_on:
      target:
        condition: service_healthy
    ports:
      - "5555:5555"
      - "5556:5556"

  worker:
    build: .
    command: python worker.py
    environment:
      - CONTROLLER_HOST=controller
    depends_on:
      - controller

What does each service do?

• target runs your Flask server on port 5000. The healthcheck pings the /health endpoint every 5 seconds to confirm the server is ready before other services start.
• controller uses depends_on with condition: service_healthy to wait until the target server is actually responding, not just started. Notice the command targets http://target:5000/random. The hostname target resolves automatically because Docker Compose networking maps service names to container IPs.
• worker sets CONTROLLER_HOST=controller as an environment variable so the worker process knows where to find the controller's ZeroMQ sockets. No port mapping is needed for workers because they only make outbound connections.

Why can workers scale without port conflicts?

The worker service has no ports mapping. Workers only connect outward to the controller. Since they never bind to a host port, Docker can run as many worker containers as you want without any port conflicts.

• Save compose.yaml.
• You won't see a change yet. The next substep builds and launches the full stack.

✔️ Awesome, I've got everything!

Great. Double check you have saved your file.

ⓧ I'd like to double check the full code

services:
  target:
    build: .
    command: python target_server.py
    ports:
      - "5000:5000"
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:5000/health')"]
      interval: 5s
      timeout: 3s
      retries: 5

  controller:
    build: .
    command: python controller.py http://target:5000/random 30 5
    depends_on:
      target:
        condition: service_healthy
    ports:
      - "5555:5555"
      - "5556:5556"

  worker:
    build: .
    command: python worker.py
    environment:
      - CONTROLLER_HOST=controller
    depends_on:
      - controller

Build and launch the full stack

Everything is defined. Time to build the Docker images and launch all services with a single command.

• If your target server from the previous step is still running, stop it by pressing Ctrl+C in its terminal. Ports 5000, 5555, and 5556 need to be free for Docker to map them.
• Make sure Docker Desktop is running.
• In your terminal, make sure you are inside the load-tester project folder.
• Build the images and launch the full stack with 4 workers by running this command:

docker compose up --build --scale worker=4

What does this command do?

• docker compose up starts all services defined in compose.yaml.
• --build forces Docker to build the images from your Dockerfile before starting containers.
• --scale worker=4 launches 4 separate worker containers instead of the default 1. Each worker independently connects to the controller and spawns its own virtual users.

The build will take a moment the first time. After that, you will see interleaved logs from all containers.

• Watch the terminal output. You should see the target server's healthcheck pass, then the controller start, then all 4 workers connect.
• Look for the controller's live stats lines. They should show Workers: 4 confirming all 4 worker containers are reporting metrics.
• Wait for the 30-second test to complete. The controller will print a final report with aggregated p50, p95, and p99 latencies, total requests, and RPS across all workers.

Not seeing any output or getting an error?

• If you see an error mentioning the Docker daemon not running, open Docker Desktop and wait for it to fully start before running the command again.
• If you see a port conflict error (e.g. port 5000 already in use), make sure you stopped the local target server from the previous step.
• If the controller shows Workers: 0 or fewer than 4, check that your compose.yaml has CONTROLLER_HOST=controller in the worker's environment. Without this, workers try to connect to localhost inside their own container.

Why isn't my Docker Compose stack starting?

Once the test completes and you see the final report, tear down the entire stack by running this command:

docker compose down

• You should see each container stopping and being removed.

What just happened?

You just ran a fully distributed load test without opening a single extra terminal. Docker Compose handled building the images, creating a shared network, starting services in the right order (target first, then controller, then workers), and scaling workers horizontally.

This is the same pattern production tools like Locust use. Locust's distributed mode has a master (your controller) coordinating workers over ZeroMQ. You built that architecture from scratch.

Your load testing platform is fully containerized and horizontally scalable. Next up, the Secret Mission challenges you to add a ramp-up load profile that gradually increases virtual users over time.

Secret mission

Add a Ramp-Up Load Profile

Real-world traffic doesn't hit a server at full blast from the first millisecond. Production load tests ramp up gradually, adding virtual users over time to find the exact breaking point. Can you add a ramp-up profile that gradually increases the load, so the controller orchestrates a smooth traffic curve instead of a flat one?

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally with Docker and Python, so there are no ongoing cloud costs.

Resources you used:

• Project files on your Desktop (target_server.py, controller.py, worker.py, requirements.txt, Dockerfile, compose.yaml).
• Docker images and containers built by Docker Compose.
• Python packages installed via pip (pyzmq, flask, requests).

Keep everything running

No action needed. Choose this if you're still actively building or experimenting with your load testing platform.

• Your project files stay on your Desktop.
• Docker images remain cached locally so future builds are faster.
• The pyzmq, flask, and requests packages remain installed in your Python environment.

Pause - I'll come back to this later

Shut down running containers to free up memory, but keep your files and images so you can resume quickly.

• If your Docker Compose stack is still running, stop it by running:

docker compose down

• This stops and removes the containers but keeps your Docker images and project files intact.
• When you're ready to resume, run docker compose up --build from your project folder to start everything again.

Delete - I don't want to use this again

Remove all project resources and start fresh. Follow each step below to clean everything up.

• Stop and remove all containers and images built by Docker Compose by running:

docker compose down --rmi all

• Uninstall the Python packages you installed for this project by running:

pip uninstall pyzmq flask requests -y

• Delete your project folder from the Desktop.

macOS

• Remove the project folder by running:

cd ~/Desktop
rm -rf distributed-load-tester

Windows

• Remove the project folder by running:

cd ~\Desktop
rmdir /s /q distributed-load-tester

• Confirm the folder is gone by running ls on your Desktop. You should no longer see the project folder in the output.

Nice Work!

Nice Work!

You just built a distributed load testing platform from scratch. Your system coordinates multiple workers over ZeroMQ, aggregates real-time metrics, and scales horizontally with a single Docker Compose command.

Here's what you accomplished:

• Built a controller/worker architecture using ZeroMQ PUB/SUB to broadcast commands and PUSH/PULL to collect timing samples from distributed workers.
• Computed real-time latency percentiles (p50, p95, p99), throughput (RPS), and error rates by aggregating metrics across multiple workers.
• Containerized the entire platform with Docker Compose and scaled workers horizontally with --scale.
• Secret Mission: Added a ramp-up load profile that gradually increases virtual users over time, demonstrating load shaping for realistic traffic simulation.

Ready to quiz yourself?