System Design Scaling Simulator

Introduction

30 Second Summary

You have a system design interview tomorrow. You have watched the YouTube videos and read the blog posts, but when you close your eyes, you cannot actually feel the difference between 1,000 users and 100,000 users hitting your server. Passive learning does not stick under pressure.

In this project, you will build a Python CLI application that simulates scaling a web service from 100 to 1,000,000 users. You will watch real metrics degrade under load, then recover as you apply each scaling pattern.

What You'll Build

When you run your simulator before an interview, you walk through 6 scaling stages and watch a single server crumble under pressure, then see latency drop and throughput climb as load balancing, caching, and sharding kick in.

By the end of this project, you'll have:

• A running program that fires concurrent requests using ThreadPoolExecutor and shows you real p50/p95/p99 latency, throughput, and error rates at each scaling stage.
• Real observable proof that caching drops p95 latency by 80%+, that consistent hashing distributes keys evenly across shards, and that circuit breakers protect your system from cascading failures.
• A built-in Lead Engineer theory reference woven into each stage with trade-off reasoning, cost arguments, failure modes, and cloud service mappings you can use verbatim in your interview.
• Secret Mission: an interactive quiz mode that tests your ability to pick the right scaling pattern for real-world scenarios under pressure.

Are there any prerequisites?

You need Python 3.12 installed and a code editor (VS Code or similar). This project uses only the Python standard library, so there are no packages to install.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

Set Up the Simulator Foundation

Every scaling pattern you will simulate in later steps depends on a working engine underneath. The engine needs to fire concurrent requests, measure latency percentiles, and display results.

In this step you will build that engine from scratch in a single Python file. By the end, you will have a runnable program that shows an interactive menu for every stage of the scaling journey.

In this step, get ready to:

• Create the data structures that hold request and metrics results.
• Build a simulated server and a load test runner powered by Python threads.
• Wire up display helpers and an interactive menu so you can launch any stage.

Create the file with imports and data structures

Before writing any simulation logic, you need a file that imports Python's concurrency and timing tools and defines the shape of the data flowing through the system.

• Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
• Type Terminal and press Enter to open it.

• Move to your Desktop by running this command:

cd ~/Desktop

• Create your project folder and move into it by running these commands:

mkdir scaling-simulator
cd scaling-simulator

• Create the simulator file by running:

touch system_design_simulator.py

• Confirm the file exists by running:

ls

You should see system_design_simulator.py in the output.

• Open VS Code in this folder by running:

code .

code command not found?

If code . does not work, open VS Code manually, click File then Open Folder, and navigate to Desktop/scaling-simulator.

Help me add the code command to my PATH

• In VS Code, click on system_design_simulator.py in the Explorer sidebar to open it.
• Paste the following file header and imports at the top of the file:

#!/usr/bin/env python3
"""
System Design Scaling Simulator
A hands-on learning tool for Lead Engineer system design interviews.
Simulates scaling from 100 to 1,000,000 users with real metrics.

Run: python3 system_design_simulator.py
"""

import time
import random
import threading
import statistics
import hashlib
import queue
from concurrent.futures import ThreadPoolExecutor, as_completed
from collections import OrderedDict
from dataclasses import dataclass, field
from typing import Optional

What do these imports do?

• time and random simulate realistic server processing delays.
• threading provides locks and semaphores to simulate server capacity limits.
• ThreadPoolExecutor fires concurrent requests just like real clients hitting a server simultaneously.
• hashlib, queue, and OrderedDict power the consistent hashing, message queue, and LRU cache you will build in later steps.

• Below the imports, add the data classes that define the shape of every request and metrics result:

# =============================================================================
# DATA CLASSES
# =============================================================================

@dataclass
class RequestResult:
    """Result of a single simulated request."""
    latency_ms: float
    success: bool
    server_id: Optional[str] = None
    from_cache: bool = False


@dataclass
class MetricsResult:
    """Aggregated metrics from a load test."""
    total_requests: int
    successful: int
    failed: int
    avg_latency_ms: float
    p50_latency_ms: float
    p95_latency_ms: float
    p99_latency_ms: float
    throughput_rps: float
    error_rate: float
    duration_seconds: float
    extra: dict = field(default_factory=dict)

What do these data classes do?

RequestResult holds the outcome of one simulated HTTP request: how long it took, whether it succeeded, and which server handled it.

MetricsResult aggregates hundreds of individual results into the numbers you discuss in an interview: p50, p95, p99 latency, throughput, and error rate.

• Now add the metrics engine below your data classes. This class turns raw request results into interview-ready percentiles:

# =============================================================================
# METRICS COLLECTOR
# =============================================================================

class MetricsCollector:
    """Collects and calculates performance metrics from request results."""

    @staticmethod
    def calculate(results: list[RequestResult], duration: float) -> MetricsResult:
        if not results:
            return MetricsResult(0, 0, 0, 0, 0, 0, 0, 0, 0, duration)

        successful = [r for r in results if r.success]
        failed = [r for r in results if not r.success]
        latencies = [r.latency_ms for r in results]

        sorted_latencies = sorted(latencies)
        n = len(sorted_latencies)

        p50 = sorted_latencies[n // 2] if n > 0 else 0
        p95 = sorted_latencies[int(n * 0.95)] if n > 1 else sorted_latencies[-1]
        p99 = sorted_latencies[int(n * 0.99)] if n > 1 else sorted_latencies[-1]

        return MetricsResult(
            total_requests=len(results),
            successful=len(successful),
            failed=len(failed),
            avg_latency_ms=statistics.mean(latencies) if latencies else 0,
            p50_latency_ms=p50,
            p95_latency_ms=p95,
            p99_latency_ms=p99,
            throughput_rps=len(results) / duration if duration > 0 else 0,
            error_rate=len(failed) / len(results) if results else 0,
            duration_seconds=duration,
        )

What does MetricsCollector do?

It sorts all latencies and picks values at the 50th, 95th, and 99th percentile positions. p95 means "95% of requests completed faster than this." In interviews, p99 is the number that matters most because it represents your worst-case user experience.

• Add a temporary display banner and entry point below the MetricsCollector so you can see something when you run the program:

# =============================================================================
# DISPLAY HELPERS
# =============================================================================

def display_banner() -> None:
    print("\n" + "=" * 70)
    print("   SYSTEM DESIGN SCALING SIMULATOR")
    print("   Lead Engineer Interview Preparation Tool")
    print("=" * 70)
    print("\n   Scaling Journey: 100 users --> 1,000,000 users")
    print("   Each stage shows: Theory --> Simulation --> Metrics --> Talking Points")
    print("\n" + "-" * 70)
    print("   STAGES:")
    print("   [1] Single Server (100-1K users) + Back-of-Envelope Estimation")
    print("   [2] Load Balancing (1K-5K users) + Stateless Design")
    print("   [3] Caching Layer (5K-100K users) + CAP Theorem")
    print("   [4] DB Replication & Sharding (100K-500K) + Consistency Models")
    print("   [5] Async Processing & Rate Limiting (500K-1M) + Circuit Breakers")
    print("   [6] Full Journey (all stages sequentially)")
    print("   [T] Theory Reference (all concepts)")
    print("   [Q] Quit")
    print("-" * 70)


if __name__ == "__main__":
    display_banner()

Why the temporary entry point?

The if __name__ == "__main__" block at the bottom runs display_banner() directly so you can see the menu right now. You will replace this with the full interactive menu at the end of this step.

• Save system_design_simulator.py and run it:

python3 system_design_simulator.py

You should see the full menu banner listing all 6 stages, Theory mode, and Quit.

Banner not showing?

Make sure if __name__ == "__main__": is at the very bottom of the file with no indentation. Confirm the file is saved before running.

Help me debug why the banner is not showing

Add the server simulation and load test runner

With data structures and metrics in place, you need something that actually handles requests. The SimulatedServer class mimics a real web server that slows down under load and starts failing when overwhelmed.

• In system_design_simulator.py, find the line that says # DISPLAY HELPERS.
• Add the following code ABOVE that display helpers section. This is the simulated server class:

# =============================================================================
# SIMULATED SERVER
# =============================================================================

class SimulatedServer:
    """Simulates a server with configurable capacity and processing time."""

    def __init__(self, server_id: str, max_concurrent: int = 10,
                 base_latency_ms: float = 20.0):
        self.server_id = server_id
        self.max_concurrent = max_concurrent
        self.base_latency_ms = base_latency_ms
        self._semaphore = threading.Semaphore(max_concurrent)
        self._current_load = 0
        self._lock = threading.Lock()

What does the Semaphore do?

A Semaphore limits how many threads can enter a section of code at once. Here it enforces a maximum concurrent request limit. When all slots are taken, new requests must wait (adding latency) or time out (producing errors). This is exactly how real servers behave under overload.

• Still inside the SimulatedServer class, add the request handler method below __init__:

    def handle_request(self, timeout: float = 5.0) -> RequestResult:
        acquired = self._semaphore.acquire(timeout=timeout)
        if not acquired:
            return RequestResult(latency_ms=timeout * 1000, success=False,
                                 server_id=self.server_id)

        try:
            with self._lock:
                self._current_load += 1
                load_factor = self._current_load / self.max_concurrent

            # Latency increases with load (simulates resource contention)
            jitter = random.uniform(0.5, 1.5)
            latency = self.base_latency_ms * (1 + load_factor * 2) * jitter
            time.sleep(latency / 1000.0)

            # Higher load = higher chance of errors
            error_chance = max(0, (load_factor - 0.8) * 0.5)
            success = random.random() > error_chance

            return RequestResult(latency_ms=latency, success=success,
                                 server_id=self.server_id)
        finally:
            with self._lock:
                self._current_load -= 1
            self._semaphore.release()

How does degradation work?

The load_factor tracks how full the server is (0.0 to 1.0). As load grows, latency increases exponentially and error probability rises. Above 80% capacity, requests start failing randomly. This models real-world server saturation.

• Now add the load test runner between the SimulatedServer class and the DISPLAY HELPERS section:

# =============================================================================
# LOAD TEST RUNNER
# =============================================================================

def run_load_test(handler, num_requests: int, max_workers: int = 50) -> MetricsResult:
    """Run a load test using ThreadPoolExecutor."""
    results: list[RequestResult] = []
    start_time = time.perf_counter()

    with ThreadPoolExecutor(max_workers=max_workers) as executor:
        futures = [executor.submit(handler) for _ in range(num_requests)]
        for future in as_completed(futures):
            try:
                results.append(future.result())
            except Exception:
                results.append(RequestResult(latency_ms=5000, success=False))

    duration = time.perf_counter() - start_time
    return MetricsCollector.calculate(results, duration)

What does ThreadPoolExecutor do here?

ThreadPoolExecutor spins up max_workers threads that each fire a request simultaneously. This simulates real concurrent traffic hitting your server. time.perf_counter() provides high-resolution timing so your throughput calculations are accurate.

• Save system_design_simulator.py and run it to confirm there are no syntax errors:

python3 system_design_simulator.py

The banner should still appear. The server and load test code are defined but not called yet.

Seeing a syntax error?

Check that handle_request is indented with 4 spaces (it is a method inside the SimulatedServer class). The run_load_test() function should be at the top level with no indentation.

Help me fix this syntax error

Add the remaining display helpers and the interactive menu

The engine is built. Now you need the output layer that formats metrics, draws architecture diagrams, and lets you pick which stage to run.

• In system_design_simulator.py, find the display_banner() function you already added.
• Directly below display_banner(), add the metrics display function:

def display_metrics(metrics: MetricsResult, label: str = "") -> None:
    print(f"\n   {'--- ' + label + ' ---' if label else '--- Metrics ---'}")
    print(f"   Requests:    {metrics.total_requests} total, "
          f"{metrics.successful} OK, {metrics.failed} FAILED")
    print(f"   Latency:     avg={metrics.avg_latency_ms:.1f}ms, "
          f"p50={metrics.p50_latency_ms:.1f}ms, "
          f"p95={metrics.p95_latency_ms:.1f}ms, "
          f"p99={metrics.p99_latency_ms:.1f}ms")
    print(f"   Throughput:  {metrics.throughput_rps:.1f} req/s")
    print(f"   Error Rate:  {metrics.error_rate * 100:.1f}%")
    print(f"   Duration:    {metrics.duration_seconds:.2f}s")

• Below display_metrics(), add the architecture diagram function:

def display_architecture(stage: int) -> None:
    architectures = {
        1: "\n    [Users] --> [Single Server + DB]\n                (everything on one box)\n        ",
        2: "\n    [Users] --> [Load Balancer]\n                   /    |    \\\n            [App1] [App2] [App3]\n                   \\    |    /\n                    [Database]\n        ",
        3: "\n    [Users] --> [Load Balancer]\n                   /    |    \\\n            [App1] [App2] [App3]\n                   \\    |    /\n                [Cache Layer (Redis)]\n                       |\n                    [Database]\n        ",
        4: "\n    [Users] --> [Load Balancer]\n                   /    |    \\\n            [App1] [App2] [App3]\n                   \\    |    /\n                [Cache Layer]\n                   /        \\\n        [Primary DB]    [Read Replicas]\n              |          /     \\\n        [Shard 1] [Shard 2] [Shard 3]\n        ",
        5: "\n    [Users] --> [API Gateway + Rate Limiter]\n                        |\n                [Load Balancer]\n                /       |       \\\n         [App1]    [App2]    [App3]\n           |          |          |\n    [Cache] [Message Queue] [Circuit Breaker]\n           |          |          |\n    [DB Primary] [Workers]  [External APIs]\n           |\n    [Read Replicas / Shards]\n        "
    }
    print(architectures.get(stage, ""))

What are these architecture diagrams?

Each stage of the scaling journey has a different topology. These ASCII diagrams print before each simulation so you can see how the system evolves. Stage 1 is a single box. Stage 5 has load balancers, caches, queues, and circuit breakers.

• Below display_architecture(), add the helper functions that format theory blocks and talking points:

def print_section(title: str) -> None:
    print(f"\n{'=' * 70}")
    print(f"  {title}")
    print(f"{'=' * 70}")


def print_theory(title: str, content: str) -> None:
    print(f"\n  [THEORY] {title}")
    print(f"  {'-' * 60}")
    for line in content.strip().split("\n"):
        print(f"  {line}")
    print()


def print_lead_engineer(content: str) -> None:
    print(f"\n  [LEAD ENGINEER TALKING POINTS]")
    print(f"  {'-' * 60}")
    for line in content.strip().split("\n"):
        print(f"  {line}")
    print()

What are these print helpers for?

print_section() creates visual separators between stages. print_theory() and print_lead_engineer() format the study material that appears alongside each simulation. Every stage will use these to present concepts right when they become relevant.

• Now replace the temporary entry point at the bottom of the file. Find these two lines at the very end:

if __name__ == "__main__":
    display_banner()

• Delete those two lines and replace them with the full interactive menu:

# =============================================================================
# MAIN
# =============================================================================

def main() -> None:
    display_banner()

    while True:
        choice = input("\n  Select stage [1-6, T, Q]: ").strip().upper()

        if choice == "1":
            pass
        elif choice == "2":
            pass
        elif choice == "3":
            pass
        elif choice == "4":
            pass
        elif choice == "5":
            pass
        elif choice == "6":
            pass
        elif choice == "T":
            pass
        elif choice == "Q":
            print("\n  Good luck in your interview!\n")
            break
        else:
            print("  Invalid choice. Enter 1-6, T, or Q.")

        input("\n  Press Enter to return to menu...")
        display_banner()


if __name__ == "__main__":
    main()

What does main() do?

The while True loop keeps the menu alive until you press Q. Each stage is wired to pass for now. You will replace each pass with a real simulation function in the next steps.

Before you run this, what do you expect to see? Will the program exit immediately, or will it wait for input?

• Save system_design_simulator.py and run it:

python3 system_design_simulator.py

You should see the full banner with all 6 stages listed, then a prompt waiting for your input. Type Q and press Enter to exit.

Program exits without showing the menu?

Confirm that if __name__ == "__main__": main() is at the very bottom of the file with no indentation. Make sure the old display_banner() call was fully replaced, not duplicated.

Help me debug why main() is not running

✔️ Awesome, I've got everything!

Great. Double check you have saved system_design_simulator.py and that the menu prompt appears when you run it.

ⓧ I'd like to double check the full code

Your file should contain these sections in order. Scroll through your code and confirm each is present:

• File header (shebang, docstring) and all 10 imports.
• DATA CLASSES section with RequestResult and MetricsResult dataclasses.
• METRICS COLLECTOR section with the MetricsCollector class and its calculate() static method.
• SIMULATED SERVER section with SimulatedServer class (__init__ and handle_request methods).
• LOAD TEST RUNNER section with the run_load_test() function.
• DISPLAY HELPERS section with display_banner(), display_metrics(), display_architecture(), print_section(), print_theory(), and print_lead_engineer().
• MAIN section with main() function and if __name__ == "__main__": main() at the very bottom.

Your simulation engine is ready. The data flows from SimulatedServer through run_load_test() into MetricsCollector, and the results display through your helper functions. Next up, you will put this engine under real pressure and watch a single server collapse.

Simulate a Single Server Under Pressure

Your simulator engine can spin up concurrent threads and measure latency. Now it is time to use it to answer the most fundamental question in system design: what happens when a single server gets more traffic than it can handle?

Before learning any scaling pattern, you need to feel what "breaking" looks like. In this step, you will watch a single server handle 100 users comfortably, then buckle under 1,000. You will also embed the theory you would recite first in an interview: back-of-envelope estimation and horizontal vs vertical scaling.

In this step, get ready to:

• Add theory blocks for back-of-envelope estimation and horizontal vs vertical scaling.
• Simulate a single server under 100, 500, and 1,000 concurrent users and watch metrics degrade.
• Display Lead Engineer talking points for interview preparation.

Add the theory section of stage_single_server()

In a real interview, the first thing you do after hearing the question is back-of-envelope estimation. This function will print that theory before running any simulation.

• In system_design_simulator.py, scroll down to just above the main() function.
• Add the following stage_single_server() function above main():

def stage_single_server() -> None:
    print_section("STAGE 1: Single Server (0 - 1,000 Users)")
    display_architecture(1)
    print_theory("Back-of-Envelope Estimation (DO THIS FIRST IN EVERY INTERVIEW)", """
WHAT: Quick math to estimate system requirements before designing.
WHEN: The FIRST thing after the interviewer states the problem.
WHY:  Determines which scaling patterns you actually need.
HOW:  QPS = DAU x requests/user / 86,400. Peak = QPS x 2-3.
      Storage = Users x data/user x retention.
      Bandwidth = QPS x avg response size.
      Powers of two: 1KB=1000B, 1MB=1M B, 1GB=1B B, 1TB=1T B.
    """)
    print_theory("Horizontal vs Vertical Scaling", """
Vertical (Scale Up): bigger machine. Horizontal (Scale Out): more machines.
Vertical FIRST until ceiling. Horizontal requires stateless design.
AWS: EC2 instance types (vertical), Auto Scaling Groups (horizontal).
GCP: Compute Engine machine types (vertical), Managed Instance Groups.
    """)

What does this code do?

• print_section() and display_architecture(1) show a header and ASCII diagram of the single-server topology.
• The first print_theory() prints the back-of-envelope formulas (QPS, storage, bandwidth) that you would say first in any interview.
• The second print_theory() prints the horizontal vs vertical scaling comparison with AWS and GCP equivalents.

Wire stage 1 into the menu

The function exists but the menu still routes to pass. Let's connect it so you can see the theory output right away.

• In the main() function, find this line:

        if choice == "1":
            pass

• Replace pass with stage_single_server() so the block looks like this:

        if choice == "1":
            stage_single_server()

• Save system_design_simulator.py.
• Run the program by executing:

python3 system_design_simulator.py

• When the menu appears, type 1 and press Enter.

You should see the STAGE 1 header, the ASCII architecture diagram, the back-of-envelope estimation theory, and the horizontal vs vertical scaling theory printed to your terminal.

Not seeing the theory output?

• Make sure stage_single_server() is defined above main(). Python reads top-to-bottom, so calling a function before it is defined causes a NameError.
• Check that the triple-quoted strings inside print_theory() end with """) on their own line. A missing closing triple-quote causes a syntax error.

Help me debug why Stage 1 is not printing theory output when I select option 1.

• Press Enter to return to the menu, then type Q to quit.

Add the load test simulation and Lead Engineer talking points

Theory alone does not stick. Now you will add code that fires real concurrent requests at the simulated server so you can watch metrics degrade as load increases.

• In stage_single_server(), add the following code directly after the second print_theory() call (the Horizontal vs Vertical Scaling block):

    print("\n  [SIMULATION] Running load test: Single server under pressure...")
    server = SimulatedServer("single-server", max_concurrent=10, base_latency_ms=20)
    # Test with 100 concurrent users
    print("\n  Test 1: 100 concurrent users...")
    metrics_100 = run_load_test(server.handle_request, num_requests=100, max_workers=20)
    display_metrics(metrics_100, "100 Users - Comfortable")
    # Test with 500 concurrent users
    print("\n  Test 2: 500 concurrent users (pushing limits)...")
    metrics_500 = run_load_test(server.handle_request, num_requests=500, max_workers=50)
    display_metrics(metrics_500, "500 Users - Under Pressure")
    # Test with 1000 concurrent users
    print("\n  Test 3: 1000 concurrent users (server overwhelmed)...")
    metrics_1000 = run_load_test(server.handle_request, num_requests=1000, max_workers=100)
    display_metrics(metrics_1000, "1000 Users - OVERWHELMED")
    # Lead Engineer interview tips for this stage
    print_lead_engineer("""
At 1K users: single server is fine. Do NOT add K8s or microservices.
Transition trigger: p99 > SLA or CPU > 70% sustained.
Failure modes: SPOF, no graceful degradation.
Cost: t3.medium $30/month handles 1K users easily.
    """)

What does this simulation do?

• A SimulatedServer with max_concurrent=10 uses a threading Semaphore to limit parallel processing. Requests beyond capacity queue up or time out.
• run_load_test() fires all requests concurrently using ThreadPoolExecutor. At 100 users, most requests get through smoothly. At 1,000, the server is overwhelmed and latency spikes.
• The print_lead_engineer() block gives you the exact phrasing to use in an interview: what not to build at this scale, the transition trigger, and cost reasoning.

• Save system_design_simulator.py.

Before you run this, think: what do you expect to happen to p95 latency as you go from 100 users to 1,000 on a server that can only handle 10 concurrent requests?

• Run the program again:

python3 system_design_simulator.py

• Type 1 and press Enter.

Watch the three tests run. You should see p95 latency spike from roughly 50ms at 100 users up to 2000ms or more at 1,000 users. The error rate climbs as requests time out waiting for the overwhelmed server.

Why does this matter for interviews?

You just watched a server break under load. This is the experience that makes scaling patterns intuitive rather than memorized. In an interview, you can say: "A single server with 10 concurrent connections is fine for 100 users, but at 1,000 the p99 exceeds any reasonable SLA. That is my trigger to scale horizontally."

Simulation takes too long or hangs?

• The 1,000-user test may take 10-20 seconds because requests queue behind the 10-connection limit. This is expected. If it hangs beyond 30 seconds, press Ctrl+C and check that max_workers=100 is set in the third run_load_test() call.
• If you see a NameError for SimulatedServer or run_load_test, confirm that the class and function from Step 1 are still intact above your new code.

Help me debug why my Stage 1 simulation is hanging or crashing.

✔️ Awesome, I've got everything!

Great. Double check you have saved system_design_simulator.py and that selecting Stage 1 prints theory, runs all three tests, and shows the Lead Engineer talking points.

ⓧ I'd like to double check the full code

Here is the complete stage_single_server() function. Confirm yours matches:

def stage_single_server() -> None:
    print_section("STAGE 1: Single Server (0 - 1,000 Users)")
    display_architecture(1)
    print_theory("Back-of-Envelope Estimation (DO THIS FIRST IN EVERY INTERVIEW)", """
WHAT: Quick math to estimate system requirements before designing.
WHEN: The FIRST thing after the interviewer states the problem.
WHY:  Determines which scaling patterns you actually need.
HOW:  QPS = DAU x requests/user / 86,400. Peak = QPS x 2-3.
      Storage = Users x data/user x retention.
      Bandwidth = QPS x avg response size.
      Powers of two: 1KB=1000B, 1MB=1M B, 1GB=1B B, 1TB=1T B.
    """)
    print_theory("Horizontal vs Vertical Scaling", """
Vertical (Scale Up): bigger machine. Horizontal (Scale Out): more machines.
Vertical FIRST until ceiling. Horizontal requires stateless design.
AWS: EC2 instance types (vertical), Auto Scaling Groups (horizontal).
GCP: Compute Engine machine types (vertical), Managed Instance Groups.
    """)

    print("\n  [SIMULATION] Running load test: Single server under pressure...")
    server = SimulatedServer("single-server", max_concurrent=10, base_latency_ms=20)
    print("\n  Test 1: 100 concurrent users...")
    metrics_100 = run_load_test(server.handle_request, num_requests=100, max_workers=20)
    display_metrics(metrics_100, "100 Users - Comfortable")
    print("\n  Test 2: 500 concurrent users (pushing limits)...")
    metrics_500 = run_load_test(server.handle_request, num_requests=500, max_workers=50)
    display_metrics(metrics_500, "500 Users - Under Pressure")
    print("\n  Test 3: 1000 concurrent users (server overwhelmed)...")
    metrics_1000 = run_load_test(server.handle_request, num_requests=1000, max_workers=100)
    display_metrics(metrics_1000, "1000 Users - OVERWHELMED")
    print_lead_engineer("""
At 1K users: single server is fine. Do NOT add K8s or microservices.
Transition trigger: p99 > SLA or CPU > 70% sustained.
Failure modes: SPOF, no graceful degradation.
Cost: t3.medium $30/month handles 1K users easily.
    """)

And confirm that main() routes choice "1" to stage_single_server() instead of pass.

You just watched a server crumble in real time. That visceral experience of latency spiking and errors climbing is exactly what makes scaling patterns click. Next up, you will fix this by distributing traffic across multiple servers with a load balancer and slashing latency with a caching layer.

Add Load Balancing and Caching

In the previous step, your single server collapsed under 1,000 concurrent users. Latency spiked and errors appeared. The fix is horizontal scaling: spread traffic across multiple servers using a load balancer.

But even with multiple servers, the database becomes the next bottleneck. Every request still hits the same data store. A cache layer absorbs repeated reads and drops latency dramatically. In this step, you will build both patterns and watch real metrics improve.

In this step, get ready to:

• Build a LoadBalancer class that distributes requests across a pool of servers.
• Build an LRUCache class with TTL support to reduce database hits.
• Add stage functions that simulate load balancing and caching with real metrics.

Add the LoadBalancer class and stage function

A load balancer sits between users and your servers. It decides which server handles each incoming request. You will implement two common algorithms: round-robin (each server takes turns) and least-connections (pick the server with the least active work).

• In system_design_simulator.py, find the end of the SimulatedServer class (after the handle_request method). Add the LoadBalancer class below it:

class LoadBalancer:
    def __init__(self, servers: list[SimulatedServer], algorithm: str = "round_robin"):
        self.servers = servers
        self.algorithm = algorithm
        self._rr_index = 0
        self._lock = threading.Lock()

    def get_server(self) -> SimulatedServer:
        with self._lock:
            if self.algorithm == "round_robin":
                server = self.servers[self._rr_index % len(self.servers)]
                self._rr_index += 1
                return server
            else:
                return min(self.servers, key=lambda s: s._current_load)

What does this code do?

The LoadBalancer accepts a list of servers and an algorithm choice. It uses a threading lock so multiple threads can safely request a server at the same time.

• Round-robin cycles through servers in order using _rr_index modulo the number of servers.
• Least-connections picks the server with the lowest _current_load value, sending new requests to whichever server is least busy.

Now add the stage function that uses this class. The function prints theory, runs a simulation comparing 1 server vs 3 load-balanced servers, and shows the improvement.

• Below the stage_single_server() function, add the stage_load_balancing() function:

def stage_load_balancing() -> None:
    print_section("STAGE 2: Load Balancing (1,000 - 5,000 Users)")
    display_architecture(2)
    print_theory("Load Balancing", "Round-robin, least-connections, IP-hash, weighted. L4 vs L7.")
    print_theory("The Stateless Server Rule", "No session state on server. Move to Redis/S3.")
    single = SimulatedServer("solo", max_concurrent=10, base_latency_ms=20)
    metrics_single = run_load_test(single.handle_request, num_requests=1000, max_workers=50)
    display_metrics(metrics_single, "1 Server, 1000 Requests")
    servers = [SimulatedServer(f"app-{i}", max_concurrent=10, base_latency_ms=20) for i in range(3)]
    lb = LoadBalancer(servers, algorithm="round_robin")
    def lb_request():
        return lb.get_server().handle_request()
    metrics_lb = run_load_test(lb_request, num_requests=1000, max_workers=50)
    display_metrics(metrics_lb, "3 Servers + Load Balancer, 1000 Requests")
    improvement = ((metrics_single.p95_latency_ms - metrics_lb.p95_latency_ms) / metrics_single.p95_latency_ms * 100)
    print(f"\n  >> P95 Latency Improvement: {improvement:.1f}%")
    print(f"  >> Error Rate: {metrics_single.error_rate*100:.1f}% -> {metrics_lb.error_rate*100:.1f}%")
    print_lead_engineer("Use ALB on AWS. Avoid sticky sessions. Stateless servers only.")

What does this stage simulate?

The function runs two tests back-to-back. First it fires 1,000 requests at a single server and records the metrics. Then it creates 3 servers behind a LoadBalancer and fires the same 1,000 requests.

The inner function lb_request() asks the load balancer which server should handle each request. The improvement percentage shows how much p95 latency dropped thanks to distributing the load.

• Save system_design_simulator.py.
• Confirm there are no syntax errors by running this command:

python3 -c "import system_design_simulator; print('Import OK')"

You should see Import OK printed with no errors.

Seeing a SyntaxError or IndentationError?

• Make sure the LoadBalancer class is at the top level (no extra indentation), not nested inside another class.
• Check that stage_load_balancing() is also at the top level, aligned with stage_single_server().
• Verify you have a blank line between the end of the class and the start of the function.

Why is my import failing?

Now wire Stage 2 into the menu so you can run it.

• In the main() function, find this line:

        elif choice == "2":
            pass

• Replace pass with stage_load_balancing(), so it looks like this:

        elif choice == "2":
            stage_load_balancing()

• Save the file.

Before you run Stage 2, think about what you expect. Will 3 servers reduce latency by a lot or a little compared to 1 server handling 1,000 requests?

• Run the simulator and select Stage 2 by entering 2:

python3 system_design_simulator.py

You should see the theory blocks print first, then two sets of metrics. The p95 latency for 3 servers should be noticeably lower than for 1 server, and the error rate should drop.

Stage 2 not appearing in the menu?

• Check that you replaced pass with stage_load_balancing() in the elif choice == "2" block.
• Make sure you saved the file after making the change.

Why does selecting 2 do nothing in my simulator?

Add the LRU cache and its stage function

Load balancing solved the server CPU bottleneck. But all 3 servers still hit the same database for every request. A cache stores frequently accessed data in memory so repeat reads skip the database entirely.

• Below the LoadBalancer class, add the first part of the LRUCache class with its constructor and get() method:

class LRUCache:
    def __init__(self, max_size: int = 1000, ttl_seconds: float = 60.0):
        self.max_size = max_size
        self.ttl_seconds = ttl_seconds
        self._cache: OrderedDict = OrderedDict()
        self._timestamps: dict = {}
        self._lock = threading.Lock()
        self.hits = 0
        self.misses = 0

    def get(self, key: str) -> Optional[str]:
        with self._lock:
            if key in self._cache:
                if time.time() - self._timestamps[key] > self.ttl_seconds:
                    del self._cache[key]
                    del self._timestamps[key]
                    self.misses += 1
                    return None
                self._cache.move_to_end(key)
                self.hits += 1
                return self._cache[key]
            self.misses += 1
            return None

What does the get() method do?

When you look up a key, the cache checks two things: does the key exist, and has it expired (TTL). If the key is fresh, it moves to the end of the ordered dict (marking it as recently used) and increments hits.

If the key is missing or expired, the method increments misses and returns None. Expired entries are cleaned up immediately on access.

• Directly below the get() method (still inside the LRUCache class), add the put() method and hit_rate property:

    def put(self, key: str, value: str) -> None:
        with self._lock:
            if key in self._cache:
                self._cache.move_to_end(key)
            else:
                if len(self._cache) >= self.max_size:
                    oldest_key, _ = self._cache.popitem(last=False)
                    del self._timestamps[oldest_key]
            self._cache[key] = value
            self._timestamps[key] = time.time()

    @property
    def hit_rate(self) -> float:
        total = self.hits + self.misses
        return self.hits / total if total > 0 else 0.0

What do put() and hit_rate do?

• put() stores a key-value pair. If the cache is full, it evicts the least recently used item using popitem(last=False) which removes the oldest entry from the ordered dict.
• hit_rate is a property that calculates the percentage of lookups that found data in the cache vs going to the database. A 90% hit rate means 90% of reads skip the database.

• Save and confirm the cache class imported cleanly by running:

python3 -c "from system_design_simulator import LRUCache; print('LRUCache ready')"

You should see LRUCache ready with no errors.

Seeing an IndentationError in LRUCache?

• The put() and hit_rate must be indented one level inside the class (4 spaces), matching get().
• Make sure there is no gap or extra blank line between the end of get() and the start of put() that breaks the class scope.

Help me fix an indentation error in my LRUCache class.

Now add the stage function that demonstrates caching. This function includes both the caching strategies theory and the CAP Theorem theory, because distributed caches force you to choose between consistency and availability.

• Below stage_load_balancing(), add the stage_caching() function:

def stage_caching() -> None:
    print_section("STAGE 3: Caching Layer (5,000 - 100,000 Users)")
    display_architecture(3)
    print_theory("Caching Strategies", "Cache-aside, write-through, write-back, write-around. LRU/LFU/TTL eviction.")
    print_theory("CAP Theorem", "C+A+P: choose CP or AP during partition. Redis=AP, ZooKeeper=CP.")
    server = SimulatedServer("app", max_concurrent=20, base_latency_ms=40)
    cache = LRUCache(max_size=200, ttl_seconds=30.0)
    metrics_no_cache = run_load_test(server.handle_request, num_requests=500, max_workers=50)
    display_metrics(metrics_no_cache, "No Cache - All DB Hits")
    popular_keys = [f"item:{i}" for i in range(20)]
    def cached_request():
        key = random.choice(popular_keys)
        cached = cache.get(key)
        if cached:
            return RequestResult(latency_ms=random.uniform(0.5, 2.0), success=True, from_cache=True)
        result = server.handle_request()
        if result.success:
            cache.put(key, f"data_for_{key}")
        return result
    metrics_cached = run_load_test(cached_request, num_requests=500, max_workers=50)
    display_metrics(metrics_cached, "With Cache")
    print(f"\n  >> Cache Hit Rate: {cache.hit_rate * 100:.1f}%")
    improvement = ((metrics_no_cache.p95_latency_ms - metrics_cached.p95_latency_ms) / metrics_no_cache.p95_latency_ms * 100)
    print(f"  >> P95 Latency Improvement: {improvement:.1f}%")
    print_lead_engineer("Redis cluster ~$200/month. 90% hit rate = 72% DB load reduction.")

What does the caching simulation demonstrate?

The function runs the same 500 requests twice. First without a cache (every request hits the slow server). Then with the LRUCache in front.

The cached_request() inner function checks the cache first. On a hit, it returns in under 2ms instead of the server's 40ms+ base latency. On a miss, it fetches from the server and stores the result for next time. The 20 popular keys simulate real access patterns where a small set of data is requested repeatedly.

• In the main() function, find this line:

        elif choice == "3":
            pass

• Replace pass with stage_caching(), so it looks like this:

        elif choice == "3":
            stage_caching()

• Save system_design_simulator.py.

Before you run Stage 3, predict: do you think the cache will reduce p95 latency by more or less than 50%? Why?

• Run the simulator and select Stage 3 by entering 3:

python3 system_design_simulator.py

You should see the caching theory print, then two metric blocks. The "With Cache" metrics should show p95 latency dropping by 80% or more compared to "No Cache". The cache hit rate should climb above 90% because 500 requests across only 20 keys means most are repeated.

Cache hit rate showing 0%?

• Make sure cache.put(key, f"data_for_{key}") is inside the if result.success: block of cached_request(). If the put never runs, nothing is cached.
• Confirm that cache = LRUCache(max_size=200, ttl_seconds=30.0) is defined before the first run_load_test call, not after it.

Why is my cache hit rate 0% even though I added the LRUCache?

Why does CAP theorem appear in the caching stage?

The moment you add a distributed cache like Redis, you have data in two places (cache and database). During a network partition, you must choose: reject requests until the cache is consistent (CP), or serve potentially stale cached data to stay available (AP).

In interviews, the Lead Engineer answer is: "CAP is a spectrum, not a binary. I choose consistency per operation. User login = strong consistency. Like count = eventual consistency."

✔️ Awesome, I've got everything!

Great. Double check that both Stages 2 and 3 run without errors and show metrics.

ⓧ I'd like to double check the full code

Scroll up and confirm you have each of these sections in order:

• The LoadBalancer class (after SimulatedServer, before LRUCache).
• The LRUCache class with __init__, get(), put(), and hit_rate (after LoadBalancer, before run_load_test).
• The stage_load_balancing() function (after stage_single_server()).
• The stage_caching() function (after stage_load_balancing()).
• In main(): choice "2" calls stage_load_balancing() and choice "3" calls stage_caching().

You now have two working scaling stages. Stage 2 proves that spreading load across servers reduces latency. Stage 3 proves that caching repeated reads drops latency by an order of magnitude. Next up, you will tackle the database itself with replication and sharding.

Implement Database Replication and Sharding

Your caching layer brought read latency down dramatically, but there is still a fundamental bottleneck lurking underneath.

Write-heavy workloads still funnel through a single database. Read replicas distribute read traffic, but when writes overwhelm the primary, sharding becomes the nuclear option. In this step, you will build both patterns and see exactly how consistent hashing prevents catastrophic key redistribution when you add or remove nodes.

In this step, get ready to:

• Build a ConsistentHashRing class that distributes keys evenly using virtual nodes.
• Build a ReplicatedDatabase class that routes reads to replicas and writes to a primary.
• Run Stage 4 and watch consistent hashing distribute 10,000 keys with minimal redistribution when a shard is added.

Add the ConsistentHashRing class

Naive modulo-based sharding (key % N) breaks when you add or remove a server. Almost every key gets remapped. Consistent hashing solves this by placing nodes on a virtual ring so only a fraction of keys move.

• In system_design_simulator.py, find the end of the LRUCache class (after the hit_rate property).
• Add the first part of the ConsistentHashRing class below it by copying this code:

class ConsistentHashRing:
    def __init__(self, nodes: list[str], virtual_nodes: int = 150):
        self.virtual_nodes = virtual_nodes
        self.ring: dict[int, str] = {}
        self.sorted_keys: list[int] = []
        self._key_distribution: dict[str, int] = {}
        for node in nodes:
            self.add_node(node)

    def _hash(self, key: str) -> int:
        return int(hashlib.md5(key.encode()).hexdigest(), 16)

    def add_node(self, node: str) -> None:
        self._key_distribution[node] = 0
        for i in range(self.virtual_nodes):
            virtual_key = f"{node}:v{i}"
            hash_val = self._hash(virtual_key)
            self.ring[hash_val] = node
        self.sorted_keys = sorted(self.ring.keys())

    def remove_node(self, node: str) -> None:
        for i in range(self.virtual_nodes):
            virtual_key = f"{node}:v{i}"
            hash_val = self._hash(virtual_key)
            if hash_val in self.ring:
                del self.ring[hash_val]
        self.sorted_keys = sorted(self.ring.keys())
        if node in self._key_distribution:
            del self._key_distribution[node]

What does this code do?

Each physical node gets 150 virtual positions on the ring. This ensures even key distribution even with a small number of physical nodes.

• _hash() converts any string into a large integer using MD5. This positions keys and nodes on the same ring.
• add_node() creates 150 virtual entries per node. More virtual nodes means smoother distribution.
• remove_node() deletes all virtual entries for a node. Only keys that belonged to that node need remapping.

• Immediately below the code you just added, add the remaining methods that look up which node owns a key:

    def get_node(self, key: str) -> str:
        if not self.ring:
            raise ValueError("Hash ring is empty")
        hash_val = self._hash(key)
        for ring_key in self.sorted_keys:
            if hash_val <= ring_key:
                node = self.ring[ring_key]
                self._key_distribution[node] = self._key_distribution.get(node, 0) + 1
                return node
        node = self.ring[self.sorted_keys[0]]
        self._key_distribution[node] = self._key_distribution.get(node, 0) + 1
        return node

    def get_distribution(self) -> dict[str, int]:
        return dict(self._key_distribution)

How does get_node work?

It hashes the key, then walks clockwise around the ring (through sorted keys) until it finds the first node position that is greater than or equal to the key's hash. If no such position exists (the key hashes past the last node), it wraps around to the first node on the ring.

• Save system_design_simulator.py.
• Confirm the file still imports cleanly by running:

python3 -c "import system_design_simulator; print('OK')"

You should see OK printed with no errors.

Seeing a SyntaxError or IndentationError?

• Make sure the get_node and get_distribution methods are indented with 4 spaces inside the ConsistentHashRing class body.
• Check that you placed the new class after the LRUCache class and before the run_load_test() function. Inserting it inside another class or function causes errors.
• Help me debug my ConsistentHashRing import error.

Add the ReplicatedDatabase class

A replicated database separates reads from writes. The primary handles all writes, while read replicas absorb the 70%+ of traffic that is read-only.

• Directly below the ConsistentHashRing class you just finished, add the ReplicatedDatabase class:

class ReplicatedDatabase:
    def __init__(self, num_replicas: int = 2):
        self.primary = SimulatedServer("primary-db", max_concurrent=20, base_latency_ms=30.0)
        self.replicas = [
            SimulatedServer(f"replica-{i}", max_concurrent=20, base_latency_ms=25.0)
            for i in range(num_replicas)
        ]
        self._read_index = 0
        self._lock = threading.Lock()

    def write(self, timeout: float = 5.0) -> RequestResult:
        return self.primary.handle_request(timeout)

    def read(self, timeout: float = 5.0) -> RequestResult:
        with self._lock:
            replica = self.replicas[self._read_index % len(self.replicas)]
            self._read_index += 1
        return replica.handle_request(timeout)

What does this code do?

• The primary server has a slightly higher base latency (30ms) than replicas (25ms) because it also handles replication overhead in a real system.
• read() round-robins across replicas using a lock-protected index. This distributes read load evenly.
• write() always routes to the primary. In a real system, writes propagate to replicas asynchronously.

• Save system_design_simulator.py.
• Verify the import still works by running:

python3 -c "import system_design_simulator; print('OK')"

You should see OK again with no errors.

Getting a NameError for SimulatedServer?

• Make sure ReplicatedDatabase is placed AFTER the SimulatedServer class in the file. Python reads top to bottom, so the class must exist before it is referenced.
• Help me fix my ReplicatedDatabase class error.

Build and run Stage 4

Now you have the building blocks. The stage function ties them together: it prints theory, simulates a replicated database under a 70/30 read/write workload, and demonstrates consistent hashing distributing 10,000 keys.

• Below the stage_caching() function and above main(), add the stage_database_scaling() function:

def stage_database_scaling() -> None:
    print_section("STAGE 4: DB Replication & Sharding (100K - 500K Users)")
    display_architecture(4)
    print_theory("Database Replication", "Leader-follower. Reads to replicas. Replication lag.")
    print_theory("Database Sharding", "Range, hash, directory, geo. Consistent hashing.")
    print_theory("Consistency Models", "Strong, eventual, causal, read-your-writes.")
    single_db = SimulatedServer("single-db", max_concurrent=15, base_latency_ms=30)
    display_metrics(run_load_test(single_db.handle_request, num_requests=500, max_workers=50), "Single DB")
    rep_db = ReplicatedDatabase(num_replicas=2)
    def replicated_request():
        if random.random() < 0.7:
            return rep_db.read()
        return rep_db.write()
    display_metrics(run_load_test(replicated_request, num_requests=500, max_workers=50), "Replicated DB")
    ring = ConsistentHashRing(["shard-1", "shard-2", "shard-3", "shard-4"])
    for i in range(10000):
        ring.get_node(f"user:{i}")
    dist = ring.get_distribution()
    print("\n  Key distribution across 4 shards (10,000 keys):")
    for shard, count in sorted(dist.items()):
        bar = "#" * (count // 100)
        print(f"    {shard}: {count:5d} keys  {bar}")
    ring_new = ConsistentHashRing(["shard-1", "shard-2", "shard-3", "shard-4", "shard-5"])
    moved = sum(1 for i in range(10000) if ring.get_node(f"user:{i}") != ring_new.get_node(f"user:{i}"))
    print(f"  Keys redistributed: {moved}/10000 ({moved/100:.1f}%)")
    print_lead_engineer("Exhaust caching and replicas before sharding.")

What does this function do?

• It first prints theory for replication, sharding strategies, and consistency models.
• Then it runs 500 requests against a single database, followed by 500 requests against a replicated database with a 70/30 read/write split. You will see latency improve because reads go to faster replicas.
• Finally it creates a consistent hash ring with 4 shards, distributes 10,000 keys, then adds a 5th shard to show that far fewer than 50% of keys get redistributed.

• In the main() function, find this line:

        elif choice == "4":
            pass

• Replace pass with stage_database_scaling() so the line now looks like this:

        elif choice == "4":
            stage_database_scaling()

• Save system_design_simulator.py.

Before you run the program and select Stage 4, predict what will happen to latency when reads are distributed across replicas instead of hitting a single database. Will it improve by 10%? 50%? More?

• Run the simulator:

python3 system_design_simulator.py

• Type 4 and press Enter to run Stage 4.

You should see the replication simulation show lower latency for the replicated DB compared to the single DB. Then the consistent hashing output shows 10,000 keys distributed roughly evenly across 4 shards, with less than 25% of keys redistributed when shard-5 is added.

Why is less than 25% redistribution significant?

With naive modulo hashing, adding a 5th node to 4 nodes would remap roughly 80% of all keys. Consistent hashing keeps it close to the ideal 20% (K/N, where K is total keys and N is new node count). This means adding capacity does not require a massive data migration.

Seeing a NameError for ConsistentHashRing or ReplicatedDatabase?

• Both classes must appear in the file BEFORE stage_database_scaling(). Python executes top-to-bottom, so the class definitions must come first.
• Check that both classes are at the top level (not indented inside another class or function).
• Help me debug my Stage 4 NameError.

✔️ Awesome, I've got everything!

Great. Double check you have saved your file.

ⓧ I'd like to double check the full code

Your file is over 30 lines at this point. Scroll through and confirm these sections exist in order:

• Imports and data classes (RequestResult, MetricsResult).
• MetricsCollector, SimulatedServer, LoadBalancer, LRUCache classes.
• ConsistentHashRing class with __init__, _hash, add_node, remove_node, get_node, get_distribution.
• ReplicatedDatabase class with write() and read() methods.
• run_load_test() function and all display helpers.
• stage_single_server(), stage_load_balancing(), stage_caching(), stage_database_scaling() functions.
• main() routing choice "4" to stage_database_scaling().

Your simulator now demonstrates the full database scaling progression: from a single bottlenecked database, to read replicas distributing load, to sharding with consistent hashing for write scalability. Next, you will add async processing, rate limiting, and circuit breakers to handle traffic spikes and protect against cascading failures.

Build Async Processing, Rate Limiting, and the Full Journey

Your system can now handle reads at scale with caching, replicas, and sharding. But what happens when a flash sale sends 50x normal traffic in a 30-minute burst? Synchronous processing collapses under spikes because every request ties up a thread waiting for work to finish.

In this step, you will build three protection patterns that handle traffic spikes and cascading failures. You will add a message queue to absorb bursts, a rate limiter to throttle excess traffic, and a circuit breaker to stop cascading failures. Then you will run the complete scaling journey from 100 to 1,000,000 users.

In this step, get ready to:

• Build a message queue that absorbs a 500-message traffic spike without dropping requests.
• Implement rate limiting and circuit breaker patterns that protect your system from abuse and dependency failures.
• Run all scaling stages sequentially and see a comparison table of metrics at each scale point.

Add the MessageQueue class

A message queue decouples request acceptance from processing. Producers push messages instantly, and worker threads consume them at a sustainable rate.

• In system_design_simulator.py, find the ReplicatedDatabase class. Add the following MessageQueue class below it:

# =============================================================================
# MESSAGE QUEUE
# =============================================================================

class MessageQueue:
    """Simulates an async message queue with workers."""

    def __init__(self, num_workers: int = 3, processing_time_ms: float = 50.0):
        self._queue: queue.Queue = queue.Queue(maxsize=10000)
        self._results: list[RequestResult] = []
        self._lock = threading.Lock()
        self._workers_running = False
        self.num_workers = num_workers
        self.processing_time_ms = processing_time_ms
        self.enqueued = 0
        self.processed = 0
        self.dropped = 0

    def enqueue(self, message: str) -> bool:
        try:
            self._queue.put_nowait(message)
            self.enqueued += 1
            return True
        except queue.Full:
            self.dropped += 1
            return False

What does this code do?

The queue has a max capacity of 10,000 messages. If you try to push beyond that, the message is dropped and the dropped counter increments. This simulates backpressure in real systems like Kafka or SQS.

The put_nowait() call is non-blocking. It either succeeds instantly or raises queue.Full. This lets the producer continue without waiting.

• Add the worker methods directly below the enqueue method, still inside the MessageQueue class:

    def _worker(self) -> None:
        while self._workers_running:
            try:
                msg = self._queue.get(timeout=0.1)
                jitter = random.uniform(0.5, 1.5)
                latency = self.processing_time_ms * jitter
                time.sleep(latency / 1000.0)
                with self._lock:
                    self._results.append(
                        RequestResult(latency_ms=latency, success=True)
                    )
                    self.processed += 1
                self._queue.task_done()
            except queue.Empty:
                continue

    def start_workers(self) -> None:
        self._workers_running = True
        self._worker_threads = []
        for i in range(self.num_workers):
            t = threading.Thread(target=self._worker, daemon=True)
            t.start()
            self._worker_threads.append(t)

    def stop_workers(self) -> None:
        self._workers_running = False
        for t in self._worker_threads:
            t.join(timeout=2.0)

    @property
    def queue_depth(self) -> int:
        return self._queue.qsize()

What does this code do?

• Each worker thread loops, pulling messages from the queue with a 0.1s timeout. If the queue is empty, it just continues the loop.
• Processing time uses random jitter to simulate real-world variance. The lock protects the shared results list from race conditions.
• Workers are daemon threads, so they automatically exit when the main program ends. The stop_workers() method cleanly shuts them down.

• Save system_design_simulator.py.
• Run a quick syntax check by running:

python3 system_design_simulator.py

• You should see the banner menu. Press Q to quit. If you see a syntax error, check the indentation of your new class.

Seeing an IndentationError?

Make sure the worker methods are indented with 4 spaces inside the MessageQueue class. The def _worker line should be at the same indent level as def enqueue.

help me fix an IndentationError in my MessageQueue class

Add rate limiting and circuit breaker classes

A rate limiter controls how many requests a client can make per second. A circuit breaker stops calls to a failing dependency so it can recover instead of dragging your entire system down.

• Below the MessageQueue class, add the TokenBucketRateLimiter class:

# =============================================================================
# RATE LIMITER (Token Bucket)
# =============================================================================

class TokenBucketRateLimiter:
    """Token bucket rate limiter."""

    def __init__(self, rate: float, capacity: int):
        self.rate = rate  # tokens per second
        self.capacity = capacity
        self.tokens = capacity
        self._last_refill = time.time()
        self._lock = threading.Lock()
        self.allowed = 0
        self.rejected = 0

    def allow_request(self) -> bool:
        with self._lock:
            now = time.time()
            elapsed = now - self._last_refill
            self.tokens = min(self.capacity, self.tokens + elapsed * self.rate)
            self._last_refill = now

            if self.tokens >= 1:
                self.tokens -= 1
                self.allowed += 1
                return True
            self.rejected += 1
            return False

What does this code do?

The token bucket algorithm works like a bucket that refills at a steady rate. Each request costs one token. If the bucket is empty, the request is rejected. This allows short bursts (up to the bucket capacity) while enforcing a long-term rate limit.

AWS API Gateway and Cloudflare use token bucket internally. It is the most common rate limiting algorithm in production systems.

• Below TokenBucketRateLimiter, add the first part of the CircuitBreaker class with __init__ and the call method:

# =============================================================================
# CIRCUIT BREAKER
# =============================================================================

class CircuitBreaker:
    """Circuit breaker with closed/open/half-open states."""

    def __init__(self, failure_threshold: int = 5, recovery_timeout: float = 3.0):
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        self.state = "closed"
        self._failure_count = 0
        self._last_failure_time = 0.0
        self._lock = threading.Lock()
        self.trips = 0

    def call(self, func, *args, **kwargs) -> RequestResult:
        with self._lock:
            if self.state == "open":
                if time.time() - self._last_failure_time > self.recovery_timeout:
                    self.state = "half_open"
                else:
                    return RequestResult(latency_ms=1.0, success=False)

• Continue the call method and add the _record_failure helper directly below:

        try:
            result = func(*args, **kwargs)
            if result.success:
                with self._lock:
                    self._failure_count = 0
                    if self.state == "half_open":
                        self.state = "closed"
            else:
                self._record_failure()
            return result
        except Exception:
            self._record_failure()
            return RequestResult(latency_ms=0, success=False)

    def _record_failure(self) -> None:
        with self._lock:
            self._failure_count += 1
            self._last_failure_time = time.time()
            if self._failure_count >= self.failure_threshold:
                self.state = "open"
                self.trips += 1

How does the circuit breaker work?

• Closed (normal): requests pass through. If failures hit the threshold (5), the circuit trips open.
• Open (tripped): all requests are rejected instantly (1ms latency, no actual call). This protects the failing service from further load.
• Half-open (testing): after the recovery timeout, one request is allowed through. If it succeeds, the circuit closes. If it fails, it opens again.

• Save system_design_simulator.py and run it to confirm no syntax errors:

python3 system_design_simulator.py

• The banner menu should appear. Press Q to exit.

Seeing a SyntaxError or NameError?

Make sure the try: block in call() is indented 8 spaces (two levels inside the method). The _record_failure method should be at the same indent as call (4 spaces inside the class).

help me fix my CircuitBreaker class

Build the stage 5 simulation and full journey

Now you will wire these three classes into a simulation that shows each protection pattern in action. Then you will build the full journey that runs all stages sequentially.

• Find the stage_database_scaling() function. Add the stage_async_and_protection() function below it:

# =============================================================================
# STAGE 5: ASYNC PROCESSING & RATE LIMITING
# =============================================================================

def stage_async_and_protection() -> None:
    print_section("STAGE 5: Async Processing & Rate Limiting (500K - 1M Users)")
    display_architecture(5)
    print_theory("Message Queues & Event-Driven Architecture", "Decouple producers from consumers. Absorb spikes. DLQ for failures.")
    print_theory("Rate Limiting & Circuit Breakers", "Token bucket, sliding window. CB states: closed/open/half-open.")
    print("\n  [SIMULATION] Message queue absorbing traffic spike...")
    mq = MessageQueue(num_workers=3, processing_time_ms=30)
    mq.start_workers()
    print("\n  Sending 500 messages in a burst (simulating viral traffic)...")
    burst_start = time.perf_counter()
    for i in range(500):
        mq.enqueue(f"task-{i}")
    burst_duration = time.perf_counter() - burst_start
    print(f"  Enqueue time: {burst_duration*1000:.1f}ms (burst absorbed instantly)")
    print(f"  Queue depth after burst: {mq.queue_depth}")
    print("  Waiting for workers to process...")
    time.sleep(3)
    print(f"  Processed: {mq.processed}/500")
    print(f"  Dropped: {mq.dropped}")
    mq.stop_workers()

What does this simulation show?

500 messages flood in almost instantly (the enqueue loop completes in milliseconds). The 3 worker threads then process them steadily over 3 seconds. Zero messages are dropped because the queue capacity (10,000) far exceeds the burst size.

This is how real systems handle viral traffic spikes. Accept immediately, process at sustainable rate.

• Continue stage_async_and_protection() by adding the rate limiting and circuit breaker simulations directly below the mq.stop_workers() line:

    print("\n  [SIMULATION] Rate limiting (100 req/s limit)...")
    limiter = TokenBucketRateLimiter(rate=100, capacity=100)
    allowed = 0
    rejected = 0
    for _ in range(300):
        if limiter.allow_request():
            allowed += 1
        else:
            rejected += 1
    print(f"  300 requests sent instantly:")
    print(f"    Allowed: {allowed}")
    print(f"    Rejected: {rejected}")
    print("\n  [SIMULATION] Circuit breaker protecting from failing service...")
    def failing_service() -> RequestResult:
        if random.random() < 0.8:
            return RequestResult(latency_ms=200, success=False)
        return RequestResult(latency_ms=50, success=True)
    cb = CircuitBreaker(failure_threshold=5, recovery_timeout=1.0)
    for i in range(20):
        result = cb.call(failing_service)
        if i % 5 == 0:
            print(f"    Request {i:2d}: state={cb.state:10s} success={result.success}")
    print(f"  Circuit breaker tripped {cb.trips} time(s)")
    print_theory("Remaining Lead Engineer Concepts", "CDN, Observability (RED/SLIs/SLOs), API Design (REST/gRPC/GraphQL), Security (Zero Trust/mTLS), AI/ML Infrastructure.")
    print_lead_engineer("At 1M users: CDN + API Gateway + LB + Cache + Queue + Circuit Breakers + Observability.")

What does this code do?

• The rate limiter starts with 100 tokens. 300 requests fire instantly. The first ~100 succeed (they drain the bucket), and the rest are rejected because tokens refill at 100/second but all 300 arrive in the same instant.
• The circuit breaker calls a service with an 80% failure rate. After 5 failures, it trips open and fast-rejects all subsequent requests (1ms latency instead of waiting 200ms for a guaranteed failure).

• Now wire Stage 5 into the menu. Find the main() function and locate this line:

        elif choice == "5":
            pass

• Replace pass with stage_async_and_protection(), so the lines now look like:

        elif choice == "5":
            stage_async_and_protection()

• Save system_design_simulator.py and run the program:

python3 system_design_simulator.py

• Select 5 from the menu. Before you press Enter, predict: will the queue drop any of the 500 messages?

You should see the queue absorb all 500 messages instantly (enqueue time under 5ms), then workers process them over 3 seconds with 0 dropped. The rate limiter allows about 100 of 300 instant requests. The circuit breaker trips after 5 failures and begins fast-rejecting.

Stage 5 hangs or workers never finish?

The simulation includes a time.sleep(3) to give workers time to process. If the program hangs beyond that, check that mq.stop_workers() is called after the sleep. Without it, worker threads keep looping indefinitely.

help me debug my stage 5 simulation

• Now add the run_full_journey() function below stage_async_and_protection(). This runs all scaling stages sequentially with increasing load:

# =============================================================================
# FULL JOURNEY
# =============================================================================

def run_full_journey() -> None:
    print_section("FULL SCALING JOURNEY: 100 -> 1,000,000 Users")
    results = []
    print("\n  [Stage 1] Single server, 200 requests...")
    server = SimulatedServer("solo", max_concurrent=10, base_latency_ms=20)
    m = run_load_test(server.handle_request, num_requests=200, max_workers=30)
    results.append(("Single Server (200 users)", m))
    print("  [Stage 2] Load balanced (3 servers), 500 requests...")
    servers = [SimulatedServer(f"app-{i}", max_concurrent=10, base_latency_ms=20) for i in range(3)]
    lb = LoadBalancer(servers)
    m = run_load_test(lambda: lb.get_server().handle_request(), num_requests=500, max_workers=50)
    results.append(("Load Balanced (500 users)", m))
    print("  [Stage 3] Cached, 800 requests...")
    cache = LRUCache(max_size=100, ttl_seconds=30)
    popular = [f"item:{i}" for i in range(15)]
    def cached_req():
        key = random.choice(popular)
        if cache.get(key):
            return RequestResult(latency_ms=random.uniform(1, 3), success=True, from_cache=True)
        result = lb.get_server().handle_request()
        if result.success:
            cache.put(key, "data")
        return result
    m = run_load_test(cached_req, num_requests=800, max_workers=50)
    results.append(("+ Cache Layer (800 users)", m))

• Continue run_full_journey() by adding stages 4-5 and the summary table directly below:

    print("  [Stage 4] Replicated DB, 1000 requests...")
    rep_db = ReplicatedDatabase(num_replicas=3)
    def rep_req():
        key = random.choice(popular)
        if cache.get(key):
            return RequestResult(latency_ms=random.uniform(1, 3), success=True, from_cache=True)
        if random.random() < 0.7:
            result = rep_db.read()
        else:
            result = rep_db.write()
        if result.success:
            cache.put(key, "data")
        return result
    m = run_load_test(rep_req, num_requests=1000, max_workers=80)
    results.append(("+ DB Replicas (1000 users)", m))
    print("  [Stage 5] Full stack + rate limiting, 1500 requests...")
    rl = TokenBucketRateLimiter(rate=500, capacity=500)
    def full_stack_req():
        if not rl.allow_request():
            return RequestResult(latency_ms=1, success=False)
        return rep_req()
    m = run_load_test(full_stack_req, num_requests=1500, max_workers=100)
    results.append(("+ Rate Limiting (1500 users)", m))

• Finally, add the summary table that prints the comparison. Still inside run_full_journey(), add:

    print("\n" + "=" * 70)
    print("  SCALING JOURNEY SUMMARY")
    print("=" * 70)
    print(f"  {'Stage':<32} {'p50':>6} {'p95':>6} {'p99':>6} {'RPS':>7} {'Err%':>5}")
    print(f"  {'-'*32} {'-'*6} {'-'*6} {'-'*6} {'-'*7} {'-'*5}")
    for label, m in results:
        print(f"  {label:<32} {m.p50_latency_ms:>5.0f}ms {m.p95_latency_ms:>5.0f}ms {m.p99_latency_ms:>5.0f}ms {m.throughput_rps:>6.0f} {m.error_rate*100:>4.1f}%")
    print("\n  Key insight: Each layer addresses a specific bottleneck.")

What does the full journey show?

The comparison table reveals how each scaling pattern improves specific metrics. Load balancing reduces latency through distribution. Caching slashes p95 by serving from RAM. Replicas handle more reads. Rate limiting introduces controlled errors (rejected requests) to protect the system from total collapse.

• Below run_full_journey(), add the theory_reference() function that serves as a quick-reference guide:

# =============================================================================
# THEORY REFERENCE
# =============================================================================

def theory_reference() -> None:
    print_section("THEORY REFERENCE: All Concepts At A Glance")
    print("""
  SCALING JOURNEY DECISION TREE:
  Users      | Bottleneck         | Solution            | Stage
  0-1K       | None               | Single server       | 1
  1K-5K      | Server CPU/memory  | Load balancer       | 2
  5K-100K    | Database reads     | Cache (Redis)       | 3
  100K-500K  | DB reads + writes  | Replicas + Sharding | 4
  500K-1M    | Spikes + failures  | Queues + Protection | 5

  INTERVIEW FRAMEWORK (45 minutes):
  [0:00-0:05] Clarify requirements + back-of-envelope estimation
  [0:05-0:08] Define API and data model
  [0:08-0:25] High-level design
  [0:25-0:40] Deep dive (1-2 components)
  [0:40-0:45] Scaling, failure modes, monitoring, cost
    """)

• Now update the remaining main() stubs. Find these lines:

        elif choice == "6":
            pass
        elif choice == "T":
            pass
        elif choice == "Q":
            print("\n  Good luck in your interview!\n")
            break

• Replace those lines with the complete routing:

        elif choice == "6":
            run_full_journey()
        elif choice == "T":
            theory_reference()
        elif choice == "Q":
            print("\n  Good luck in your interview! You've got this.")
            print("  Remember: Drive the discussion. Mention trade-offs unprompted.")
            print("  State what you would NOT build. Mention cost and failure modes.\n")
            break

• Save system_design_simulator.py.
• Run the program and select the full journey:

python3 system_design_simulator.py

• Select 6 from the menu. Before you press Enter, predict: which stage will have the lowest error rate in the summary table?

You should see all 5 stages execute sequentially, each with its own load test. The final summary table shows p50, p95, p99 latency, throughput, and error rate for each scaling level. Watch how each pattern addresses a specific bottleneck.

• Try T to see the theory reference decision tree, then Q to quit. You should see the encouraging exit message with interview tips.

Seeing a NameError for run_full_journey or theory_reference?

Make sure both functions are defined ABOVE the main() function in your file. Python reads top-to-bottom, so functions must be defined before they are called.

help me fix NameError in my simulator

✔️ Awesome, I've got everything!

Great. Double check you have saved your file. Your program should now handle all menu options: stages 1-5, the full journey (6), theory reference (T), and quit (Q).

ⓧ I'd like to double check the full code

Your file is now large, so scroll through and confirm each section is present in this order:

• Imports and data classes (RequestResult, MetricsResult).
• MetricsCollector class.
• SimulatedServer class.
• LoadBalancer class.
• LRUCache class.
• ConsistentHashRing class.
• ReplicatedDatabase class.
• MessageQueue class (new this step).
• TokenBucketRateLimiter class (new this step).
• CircuitBreaker class (new this step).
• run_load_test() function.
• Display helpers (display_banner, display_metrics, display_architecture, print_section, print_theory, print_lead_engineer).
• stage_single_server(), stage_load_balancing(), stage_caching(), stage_database_scaling().
• stage_async_and_protection() (new this step).
• run_full_journey() (new this step).
• theory_reference() (new this step).
• main() with choices 1-6, T, and Q all routing to their functions (no pass stubs remaining).

Your simulator is now fully functional. You can walk through every scaling stage, see real metrics degrade and recover, and run the complete journey from 100 to 1,000,000 users with a comparison table. Next up: an optional Secret Mission to add an interactive quiz mode that tests your pattern-selection skills under pressure.

Secret mission

Add an Interactive Quiz Mode

Passively reading theory does not stick. Active recall (being asked 'what breaks next?' and having to answer before seeing the solution) is how you internalize patterns under interview pressure. Build an interactive quiz system that presents real system design scenarios and tests your ability to pick the right scaling pattern.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally with no external dependencies, cloud resources, or API calls. There are no ongoing costs.

Resources you used:

• system_design_simulator.py file on your Desktop.

Keep everything running

No action needed. The file stays on your machine as a pre-interview review tool. Run it again before your next system design interview to refresh the scaling patterns and Lead Engineer talking points.

Pause - I'll come back to this later

There is nothing to pause. The program only runs when you explicitly execute it. No background processes, no servers, no connections to shut down.

• The file sits idle on disk until you run it again.

Delete - I don't want to use this again

Remove the project file from your Desktop.

On macOS or Linux, run:

cd ~/Desktop
rm system_design_simulator.py

On Windows, run:

cd ~\Desktop
Remove-Item system_design_simulator.py

• Confirm the file is gone by running ls (macOS/Linux) or dir (Windows) in your Desktop directory. You should no longer see system_design_simulator.py listed.

Nice Work!

Nice Work!

You just built a complete system design scaling simulator that walks you through the journey from 100 to 1,000,000 users with real concurrency metrics at every stage.

You've learned how to:

• Simulate real traffic with Python's ThreadPoolExecutor and measure p50/p95/p99 latency, throughput, and error rates to observe exactly when and why a system breaks under load.
• Implement core distributed systems patterns including consistent hashing, LRU caching, load balancing, circuit breakers, and rate limiting in a single Python file with zero external dependencies.
• Articulate Lead Engineer talking points for every scaling decision including trade-offs, cost reasoning, failure modes, what NOT to build, and cloud service equivalents.
• Built an interactive quiz mode that tests your ability to pick the right scaling pattern for real system design scenarios under pressure (Secret Mission).

Ready to quiz yourself?