Design a 12 Million-Player Leaderboard

Introduction

30 Second Summary

A leaderboard feels effortless when a few friends are playing. With 12 million people online at once, a single crowded path can turn score updates into a traffic jam.

In this project, you will build a terminal-based leaderboard laboratory backed by Redis. You will use its measurements to justify partitioning in a production-scale AWS architecture with explicit failure behavior.

What You'll Build

Your finished terminal lab turns a 12-million-player requirement into reports you can use to expose a hot key, prove exact ranking behavior, and defend failure trade-offs.

By the end of this project, you'll have:

• A traffic report that calculates 240,000 peak score updates per second plus 100,000 leaderboard reads per second from explicit product assumptions.
• A Redis comparison where the first run sends 100 percent of writes to one key. The partitioned run proves the global top K plus the exact competition rank.
• An AWS architecture defense that explains how retries, cache loss, replay, and stale aggregation affect each leaderboard answer.
• Secret Mission: Add per-player sequence numbers so delayed older events cannot overwrite newer scores.

Are there any prerequisites?

Some backend and system design experience will help with the trade-offs. Step 1 checks your Python version plus your local Docker Desktop and Visual Studio Code setup.

The AWS portion remains architecture-only, so this project adds no cloud cost.

Before We Start

Before the experiments begin, choose the leaderboard guarantee you care about most. This commitment helps you separate measured local evidence from explicit production assumptions.

Set Up the Local Redis Lab

A useful benchmark needs a known runtime. You will pin the Redis server plus the redis-py client so version drift cannot distort later measurements.

Everything runs on your Mac through Docker Compose. This step creates no AWS resources.

In this step, get ready to:

• Verify that Python 3.10 or newer is available.
• Create an isolated environment with the pinned Redis client.
• Start the pinned Redis container and prove that Python can reach it.

Verify your Mac prerequisites

The client library requires Python 3.10 or newer. Checking the installed version now prevents a confusing dependency failure later.

• Press Cmd+Space to open Spotlight.
• Type Terminal and press Enter to open it.
• Check your installed Python version by running this command:

python3 --version

What does this command check?

The command prints the version attached to python3. The next tabs cover each common result.

✔️ I see version 3.10 or higher

Your Python version meets the requirement. You can continue with the installed interpreter.

ⓧ I see an older version

Your current interpreter is too old for redis-py 8.1.0. Install the current stable macOS release before continuing.

• Open the Python macOS downloads page in your browser.
• Select Download macOS installer for Python 3.14.7.
• Run the downloaded installer.
• Return to Terminal after the installation finishes.
• Confirm the newer version by running this command:

python3 --version

What should you see?

The output should report Python 3.10 or newer. Python 3.14.7 satisfies that requirement.

ⓧ Command not found

Your shell cannot find Python 3. Install the current stable macOS release to add the interpreter.

• Open the Python macOS downloads page in your browser.
• Select Download macOS installer for Python 3.14.7.
• Run the downloaded installer.
• Return to Terminal after the installation finishes.
• Verify the installation by running this command:

python3 --version

What should you see?

The output should now report Python 3.14.7 or another version newer than 3.10.

The local database runs inside Docker Desktop. Its container gives every experiment the same Redis version.

• Press Cmd+Space to open Spotlight.
• Type Docker Desktop and press Enter to open it.
• Wait for Docker Desktop to finish starting.

The first launch can take a minute or two. When startup finishes, Docker Desktop reports that its engine is running.

Docker Desktop still starting?

Keep Docker Desktop open until its startup status completes. A pending operating-system permission request can also block the engine.

Use help me start Docker Desktop on macOS if the status does not settle.

Create the isolated project and start Redis

A virtual environment keeps this project's Python package separate from packages used elsewhere on your Mac. The Compose file pins the server image so the local runtime stays reproducible.

• Return to the Terminal window from the Python version check.
• Create the leaderboard-lab folder on your Desktop by running these commands:

cd ~/Desktop
mkdir leaderboard-lab
cd leaderboard-lab

What do these commands do?

• The first command moves Terminal to your Desktop.
• The second command creates the leaderboard-lab folder.
• The third command makes that folder the current location.

• Create the isolated Python environment by running this command:

python3 -m venv .venv

What does this command create?

The command creates a .venv directory inside leaderboard-lab. Packages installed while it is active stay isolated inside this project.

• Activate the virtual environment by running this command:

source .venv/bin/activate

What does activation change?

Activation points Python package commands at the environment inside .venv. Your Terminal prompt should now begin with (.venv).

• Open the current folder in Visual Studio Code by running this command:

open -a "Visual Studio Code" .

What does this command open?

The open command launches Visual Studio Code. The final period selects the current leaderboard-lab folder.

You should see leaderboard-lab at the top of the Visual Studio Code Explorer sidebar.

The first runtime file pins the exact Python package used by the lab. This makes a fresh installation produce the same client version.

• Click the New File button in the Visual Studio Code Explorer sidebar.
• Enter requirements.txt as the file name.
• Add the package pin by copying this content into requirements.txt:

redis==8.1.0

What does this package pin do?

The file tells Python to install redis-py version 8.1.0. The exact pin keeps the client API consistent throughout the lab.

• Save requirements.txt.
• Confirm that requirements.txt appears in the Explorer sidebar.

File missing from the Explorer?

Check that Visual Studio Code has the leaderboard-lab folder open. Make sure the file name ends with .txt.

Use help me create requirements.txt in Visual Studio Code if the file still does not appear.

The second runtime file defines the Redis container. Its health check lets Docker test whether the server responds before the Python lab connects.

• Click the New File button in the Explorer sidebar.
• Enter compose.yaml as the file name.
• Define the local Redis service by copying this content into compose.yaml:

services:
  redis:
    image: redis:8.10.1-alpine
    ports:
      - "6379:6379"
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5
      start_period: 5s

What does this configuration do?

• The image pin selects Redis 8.10.1 on Alpine Linux.
• The port mapping exposes Redis on localhost:6379.
• The health check asks Redis to respond to a ping before the lab treats it as ready.

• Save compose.yaml.
• Confirm that compose.yaml appears beside requirements.txt in the Explorer sidebar.

Compose file showing an error?

YAML spacing controls the structure. Compare the indentation under redis with the reference block.

Use help me check this Compose YAML if the editor still reports a problem.

• Return to the active Terminal window.
• Install the pinned Python dependency by running this command:

python3 -m pip install -r requirements.txt

What does this installation do?

The command reads requirements.txt and installs redis-py 8.1.0 inside the active .venv environment.

That dependency is now isolated inside your project. Your system-level Python packages remain untouched.

Dependency installation failed?

Check that the Terminal prompt starts with (.venv). Confirm that requirements.txt contains the exact package pin.

Use help me fix this redis-py installation if the installation still fails.

The first image download can take a few minutes. Docker is fetching the pinned Redis image during this wait.

• Start the Redis service in the background by running this command:

docker compose up -d

What does this command start?

Docker Compose creates the local Redis container from compose.yaml. Detached mode leaves it running after Terminal returns to the prompt.

You should see Docker report that the Redis service started. The container remains available for the experiments in later steps.

Redis service did not start?

Make sure Docker Desktop has finished starting. Check that no other local process is already using port 6379.

Use help me start this Redis Compose service if the service still fails.

Add the starter connection check

A running container is only useful when the client can reach it. The starter script connects to Redis and waits briefly for a successful response.

• Return to the Visual Studio Code window.
• Click the New File button in the Explorer sidebar.
• Enter leaderboard_lab.py as the file name.
• Add the imports plus the connection helpers by copying this first section into leaderboard_lab.py:

import argparse
import time

import redis
from redis.exceptions import RedisError

def connect() -> redis.Redis:
    return redis.Redis(
        host="localhost",
        port=6379,
        db=0,
        decode_responses=True,
        socket_connect_timeout=2,
        socket_timeout=2,
    )

def wait_for_redis(client: redis.Redis, attempts: int = 10) -> None:
    for _ in range(attempts):
        try:
            if client.ping():
                return
        except RedisError:
            time.sleep(1)
    raise RuntimeError("Redis did not become ready within 10 seconds.")

What does this code do?

• The imports provide command parsing plus retry timing. They also load the Redis client and its base error type.
• The connect() function targets the container on localhost:6379.
• The wait_for_redis() function retries a ping before reporting that the server failed to become ready.

• Save leaderboard_lab.py.
• Confirm that leaderboard_lab.py appears with the other two files in the Explorer sidebar.

Python file missing?

Check that the file is inside the open leaderboard-lab folder. Make sure its name uses an underscore before lab.

Use help me locate leaderboard_lab.py if it still does not appear.

• Place the command router immediately below the raise RuntimeError("Redis did not become ready within 10 seconds.") line by copying this section:

def run_setup(client: redis.Redis) -> None:
    wait_for_redis(client)
    print(f"Redis ping: {client.ping()}")
    print("Redis image: redis:8.10.1-alpine")
    print("redis-py version: 8.1.0")

def main() -> None:
    parser = argparse.ArgumentParser(description="Local leaderboard scaling and correctness lab")
    parser.add_argument("command", choices=("setup",))
    args = parser.parse_args()
    client = connect()
    if args.command == "setup":
        run_setup(client)

if __name__ == "__main__":
    main()

What does this code do?

• The run_setup() function waits for Redis before printing the connection result and pinned versions.
• The main() function accepts setup as the current command.
• The final condition runs main() when you execute the file directly.

• Save leaderboard_lab.py.

Before you run the check, do you expect the Python client to reach the container immediately?

• Return to the active Terminal window.
• Verify the complete local setup by running this command:

python3 leaderboard_lab.py setup

What should you see?

You should see Redis ping: True. This proves the Python client reached the local Redis server.

The next lines should report Redis image: redis:8.10.1-alpine plus redis-py version: 8.1.0.

That is the local lab online. Your pinned Python client can now send requests to the pinned Redis container.

Connection check failed?

Confirm that Docker Desktop is running. Check that you launched the Compose service from the leaderboard-lab folder.

Make sure the active Terminal prompt starts with (.venv). A Redis readiness error can also mean the container still needs a few seconds to pass its health check.

Use help me debug this Redis connection check if the check still fails.

✔️ Awesome, I've got everything!

Your three project files match the starter lab. Keep the Redis container running for the remaining experiments.

ⓧ I'd like to double check the full code

The complete requirements.txt file should match this reference.

redis==8.1.0

What to compare

Confirm that the package name uses lowercase letters. Check that the version has two equals signs before 8.1.0.

The complete compose.yaml file should match this reference.

services:
  redis:
    image: redis:8.10.1-alpine
    ports:
      - "6379:6379"
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5
      start_period: 5s

What to compare

Check the image tag plus the port mapping. Compare every level of indentation in the health check.

The complete leaderboard_lab.py file should match this reference.

import argparse
import time

import redis
from redis.exceptions import RedisError

def connect() -> redis.Redis:
    return redis.Redis(
        host="localhost",
        port=6379,
        db=0,
        decode_responses=True,
        socket_connect_timeout=2,
        socket_timeout=2,
    )

def wait_for_redis(client: redis.Redis, attempts: int = 10) -> None:
    for _ in range(attempts):
        try:
            if client.ping():
                return
        except RedisError:
            time.sleep(1)
    raise RuntimeError("Redis did not become ready within 10 seconds.")

def run_setup(client: redis.Redis) -> None:
    wait_for_redis(client)
    print(f"Redis ping: {client.ping()}")
    print("Redis image: redis:8.10.1-alpine")
    print("redis-py version: 8.1.0")

def main() -> None:
    parser = argparse.ArgumentParser(description="Local leaderboard scaling and correctness lab")
    parser.add_argument("command", choices=("setup",))
    args = parser.parse_args()
    client = connect()
    if args.command == "setup":
        run_setup(client)

if __name__ == "__main__":
    main()

What to compare

Check the indentation inside every function. Confirm that the command choice contains only setup at this stage.

Your reproducible Redis lab is ready. Next, you will turn 12 million concurrent players into explicit traffic and partition assumptions.

Model the 12 Million-Player Workload

Your local Redis lab now proves that the pinned client can reach the container. You have a stable place to test the leaderboard.

A requirement for 12 million concurrent players does not reveal the request volume. Player behavior determines the traffic.

You will encode those behavior assumptions in a workload model. The model calculates each traffic path plus the average partition size.

In this step, get ready to:

• Encode the fixed player requirement plus the adjustable product assumptions.
• Calculate update traffic plus leaderboard read traffic.
• Record the requirement boundary in the architecture document.

Encode the workload assumptions

The 12 million-player count is the fixed requirement. Every rate below it is an adjustable product assumption that needs telemetry during a real design review.

• In leaderboard_lab.py, replace the existing import section with this expanded import group:

import argparse
import time
from dataclasses import dataclass
from itertools import product

import redis
from redis.exceptions import RedisError

Why add these imports?

• The dataclass decorator gives the scenario a named structure with typed defaults.
• The product helper creates every region, mode, and season combination.

• Save leaderboard_lab.py.
• Confirm the expanded imports load by running the existing setup check:

python3 leaderboard_lab.py setup

What does this check prove?

The existing setup route still imports the file successfully. It also confirms that the local Redis connection remains healthy.

You should still see the successful Redis ping plus the pinned image and client versions.

Seeing an import error?

Check that both new import lines sit above import redis. Keep the virtual environment from the previous step active.

Help me fix the new imports in leaderboard_lab.py

A frozen Python dataclass keeps the baseline scenario together. Its values cannot change accidentally after the model is created.

• Add this class below the import section in leaderboard_lab.py and above connect():

@dataclass(frozen=True)
class WorkloadAssumptions:
    concurrent_players: int = 12_000_000
    update_fraction_per_second: float = 0.02
    leaderboard_reads_per_player_per_minute: float = 0.5
    top_k_share: float = 0.25
    around_me_share: float = 0.75
    regions: tuple[str, ...] = ("na-east", "na-west", "eu", "apac")
    modes: tuple[str, ...] = ("solo", "duo", "squad")
    active_seasons: tuple[str, ...] = ("s1",)
    hot_partition_multiplier: float = 2.0

What do these assumptions represent?

• The player count captures the fixed scale requirement.
• The update fraction models the share of players who submit a score during one peak second.
• The read rate models how often each player requests leaderboard data.
• The remaining values describe the read mix plus the proposed product partitions.

• In the existing connect() function, find this code:

def connect() -> redis.Redis:
    return redis.Redis(
        host="localhost",
        port=6379,
        db=0,
        decode_responses=True,
        socket_connect_timeout=2,
        socket_timeout=2,
    )

Why locate this function?

The connection settings already work. You are only compacting the same constructor call so the growing lab remains easier to scan.

• Replace the existing connect() function with this equivalent version:

def connect() -> redis.Redis:
    return redis.Redis(host="localhost", port=6379, db=0, decode_responses=True, socket_connect_timeout=2, socket_timeout=2)

What stays the same?

The client still connects to the same local host plus database. The response decoding and timeout behavior also remain unchanged.

• Save leaderboard_lab.py.
• Confirm the class and connection function load by running the setup check again:

python3 leaderboard_lab.py setup

What does this second check prove?

The new dataclass is valid Python. The compact connection function still reaches the running Redis container.

The setup report should match the successful output from the previous step.

Seeing a syntax error?

Check the indentation inside WorkloadAssumptions. Every field must have four leading spaces.

Help me fix the WorkloadAssumptions syntax

Derive the traffic and partition sizes

Each property turns an assumption into a calculated rate. This keeps the reported QPS tied to visible inputs instead of unexplained constants.

• Inside WorkloadAssumptions, add these properties directly below hot_partition_multiplier:

    @property
    def score_updates_per_second(self) -> int:
        return int(self.concurrent_players * self.update_fraction_per_second)

    @property
    def leaderboard_reads_per_second(self) -> int:
        return int(self.concurrent_players * self.leaderboard_reads_per_player_per_minute / 60)

    @property
    def top_k_reads_per_second(self) -> int:
        return int(self.leaderboard_reads_per_second * self.top_k_share)

    @property
    def around_me_reads_per_second(self) -> int:
        return int(self.leaderboard_reads_per_second * self.around_me_share)

    @property
    def active_partition_count(self) -> int:
        return len(self.regions) * len(self.modes) * len(self.active_seasons)

    @property
    def average_players_per_partition(self) -> int:
        return self.concurrent_players // self.active_partition_count

    @property
    def hot_partition_players(self) -> int:
        return int(self.average_players_per_partition * self.hot_partition_multiplier)

How are the results derived?

• Score-update QPS multiplies concurrent players by the peak update fraction.
• Read QPS converts the per-player minute rate into a per-second total.
• Top-K plus around-me QPS apply the read split to that total.
• Partition size divides players across every active region, mode, and season combination.

• Add the baseline instance plus generated partition keys immediately below the class:

ASSUMPTIONS = WorkloadAssumptions()
PARTITION_KEYS = tuple(
    f"lb:{region}:{mode}:{season}"
    for region, mode, season in product(
        ASSUMPTIONS.regions,
        ASSUMPTIONS.modes,
        ASSUMPTIONS.active_seasons,
    )
)

Why generate the keys now?

The Cartesian product produces 12 distinct keys from four regions, three modes, and one season. Later experiments can reuse the exact same partition boundary.

• Save leaderboard_lab.py.
• Confirm the derived properties plus generated keys load by running the setup check:

python3 leaderboard_lab.py setup

What does this checkpoint prove?

The model now loads without changing the existing setup behavior. This isolates syntax problems before you add the capacity report.

You should see the same three successful setup lines.

Does the file fail to load?

Make sure each property remains indented inside WorkloadAssumptions. Place ASSUMPTIONS at the left edge after the class ends.

Help me fix the workload model structure

The report needs to show both average load and planned skew. The 2x multiplier prevents an even-distribution assumption from hiding a hot partition.

• Add print_capacity_model() below run_setup() and above main():

def print_capacity_model() -> None:
    a = ASSUMPTIONS
    average_partition_updates = a.score_updates_per_second / a.active_partition_count
    hot_partition_updates = average_partition_updates * a.hot_partition_multiplier
    print("12 MILLION PLAYER WORKLOAD MODEL")
    print("Scenario assumptions, not measured production traffic")
    print(f"Concurrent players: {a.concurrent_players:,}")
    print(f"Peak score-update assumption: {a.update_fraction_per_second:.1%} of players per second")
    print(f"Leaderboard-read assumption: {a.leaderboard_reads_per_player_per_minute} reads/player/minute")
    print(f"Peak score updates/s: {a.score_updates_per_second:,}")
    print(f"Leaderboard reads/s: {a.leaderboard_reads_per_second:,}")
    print(f"Top-K reads/s: {a.top_k_reads_per_second:,}")
    print(f"Around-me reads/s: {a.around_me_reads_per_second:,}")
    print(f"Active partitions: {a.active_partition_count}")
    print(f"Average players/active partition: {a.average_players_per_partition:,}")
    print(f"Hot-partition planning size: {a.hot_partition_players:,} players")
    print(f"Average projected updates/partition/s: {average_partition_updates:,.0f}")
    print(f"Hot projected updates/partition/s: {hot_partition_updates:,.0f}")

What does the report show?

The report prints every baseline input beside its derived result. It labels the numbers as scenario assumptions so local output cannot be mistaken for production telemetry.

The average and hot-partition rates make skew visible before you choose a storage topology.

• Replace the existing main() function with this command router:

def main() -> None:
    parser = argparse.ArgumentParser(description="Local leaderboard scaling and correctness lab")
    parser.add_argument("command", choices=("setup", "capacity"))
    args = parser.parse_args()
    if args.command == "capacity":
        print_capacity_model()
        return
    run_setup(connect())

How does the router work?

• The command choices now accept both setup and capacity.
• The capacity route prints the model without opening a Redis connection.
• The setup route still checks the local container.

• Save leaderboard_lab.py.

Before you run the new command, which output do you expect to be larger: update QPS or total leaderboard read QPS?

• Print the baseline workload model by running:

python3 leaderboard_lab.py capacity

What should the baseline reveal?

The report shows 240,000 score updates per second plus 100,000 leaderboard reads per second. Those reads split into 25,000 top-K requests plus 75,000 around-me requests.

The 12 active partitions average 1,000,000 players each. The hot-partition planning size reaches 2,000,000 players.

Does the capacity command fail?

Check that print_capacity_model() appears above main(). Confirm that capacity appears inside the command choices.

Help me debug the capacity command

✔️ Awesome, I've got everything!

Great. Save leaderboard_lab.py before you record the requirement contract.

ⓧ I'd like to double check the full code

import argparse
import time
from dataclasses import dataclass
from itertools import product

import redis
from redis.exceptions import RedisError

@dataclass(frozen=True)
class WorkloadAssumptions:
    concurrent_players: int = 12_000_000
    update_fraction_per_second: float = 0.02
    leaderboard_reads_per_player_per_minute: float = 0.5
    top_k_share: float = 0.25
    around_me_share: float = 0.75
    regions: tuple[str, ...] = ("na-east", "na-west", "eu", "apac")
    modes: tuple[str, ...] = ("solo", "duo", "squad")
    active_seasons: tuple[str, ...] = ("s1",)
    hot_partition_multiplier: float = 2.0

    @property
    def score_updates_per_second(self) -> int:
        return int(self.concurrent_players * self.update_fraction_per_second)

    @property
    def leaderboard_reads_per_second(self) -> int:
        return int(self.concurrent_players * self.leaderboard_reads_per_player_per_minute / 60)

    @property
    def top_k_reads_per_second(self) -> int:
        return int(self.leaderboard_reads_per_second * self.top_k_share)

    @property
    def around_me_reads_per_second(self) -> int:
        return int(self.leaderboard_reads_per_second * self.around_me_share)

    @property
    def active_partition_count(self) -> int:
        return len(self.regions) * len(self.modes) * len(self.active_seasons)

    @property
    def average_players_per_partition(self) -> int:
        return self.concurrent_players // self.active_partition_count

    @property
    def hot_partition_players(self) -> int:
        return int(self.average_players_per_partition * self.hot_partition_multiplier)

ASSUMPTIONS = WorkloadAssumptions()
PARTITION_KEYS = tuple(
    f"lb:{region}:{mode}:{season}"
    for region, mode, season in product(
        ASSUMPTIONS.regions,
        ASSUMPTIONS.modes,
        ASSUMPTIONS.active_seasons,
    )
)

def connect() -> redis.Redis:
    return redis.Redis(host="localhost", port=6379, db=0, decode_responses=True, socket_connect_timeout=2, socket_timeout=2)

def wait_for_redis(client: redis.Redis, attempts: int = 10) -> None:
    for _ in range(attempts):
        try:
            if client.ping():
                return
        except RedisError:
            time.sleep(1)
    raise RuntimeError("Redis did not become ready within 10 seconds.")

def run_setup(client: redis.Redis) -> None:
    wait_for_redis(client)
    print(f"Redis ping: {client.ping()}")
    print("Redis image: redis:8.10.1-alpine")
    print("redis-py version: 8.1.0")

def print_capacity_model() -> None:
    a = ASSUMPTIONS
    average_partition_updates = a.score_updates_per_second / a.active_partition_count
    hot_partition_updates = average_partition_updates * a.hot_partition_multiplier
    print("12 MILLION PLAYER WORKLOAD MODEL")
    print("Scenario assumptions, not measured production traffic")
    print(f"Concurrent players: {a.concurrent_players:,}")
    print(f"Peak score-update assumption: {a.update_fraction_per_second:.1%} of players per second")
    print(f"Leaderboard-read assumption: {a.leaderboard_reads_per_player_per_minute} reads/player/minute")
    print(f"Peak score updates/s: {a.score_updates_per_second:,}")
    print(f"Leaderboard reads/s: {a.leaderboard_reads_per_second:,}")
    print(f"Top-K reads/s: {a.top_k_reads_per_second:,}")
    print(f"Around-me reads/s: {a.around_me_reads_per_second:,}")
    print(f"Active partitions: {a.active_partition_count}")
    print(f"Average players/active partition: {a.average_players_per_partition:,}")
    print(f"Hot-partition planning size: {a.hot_partition_players:,} players")
    print(f"Average projected updates/partition/s: {average_partition_updates:,.0f}")
    print(f"Hot projected updates/partition/s: {hot_partition_updates:,.0f}")

def main() -> None:
    parser = argparse.ArgumentParser(description="Local leaderboard scaling and correctness lab")
    parser.add_argument("command", choices=("setup", "capacity"))
    args = parser.parse_args()
    if args.command == "capacity":
        print_capacity_model()
        return
    run_setup(connect())

if __name__ == "__main__":
    main()

Record and challenge the requirement contract

A system-design estimate needs a visible boundary between fixed requirements and adjustable assumptions. You will preserve that boundary in a Markdown decision record.

• In Visual Studio Code, create ARCHITECTURE.md inside the project folder.
• Add the workload requirement boundary with this content:

# 12 Million-Player Leaderboard Architecture

## Requirement boundary

The fixed requirement is 12,000,000 concurrent players. Concurrent users do not directly equal requests per second, so the rest of the traffic model is an explicit scenario that must be validated with product telemetry.

| Input or result | Baseline |
| --- | ---: |
| Concurrent players | 12,000,000 |
| Players updating score in a peak second | 2% |
| Peak score updates per second | 240,000 |
| Leaderboard reads per player per minute | 0.5 |
| Total leaderboard reads per second | 100,000 |
| Top-K share | 25% or 25,000 reads/s |
| Around-me share | 75% or 75,000 reads/s |
| Regions | 4 |
| Game modes | 3 |
| Active seasons | 1 |
| Active partitions | 12 |
| Average players per active partition | 1,000,000 |
| Hot-partition planning multiplier | 2x |
| Hot-partition planning size | 2,000,000 players |
| Average projected score updates per partition per second | 20,000 |
| Hot projected score updates per partition per second | 40,000 |

These are sizing assumptions, not claims about a real game's observed behavior.

Why record the formulas as a contract?

The table makes every baseline input reviewable. A reviewer can challenge one assumption without confusing it with the fixed concurrency requirement.

The final sentence also prevents the scenario from being presented as observed production traffic.

• Save ARCHITECTURE.md.
• Confirm the file shows the requirement paragraph plus the complete baseline table.

Does the table look uneven?

Check that every row begins and ends with a pipe character. Keep the separator row directly below the two column headings.

Help me fix the Markdown workload table

✔️ Awesome, I've got everything!

Your requirement boundary now matches the baseline capacity report.

ⓧ I'd like to double check the full code

# 12 Million-Player Leaderboard Architecture

## Requirement boundary

The fixed requirement is 12,000,000 concurrent players. Concurrent users do not directly equal requests per second, so the rest of the traffic model is an explicit scenario that must be validated with product telemetry.

| Input or result | Baseline |
| --- | ---: |
| Concurrent players | 12,000,000 |
| Players updating score in a peak second | 2% |
| Peak score updates per second | 240,000 |
| Leaderboard reads per player per minute | 0.5 |
| Total leaderboard reads per second | 100,000 |
| Top-K share | 25% or 25,000 reads/s |
| Around-me share | 75% or 75,000 reads/s |
| Regions | 4 |
| Game modes | 3 |
| Active seasons | 1 |
| Active partitions | 12 |
| Average players per active partition | 1,000,000 |
| Hot-partition planning multiplier | 2x |
| Hot-partition planning size | 2,000,000 players |
| Average projected score updates per partition per second | 20,000 |
| Hot projected score updates per partition per second | 40,000 |

These are sizing assumptions, not claims about a real game's observed behavior.

A useful model should react when an assumption changes. One temporary edit reveals which results depend on the peak update fraction.

• In leaderboard_lab.py, locate update_fraction_per_second inside WorkloadAssumptions.
• Temporarily replace 0.02 with 0.25.
• Save leaderboard_lab.py.

Before you rerun the model, which traffic results should change when only the update fraction increases?

• Test the changed assumption by running:

python3 leaderboard_lab.py capacity

What changed?

Peak score updates rise to 3,000,000 per second. The leaderboard read totals remain unchanged because their formulas use a separate assumption.

The projected partition update rates also rise because they depend on the score-update total.

• Restore update_fraction_per_second to 0.02.
• Save leaderboard_lab.py.

Before the final check, do you expect every value in the documented baseline to match the terminal report again?

• Verify the restored baseline by running:

python3 leaderboard_lab.py capacity

What should the final report show?

You should see 240,000 peak score updates per second plus 100,000 leaderboard reads per second. The read split should show 25,000 top-K requests plus 75,000 around-me requests.

You should also see 12 active partitions, 1,000,000 average players per partition, and a 2,000,000-player hot-partition planning size.

Does the report still show the changed scenario?

Confirm that update_fraction_per_second is back to 0.02. Save the file before rerunning the command.

Help me restore the baseline workload output

Your 12 million-player requirement now has a quantified traffic model plus a documented assumption boundary. Next, you will send every simulated write through one global Redis Sorted Set to expose the hot-key bottleneck.

Measure the Global-Key Bottleneck

Your workload model now turns 12 million concurrent players into explicit traffic assumptions. The next question is whether one Redis leaderboard key can distribute that traffic.

The naïve board sends every score update through one Redis Sorted Set. You will measure local latency while exposing the more important structural result that every request targets the same key.

In this step, get ready to:

• Build a deterministic global leaderboard experiment.
• Measure write concentration plus local operation latency.
• Record why the benchmark does not predict production capacity.

Build the single global Sorted Set

A repeatable experiment needs fixed inputs. The Python lab uses a seeded dataset so each run starts with the same players and scores.

• In leaderboard_lab.py, replace the opening import section with the code below:

import argparse
import random
import time
from collections import Counter
from dataclasses import dataclass
from itertools import product

import redis
from redis.exceptions import RedisError

GLOBAL_KEY = "lb:global:all:s1"
SIM_PLAYERS = 12_000
SIM_UPDATES = 3_000
READ_SAMPLES = 120
RANDOM_SEED = 20_260_906

What does this setup control?

• The fixed random seed makes the synthetic player data repeatable.
• The simulation constants keep this local experiment small enough to run on your Mac.
• The Counter import tracks how many writes reach each logical key.

• Find the existing ASSUMPTIONS = WorkloadAssumptions() line.
• Replace the PARTITION_KEYS definition directly below it with this compact definition:

ASSUMPTIONS = WorkloadAssumptions()
PARTITION_KEYS = tuple(f"lb:{region}:{mode}:{season}" for region, mode, season in product(ASSUMPTIONS.regions, ASSUMPTIONS.modes, ASSUMPTIONS.active_seasons))

Why keep the partition keys?

The dataset assigns each player a future region, mode, and season key. This step still stores every player under GLOBAL_KEY so the concentration remains measurable.

• Find the existing print_capacity_model() function.
• Replace that function with this shorter baseline report:

def print_capacity_model() -> None:
    a = ASSUMPTIONS
    print("12 MILLION PLAYER WORKLOAD MODEL")
    print(f"Peak score updates/s: {a.score_updates_per_second:,}")
    print(f"Leaderboard reads/s: {a.leaderboard_reads_per_second:,}")
    print(f"Top-K reads/s: {a.top_k_reads_per_second:,}")
    print(f"Around-me reads/s: {a.around_me_reads_per_second:,}")
    print(f"Active partitions: {a.active_partition_count}")

What does this report preserve?

The report keeps the traffic values needed to interpret the experiment. The assumptions still live in WorkloadAssumptions for later design reviews.

• Add the experiment helpers directly below print_capacity_model() by copying this code:

def build_dataset(players: int = SIM_PLAYERS) -> tuple[dict[str, int], dict[str, str]]:
    rng = random.Random(RANDOM_SEED)
    scores: dict[str, int] = {}
    player_keys: dict[str, str] = {}
    for index in range(players):
        player_id = f"player-{index:05d}"
        scores[player_id] = rng.randint(0, 10_000)
        player_keys[player_id] = PARTITION_KEYS[index % len(PARTITION_KEYS)]
    return scores, player_keys

def percentile(values: list[float], fraction: float) -> float:
    ordered = sorted(values)
    return ordered[int((len(ordered) - 1) * fraction)]

def seed_global(client: redis.Redis) -> dict[str, int]:
    scores, _ = build_dataset()
    pipeline = client.pipeline(transaction=False)
    for player_id, score in scores.items():
        pipeline.zadd(GLOBAL_KEY, {player_id: score})
    pipeline.execute()
    return scores

What do these helpers do?

• The build_dataset() helper generates deterministic player IDs plus scores.
• The percentile() helper extracts a latency value from sorted measurements.
• The seed_global() helper batches every player into GLOBAL_KEY without transactional wrapping.

Run the designed obstacle

The experiment now needs to exercise the global key under both write and read patterns. Score updates use zadd, while ranking reads use zrevrange and zrevrank.

• Add the first part of run_naive() directly below seed_global() by copying this code:

def run_naive(client: redis.Redis) -> None:
    wait_for_redis(client)
    client.flushdb()
    scores = seed_global(client)
    player_ids = list(scores)
    rng = random.Random(RANDOM_SEED + 1)
    write_key_hits: Counter[str] = Counter()
    write_latencies_ms: list[float] = []
    write_started = time.perf_counter()
    for _ in range(SIM_UPDATES):
        player_id = rng.choice(player_ids)
        new_score = scores[player_id] + rng.randint(1, 20)
        started = time.perf_counter_ns()
        client.zadd(GLOBAL_KEY, {player_id: new_score})
        write_latencies_ms.append((time.perf_counter_ns() - started) / 1_000_000)
        scores[player_id] = new_score
        write_key_hits[GLOBAL_KEY] += 1
    write_seconds = time.perf_counter() - write_started

How are writes measured?

• The experiment clears old lab data before loading the deterministic players.
• Each simulated update replaces one player's score in the global Sorted Set.
• The latency list records the duration of each local write.
• The key counter records GLOBAL_KEY for every update.

• Continue inside run_naive() directly below the write_seconds assignment by copying this code:

    for sample in range(READ_SAMPLES):
        if sample % 4 == 0:
            client.zrevrange(GLOBAL_KEY, 0, 9, withscores=True)
        else:
            player_id = rng.choice(player_ids)
            rank = client.zrevrank(GLOBAL_KEY, player_id)
            if rank is not None:
                client.zrevrange(GLOBAL_KEY, max(0, rank - 2), max(0, rank - 2) + 4, withscores=True)
    hottest_write_share = max(write_key_hits.values()) / sum(write_key_hits.values())
    print("NAIVE GLOBAL LEADERBOARD")
    print(f"Unique write keys: {len(write_key_hits)}")
    print(f"Hottest write-key share: {hottest_write_share:.1%}")
    print(f"Local measured writes/s: {SIM_UPDATES / write_seconds:,.0f}")
    print(f"Write p50: {percentile(write_latencies_ms, 0.50):.3f} ms")
    print(f"Projected peak writes sent to this one key: {ASSUMPTIONS.score_updates_per_second:,}/s")
    passed = len(write_key_hits) == 1 and hottest_write_share == 1.0
    print(f"Designed obstacle: {'PASS' if passed else 'FAIL'} - one key receives every write")

How are reads and concentration measured?

• Every fourth sample reads the ten highest scores.
• The remaining samples find a player's rank before reading nearby players.
• The hottest-key calculation divides the busiest key's writes by all measured writes.
• The final check passes only when one key receives every simulated write.

• Find the existing main() function near the bottom of leaderboard_lab.py.
• Replace that function with this command router:

def main() -> None:
    parser = argparse.ArgumentParser(description="Local leaderboard scaling and correctness lab")
    parser.add_argument("command", choices=("setup", "capacity", "naive"))
    args = parser.parse_args()
    if args.command == "capacity":
        print_capacity_model()
        return
    client = connect()
    if args.command == "setup":
        run_setup(client)
    else:
        run_naive(client)

How does the command router change?

The router accepts naive as a third command. That path creates one client before passing it to the experiment.

✔️ Awesome, I've got everything!

• Save leaderboard_lab.py.

ⓧ I'd like to double check the full code

import argparse
import random
import time
from collections import Counter
from dataclasses import dataclass
from itertools import product

import redis
from redis.exceptions import RedisError

GLOBAL_KEY = "lb:global:all:s1"
SIM_PLAYERS = 12_000
SIM_UPDATES = 3_000
READ_SAMPLES = 120
RANDOM_SEED = 20_260_906

@dataclass(frozen=True)
class WorkloadAssumptions:
    concurrent_players: int = 12_000_000
    update_fraction_per_second: float = 0.02
    leaderboard_reads_per_player_per_minute: float = 0.5
    top_k_share: float = 0.25
    around_me_share: float = 0.75
    regions: tuple[str, ...] = ("na-east", "na-west", "eu", "apac")
    modes: tuple[str, ...] = ("solo", "duo", "squad")
    active_seasons: tuple[str, ...] = ("s1",)
    hot_partition_multiplier: float = 2.0

    @property
    def score_updates_per_second(self) -> int:
        return int(self.concurrent_players * self.update_fraction_per_second)

    @property
    def leaderboard_reads_per_second(self) -> int:
        return int(self.concurrent_players * self.leaderboard_reads_per_player_per_minute / 60)

    @property
    def top_k_reads_per_second(self) -> int:
        return int(self.leaderboard_reads_per_second * self.top_k_share)

    @property
    def around_me_reads_per_second(self) -> int:
        return int(self.leaderboard_reads_per_second * self.around_me_share)

    @property
    def active_partition_count(self) -> int:
        return len(self.regions) * len(self.modes) * len(self.active_seasons)

    @property
    def average_players_per_partition(self) -> int:
        return self.concurrent_players // self.active_partition_count

    @property
    def hot_partition_players(self) -> int:
        return int(self.average_players_per_partition * self.hot_partition_multiplier)

ASSUMPTIONS = WorkloadAssumptions()
PARTITION_KEYS = tuple(f"lb:{region}:{mode}:{season}" for region, mode, season in product(ASSUMPTIONS.regions, ASSUMPTIONS.modes, ASSUMPTIONS.active_seasons))

def connect() -> redis.Redis:
    return redis.Redis(host="localhost", port=6379, db=0, decode_responses=True, socket_connect_timeout=2, socket_timeout=2)

def wait_for_redis(client: redis.Redis, attempts: int = 10) -> None:
    for _ in range(attempts):
        try:
            if client.ping():
                return
        except RedisError:
            time.sleep(1)
    raise RuntimeError("Redis did not become ready within 10 seconds.")

def run_setup(client: redis.Redis) -> None:
    wait_for_redis(client)
    print(f"Redis ping: {client.ping()}")
    print("Redis image: redis:8.10.1-alpine")
    print("redis-py version: 8.1.0")

def print_capacity_model() -> None:
    a = ASSUMPTIONS
    print("12 MILLION PLAYER WORKLOAD MODEL")
    print(f"Peak score updates/s: {a.score_updates_per_second:,}")
    print(f"Leaderboard reads/s: {a.leaderboard_reads_per_second:,}")
    print(f"Top-K reads/s: {a.top_k_reads_per_second:,}")
    print(f"Around-me reads/s: {a.around_me_reads_per_second:,}")
    print(f"Active partitions: {a.active_partition_count}")

def build_dataset(players: int = SIM_PLAYERS) -> tuple[dict[str, int], dict[str, str]]:
    rng = random.Random(RANDOM_SEED)
    scores: dict[str, int] = {}
    player_keys: dict[str, str] = {}
    for index in range(players):
        player_id = f"player-{index:05d}"
        scores[player_id] = rng.randint(0, 10_000)
        player_keys[player_id] = PARTITION_KEYS[index % len(PARTITION_KEYS)]
    return scores, player_keys

def percentile(values: list[float], fraction: float) -> float:
    ordered = sorted(values)
    return ordered[int((len(ordered) - 1) * fraction)]

def seed_global(client: redis.Redis) -> dict[str, int]:
    scores, _ = build_dataset()
    pipeline = client.pipeline(transaction=False)
    for player_id, score in scores.items():
        pipeline.zadd(GLOBAL_KEY, {player_id: score})
    pipeline.execute()
    return scores

def run_naive(client: redis.Redis) -> None:
    wait_for_redis(client)
    client.flushdb()
    scores = seed_global(client)
    player_ids = list(scores)
    rng = random.Random(RANDOM_SEED + 1)
    write_key_hits: Counter[str] = Counter()
    write_latencies_ms: list[float] = []
    write_started = time.perf_counter()
    for _ in range(SIM_UPDATES):
        player_id = rng.choice(player_ids)
        new_score = scores[player_id] + rng.randint(1, 20)
        started = time.perf_counter_ns()
        client.zadd(GLOBAL_KEY, {player_id: new_score})
        write_latencies_ms.append((time.perf_counter_ns() - started) / 1_000_000)
        scores[player_id] = new_score
        write_key_hits[GLOBAL_KEY] += 1
    write_seconds = time.perf_counter() - write_started
    for sample in range(READ_SAMPLES):
        if sample % 4 == 0:
            client.zrevrange(GLOBAL_KEY, 0, 9, withscores=True)
        else:
            player_id = rng.choice(player_ids)
            rank = client.zrevrank(GLOBAL_KEY, player_id)
            if rank is not None:
                client.zrevrange(GLOBAL_KEY, max(0, rank - 2), max(0, rank - 2) + 4, withscores=True)
    hottest_write_share = max(write_key_hits.values()) / sum(write_key_hits.values())
    print("NAIVE GLOBAL LEADERBOARD")
    print(f"Unique write keys: {len(write_key_hits)}")
    print(f"Hottest write-key share: {hottest_write_share:.1%}")
    print(f"Local measured writes/s: {SIM_UPDATES / write_seconds:,.0f}")
    print(f"Write p50: {percentile(write_latencies_ms, 0.50):.3f} ms")
    print(f"Projected peak writes sent to this one key: {ASSUMPTIONS.score_updates_per_second:,}/s")
    passed = len(write_key_hits) == 1 and hottest_write_share == 1.0
    print(f"Designed obstacle: {'PASS' if passed else 'FAIL'} - one key receives every write")

def main() -> None:
    parser = argparse.ArgumentParser(description="Local leaderboard scaling and correctness lab")
    parser.add_argument("command", choices=("setup", "capacity", "naive"))
    args = parser.parse_args()
    if args.command == "capacity":
        print_capacity_model()
        return
    client = connect()
    if args.command == "setup":
        run_setup(client)
    else:
        run_naive(client)

if __name__ == "__main__":
    main()

Before you run the experiment, do you expect the simulated writes to spread across the prepared partition keys or stay on one logical key?

• Run the naïve leaderboard experiment with this command:

python3 leaderboard_lab.py naive

What should you see?

You should see Unique write keys: 1 plus Hottest write-key share: 100.0%.

The report should finish with Designed obstacle: PASS. Every simulated write reached one key.

That is the obstacle exposed: the benchmark can be fast locally while its request topology remains concentrated.

Does the experiment fail to run?

Confirm that the Redis container from the earlier setup step is still running. Check that your virtual environment remains active before retrying the command.

Compare the placement of run_naive() with the full-code tab if Python reports an indentation or name error.

Help me debug the naïve leaderboard experiment

Record the design evidence

Local latency describes this Mac plus this synthetic workload. The key-concentration result reveals the production design risk because one logical key remains owned by one shard.

• Select ARCHITECTURE.md in the Explorer sidebar in Visual Studio Code.
• Replace the file contents with this architecture record:

# 12 Million-Player Leaderboard Architecture

## Requirement boundary

The fixed requirement is 12,000,000 concurrent players. Concurrent users do not directly equal requests per second, so the rest of the traffic model is an explicit scenario that must be validated with product telemetry.

The baseline assumes 240,000 peak score updates per second, 100,000 leaderboard reads per second, 25,000 top-K reads per second, 75,000 around-me reads per second, 12 active partitions, 1,000,000 average players per active partition, and a 2x hot-partition planning size of 2,000,000 players. These are sizing assumptions, not claims about observed production traffic.

## Measured local obstacle

The naïve experiment stores all players in `lb:global:all:s1`. The local report measures 100 percent of writes and reads targeting that one key. Local latency and throughput describe only this Mac and this synthetic workload. They are not production capacity claims.

A single Redis Sorted Set is one key. In a clustered deployment, repeated activity on one key remains concentrated on the shard that owns that key. Adding unrelated shards does not split the members of that one Sorted Set.

Why separate evidence from projection?

The measured evidence proves that every local operation targets one key. The projected 240,000 updates per second comes from your scenario model.

This separation keeps the laptop benchmark from becoming an unsupported AWS capacity claim.

✔️ Awesome, I've got everything!

• Save ARCHITECTURE.md.

ⓧ I'd like to double check the full code

# 12 Million-Player Leaderboard Architecture

## Requirement boundary

The fixed requirement is 12,000,000 concurrent players. Concurrent users do not directly equal requests per second, so the rest of the traffic model is an explicit scenario that must be validated with product telemetry.

The baseline assumes 240,000 peak score updates per second, 100,000 leaderboard reads per second, 25,000 top-K reads per second, 75,000 around-me reads per second, 12 active partitions, 1,000,000 average players per active partition, and a 2x hot-partition planning size of 2,000,000 players. These are sizing assumptions, not claims about observed production traffic.

## Measured local obstacle

The naïve experiment stores all players in `lb:global:all:s1`. The local report measures 100 percent of writes and reads targeting that one key. Local latency and throughput describe only this Mac and this synthetic workload. They are not production capacity claims.

A single Redis Sorted Set is one key. In a clustered deployment, repeated activity on one key remains concentrated on the shard that owns that key. Adding unrelated shards does not split the members of that one Sorted Set.

Before the final check, do you expect another run to distribute the writes differently when every update still uses GLOBAL_KEY?

• Verify the saved experiment by running this command again:

python3 leaderboard_lab.py naive

What confirms success?

The report should show Hottest write-key share: 100.0% and project 240,000 peak writes per second onto that logical key.

The final line should contain Designed obstacle: PASS. You now have measured evidence for the bottleneck.

You have proved that the global board concentrates every write on one key. Next, you will distribute those writes while preserving exact global query semantics.

Partition the Board and Prove Global Queries

Your naïve Redis experiment proved that every write lands on one key. That evidence now gives you a measured baseline for the redesign.

Partitioning spreads score updates across product-shaped keys. The trade-off is that global queries must combine results from every partition.

You will prove that the merged global top K remains exact. You will also expose the fan-out cost of exact global rank.

In this step, get ready to:

• Distribute deterministic players across 12 product-shaped keys.
• Prove the merged global top K against the in-memory truth model.
• Measure exact global competition rank across every partition.

Route writes across product partitions

A product partition keeps related rankings together. Each key represents one region plus one game mode plus one season.

The existing dataset already assigns players evenly across the 12 keys in `PARTITION_KEYS`. The next function writes those players into their assigned Sorted Sets.

• In `leaderboard_lab.py` from earlier, locate `def main()`.
• Insert the partition-seeding function immediately above `def main()` by pasting this code:

def seed_partitioned(client: redis.Redis, players: int = SIM_PLAYERS) -> tuple[dict[str, int], dict[str, str]]:
    scores, player_keys = build_dataset(players)
    pipeline = client.pipeline(transaction=False)
    for player_id, score in scores.items():
        pipeline.zadd(player_keys[player_id], {player_id: score})
    pipeline.execute()
    return scores, player_keys

What does partition seeding do?

• `build_dataset()` returns each deterministic score plus the partition key assigned to that player.
• The non-transactional pipeline batches the local seed operations.
• Each `zadd` writes to the player's region, mode, and season key.

• Save `leaderboard_lab.py`.
• Confirm your editor shows `seed_partitioned()` above `def main()` with no syntax warning.

Seeing a syntax warning?

Check that `seed_partitioned()` starts at the left edge of the file. Make sure every line inside the function uses four spaces of indentation.

Ask for help if the warning remains: Help me check the indentation in seed_partitioned().

Merge an exact global top K

Partitioning removes the single global key. A global top-K query now reads the best K candidates from every partition before merging them in Python.

Why are K candidates enough?

A player below position K in one partition already has K players from that partition ahead of them. That player cannot qualify for the global top K.

Reading K candidates from every disjoint partition therefore preserves every possible global winner.

• Locate the end of `seed_partitioned()` in `leaderboard_lab.py`.
• Add the global merge function below `seed_partitioned()` by pasting this code:

def global_top_k(client: redis.Redis, keys: tuple[str, ...] = PARTITION_KEYS, k: int = 10) -> list[tuple[str, float]]:
    candidates: list[tuple[str, float]] = []
    for key in keys:
        candidates.extend(client.zrevrange(key, 0, k - 1, withscores=True))
    return sorted(candidates, key=lambda item: (item[1], item[0]), reverse=True)[:k]

What does the merge function do?

• `zrevrange()` reads the ten highest-scoring candidates from each partition.
• The combined list is sorted by score first.
• Descending player ID provides the same tie-break used by the truth model.
• The final slice retains only the global top K.

• Save `leaderboard_lab.py`.
• Confirm `global_top_k()` appears between `seed_partitioned()` and `def main()`.

Is the merge function misplaced?

Keep `global_top_k()` at the left edge of the file. Place it after the final `return` in `seed_partitioned()`.

Use this prompt for a placement check: Help me place global_top_k() correctly.

Define exact global rank

This lab uses competition rank. A player's rank equals one plus the number of players with a strictly greater score.

Tied players therefore share a rank. Calculating that rank requires the player's score plus one exclusive count from every active partition.

• Locate the end of `global_top_k()` in `leaderboard_lab.py`.
• Add the exact-rank function below it by pasting this code:

def exact_global_rank(client: redis.Redis, player_id: str, player_key: str, keys: tuple[str, ...] = PARTITION_KEYS) -> int:
    score = client.zscore(player_key, player_id)
    if score is None:
        raise KeyError(player_id)
    players_with_higher_score = sum(client.zcount(key, f"({score}", "+inf") for key in keys)
    return 1 + players_with_higher_score

What does exact rank cost?

• `zscore()` reads the target player's score from the assigned partition.
• The exclusive lower bound counts scores strictly greater than the target score.
• One `zcount()` runs against each of the 12 partition keys.
• The result is exact with read amplification proportional to the partition count.

• Save `leaderboard_lab.py`.
• Confirm `exact_global_rank()` appears immediately above `def main()`.

Seeing a problem in the score bound?

Check the parentheses and quotes in `f"({score}"`. The opening parenthesis makes the score boundary exclusive.

Ask for a focused review if needed: Help me check the exact-rank score bound.

The experiment now needs to route the same 3,000 updates through `player_keys`. It also compares Redis results with the complete in-memory score map.

• Insert the first part of `run_partitioned()` below `exact_global_rank()` by pasting this code:

def run_partitioned(client: redis.Redis) -> None:
    wait_for_redis(client)
    client.flushdb()
    scores, player_keys = seed_partitioned(client)
    player_ids = list(scores)
    rng = random.Random(RANDOM_SEED + 1)
    write_key_hits: Counter[str] = Counter()
    write_latencies_ms: list[float] = []
    write_started = time.perf_counter()
    for _ in range(SIM_UPDATES):
        player_id = rng.choice(player_ids)
        key = player_keys[player_id]
        new_score = scores[player_id] + rng.randint(1, 20)
        started = time.perf_counter_ns()
        client.zadd(key, {player_id: new_score})
        write_latencies_ms.append((time.perf_counter_ns() - started) / 1_000_000)
        scores[player_id] = new_score
        write_key_hits[key] += 1
    write_seconds = time.perf_counter() - write_started
    hottest_write_share = max(write_key_hits.values()) / sum(write_key_hits.values())

What does the write loop measure?

• The experiment rebuilds the deterministic dataset across all 12 partition keys.
• Each update looks up the player's assigned key before calling `zadd`.
• `write_key_hits` records how the 3,000 writes spread across those keys.
• The latency list preserves the local comparison with the naïve baseline.

• Continue `run_partitioned()` directly below `hottest_write_share` by pasting this indented code:

    merged_top_k = global_top_k(client)
    expected_top_k = [(player_id, float(score)) for player_id, score in sorted(scores.items(), key=lambda item: (item[1], item[0]), reverse=True)[:10]]
    target_player = "player-00042"
    measured_rank = exact_global_rank(client, target_player, player_keys[target_player])
    target_score = scores[target_player]
    expected_rank = 1 + sum(1 for score in scores.values() if score > target_score)
    local_rank = client.zrevrank(player_keys[target_player], target_player)
    local_neighbors = []
    if local_rank is not None:
        local_neighbors = client.zrevrange(player_keys[target_player], max(0, local_rank - 2), max(0, local_rank - 2) + 4, withscores=True)

How does the truth check work?

• `expected_top_k` sorts the complete in-memory score map.
• `merged_top_k` contains only candidates returned from the partitioned Redis data.
• The target player's expected rank counts every strictly greater score in memory.
• The partition-local neighbor sample remains a single-key query.

• Finish `run_partitioned()` directly below the local-neighbor query by pasting this indented code:

    projected_average = ASSUMPTIONS.score_updates_per_second / ASSUMPTIONS.active_partition_count
    projected_hot = projected_average * ASSUMPTIONS.hot_partition_multiplier
    distribution_pass = len(write_key_hits) == len(PARTITION_KEYS)
    top_k_pass = merged_top_k == expected_top_k
    rank_pass = measured_rank == expected_rank
    print("PARTITIONED LEADERBOARD")
    print(f"Active keys touched: {len(write_key_hits)}")
    print(f"Hottest write-key share: {hottest_write_share:.1%}")
    print(f"Local measured writes/s: {SIM_UPDATES / write_seconds:,.0f}")
    print(f"Write p50: {percentile(write_latencies_ms, 0.50):.3f} ms")
    print(f"Write p95: {percentile(write_latencies_ms, 0.95):.3f} ms")
    print(f"Projected average writes/partition/s: {projected_average:,.0f}")
    print(f"Projected hot writes/partition/s: {projected_hot:,.0f}")
    print(f"Global top-K candidate count: {10 * len(PARTITION_KEYS)}")
    print(f"Exact-rank partition fan-out: {len(PARTITION_KEYS)}")
    print(f"Target score: {target_score}")
    print(f"Exact competition rank: {measured_rank:,}")
    print(f"Partition-local around-me sample: {local_neighbors}")
    print(f"Partition distribution: {'PASS' if distribution_pass else 'FAIL'}")
    print(f"Global top-K proof: {'PASS' if top_k_pass else 'FAIL'}")
    print(f"Exact global rank proof: {'PASS' if rank_pass else 'FAIL'}")

What does the report prove?

• The distribution check confirms that the workload touches all 12 partition keys.
• The top-K check compares the candidate merge with the complete truth model.
• The rank check compares partition fan-out with a full in-memory count.
• The projected rates connect the local topology to the earlier capacity assumptions.

• Save `leaderboard_lab.py`.
• Confirm every line after `hottest_write_share` remains indented inside `run_partitioned()`.

Did part of the function move outside?

Every line from `merged_top_k` through the final proof output needs four leading spaces. The three lines inside the local-rank condition need eight leading spaces.

Use this prompt to review the complete function: Help me fix the run_partitioned() indentation.

The command router currently knows only the earlier experiments. Extending it makes the partitioned proof available as its own command while retaining the existing paths.

• Locate the existing `main()` function at the bottom of `leaderboard_lab.py`.
• Replace the complete `main()` function with this version:

def main() -> None:
    parser = argparse.ArgumentParser(description="Local leaderboard scaling and correctness lab")
    parser.add_argument("command", choices=("setup", "capacity", "naive", "partitioned", "all"))
    args = parser.parse_args()
    if args.command == "capacity":
        print_capacity_model()
        return
    client = connect()
    if args.command == "setup":
        run_setup(client)
    elif args.command == "naive":
        run_naive(client)
    elif args.command == "partitioned":
        run_partitioned(client)
    else:
        run_setup(client)
        print()
        print_capacity_model()
        print()
        run_naive(client)
        print()
        run_partitioned(client)

How does the router change?

The new branch sends the `partitioned` command to `run_partitioned()`. The `all` path now runs every experiment built so far in sequence.

• Save `leaderboard_lab.py`.
• Confirm the existing `if __name__ == "__main__":` block still calls `main()` below the replacement.

Does the command exit without a report?

Check that the `partitioned` branch calls `run_partitioned(client)`. Keep the final `main()` call below the function.

Ask for a router review if needed: Help me check the partitioned command route.

Use the full-file comparison to catch missing lines before you run the proof.

✔️ Awesome, I've got everything!

Great. Double-check that you saved `leaderboard_lab.py` before continuing.

ⓧ I'd like to double check the full code

import argparse
import random
import time
from collections import Counter
from dataclasses import dataclass
from itertools import product

import redis
from redis.exceptions import RedisError

GLOBAL_KEY = "lb:global:all:s1"
SIM_PLAYERS = 12_000
SIM_UPDATES = 3_000
READ_SAMPLES = 120
RANDOM_SEED = 20_260_906

@dataclass(frozen=True)
class WorkloadAssumptions:
    concurrent_players: int = 12_000_000
    update_fraction_per_second: float = 0.02
    leaderboard_reads_per_player_per_minute: float = 0.5
    top_k_share: float = 0.25
    around_me_share: float = 0.75
    regions: tuple[str, ...] = ("na-east", "na-west", "eu", "apac")
    modes: tuple[str, ...] = ("solo", "duo", "squad")
    active_seasons: tuple[str, ...] = ("s1",)
    hot_partition_multiplier: float = 2.0

    @property
    def score_updates_per_second(self) -> int:
        return int(self.concurrent_players * self.update_fraction_per_second)

    @property
    def leaderboard_reads_per_second(self) -> int:
        return int(self.concurrent_players * self.leaderboard_reads_per_player_per_minute / 60)

    @property
    def top_k_reads_per_second(self) -> int:
        return int(self.leaderboard_reads_per_second * self.top_k_share)

    @property
    def around_me_reads_per_second(self) -> int:
        return int(self.leaderboard_reads_per_second * self.around_me_share)

    @property
    def active_partition_count(self) -> int:
        return len(self.regions) * len(self.modes) * len(self.active_seasons)

    @property
    def average_players_per_partition(self) -> int:
        return self.concurrent_players // self.active_partition_count

    @property
    def hot_partition_players(self) -> int:
        return int(self.average_players_per_partition * self.hot_partition_multiplier)

ASSUMPTIONS = WorkloadAssumptions()
PARTITION_KEYS = tuple(f"lb:{region}:{mode}:{season}" for region, mode, season in product(ASSUMPTIONS.regions, ASSUMPTIONS.modes, ASSUMPTIONS.active_seasons))

def connect() -> redis.Redis:
    return redis.Redis(host="localhost", port=6379, db=0, decode_responses=True, socket_connect_timeout=2, socket_timeout=2)

def wait_for_redis(client: redis.Redis, attempts: int = 10) -> None:
    for _ in range(attempts):
        try:
            if client.ping():
                return
        except RedisError:
            time.sleep(1)
    raise RuntimeError("Redis did not become ready within 10 seconds.")

def run_setup(client: redis.Redis) -> None:
    wait_for_redis(client)
    print(f"Redis ping: {client.ping()}")
    print("Redis image: redis:8.10.1-alpine")
    print("redis-py version: 8.1.0")

def print_capacity_model() -> None:
    a = ASSUMPTIONS
    print("12 MILLION PLAYER WORKLOAD MODEL")
    print(f"Peak score updates/s: {a.score_updates_per_second:,}")
    print(f"Leaderboard reads/s: {a.leaderboard_reads_per_second:,}")
    print(f"Top-K reads/s: {a.top_k_reads_per_second:,}")
    print(f"Around-me reads/s: {a.around_me_reads_per_second:,}")
    print(f"Active partitions: {a.active_partition_count}")

def build_dataset(players: int = SIM_PLAYERS) -> tuple[dict[str, int], dict[str, str]]:
    rng = random.Random(RANDOM_SEED)
    scores: dict[str, int] = {}
    player_keys: dict[str, str] = {}
    for index in range(players):
        player_id = f"player-{index:05d}"
        scores[player_id] = rng.randint(0, 10_000)
        player_keys[player_id] = PARTITION_KEYS[index % len(PARTITION_KEYS)]
    return scores, player_keys

def percentile(values: list[float], fraction: float) -> float:
    ordered = sorted(values)
    return ordered[int((len(ordered) - 1) * fraction)]

def seed_global(client: redis.Redis) -> dict[str, int]:
    scores, _ = build_dataset()
    pipeline = client.pipeline(transaction=False)
    for player_id, score in scores.items():
        pipeline.zadd(GLOBAL_KEY, {player_id: score})
    pipeline.execute()
    return scores

def run_naive(client: redis.Redis) -> None:
    wait_for_redis(client)
    client.flushdb()
    scores = seed_global(client)
    player_ids = list(scores)
    rng = random.Random(RANDOM_SEED + 1)
    write_key_hits: Counter[str] = Counter()
    write_latencies_ms: list[float] = []
    write_started = time.perf_counter()
    for _ in range(SIM_UPDATES):
        player_id = rng.choice(player_ids)
        new_score = scores[player_id] + rng.randint(1, 20)
        started = time.perf_counter_ns()
        client.zadd(GLOBAL_KEY, {player_id: new_score})
        write_latencies_ms.append((time.perf_counter_ns() - started) / 1_000_000)
        scores[player_id] = new_score
        write_key_hits[GLOBAL_KEY] += 1
    write_seconds = time.perf_counter() - write_started
    for sample in range(READ_SAMPLES):
        if sample % 4 == 0:
            client.zrevrange(GLOBAL_KEY, 0, 9, withscores=True)
        else:
            player_id = rng.choice(player_ids)
            rank = client.zrevrank(GLOBAL_KEY, player_id)
            if rank is not None:
                client.zrevrange(GLOBAL_KEY, max(0, rank - 2), max(0, rank - 2) + 4, withscores=True)
    hottest_write_share = max(write_key_hits.values()) / sum(write_key_hits.values())
    print("NAIVE GLOBAL LEADERBOARD")
    print(f"Unique write keys: {len(write_key_hits)}")
    print(f"Hottest write-key share: {hottest_write_share:.1%}")
    print(f"Local measured writes/s: {SIM_UPDATES / write_seconds:,.0f}")
    print(f"Write p50: {percentile(write_latencies_ms, 0.50):.3f} ms")
    print(f"Projected peak writes sent to this one key: {ASSUMPTIONS.score_updates_per_second:,}/s")
    passed = len(write_key_hits) == 1 and hottest_write_share == 1.0
    print(f"Designed obstacle: {'PASS' if passed else 'FAIL'} - one key receives every write")

def seed_partitioned(client: redis.Redis, players: int = SIM_PLAYERS) -> tuple[dict[str, int], dict[str, str]]:
    scores, player_keys = build_dataset(players)
    pipeline = client.pipeline(transaction=False)
    for player_id, score in scores.items():
        pipeline.zadd(player_keys[player_id], {player_id: score})
    pipeline.execute()
    return scores, player_keys

def global_top_k(client: redis.Redis, keys: tuple[str, ...] = PARTITION_KEYS, k: int = 10) -> list[tuple[str, float]]:
    candidates: list[tuple[str, float]] = []
    for key in keys:
        candidates.extend(client.zrevrange(key, 0, k - 1, withscores=True))
    return sorted(candidates, key=lambda item: (item[1], item[0]), reverse=True)[:k]

def exact_global_rank(client: redis.Redis, player_id: str, player_key: str, keys: tuple[str, ...] = PARTITION_KEYS) -> int:
    score = client.zscore(player_key, player_id)
    if score is None:
        raise KeyError(player_id)
    players_with_higher_score = sum(client.zcount(key, f"({score}", "+inf") for key in keys)
    return 1 + players_with_higher_score

def run_partitioned(client: redis.Redis) -> None:
    wait_for_redis(client)
    client.flushdb()
    scores, player_keys = seed_partitioned(client)
    player_ids = list(scores)
    rng = random.Random(RANDOM_SEED + 1)
    write_key_hits: Counter[str] = Counter()
    write_latencies_ms: list[float] = []
    write_started = time.perf_counter()
    for _ in range(SIM_UPDATES):
        player_id = rng.choice(player_ids)
        key = player_keys[player_id]
        new_score = scores[player_id] + rng.randint(1, 20)
        started = time.perf_counter_ns()
        client.zadd(key, {player_id: new_score})
        write_latencies_ms.append((time.perf_counter_ns() - started) / 1_000_000)
        scores[player_id] = new_score
        write_key_hits[key] += 1
    write_seconds = time.perf_counter() - write_started
    hottest_write_share = max(write_key_hits.values()) / sum(write_key_hits.values())
    merged_top_k = global_top_k(client)
    expected_top_k = [(player_id, float(score)) for player_id, score in sorted(scores.items(), key=lambda item: (item[1], item[0]), reverse=True)[:10]]
    target_player = "player-00042"
    measured_rank = exact_global_rank(client, target_player, player_keys[target_player])
    target_score = scores[target_player]
    expected_rank = 1 + sum(1 for score in scores.values() if score > target_score)
    local_rank = client.zrevrank(player_keys[target_player], target_player)
    local_neighbors = []
    if local_rank is not None:
        local_neighbors = client.zrevrange(player_keys[target_player], max(0, local_rank - 2), max(0, local_rank - 2) + 4, withscores=True)
    projected_average = ASSUMPTIONS.score_updates_per_second / ASSUMPTIONS.active_partition_count
    projected_hot = projected_average * ASSUMPTIONS.hot_partition_multiplier
    distribution_pass = len(write_key_hits) == len(PARTITION_KEYS)
    top_k_pass = merged_top_k == expected_top_k
    rank_pass = measured_rank == expected_rank
    print("PARTITIONED LEADERBOARD")
    print(f"Active keys touched: {len(write_key_hits)}")
    print(f"Hottest write-key share: {hottest_write_share:.1%}")
    print(f"Local measured writes/s: {SIM_UPDATES / write_seconds:,.0f}")
    print(f"Write p50: {percentile(write_latencies_ms, 0.50):.3f} ms")
    print(f"Write p95: {percentile(write_latencies_ms, 0.95):.3f} ms")
    print(f"Projected average writes/partition/s: {projected_average:,.0f}")
    print(f"Projected hot writes/partition/s: {projected_hot:,.0f}")
    print(f"Global top-K candidate count: {10 * len(PARTITION_KEYS)}")
    print(f"Exact-rank partition fan-out: {len(PARTITION_KEYS)}")
    print(f"Target score: {target_score}")
    print(f"Exact competition rank: {measured_rank:,}")
    print(f"Partition-local around-me sample: {local_neighbors}")
    print(f"Partition distribution: {'PASS' if distribution_pass else 'FAIL'}")
    print(f"Global top-K proof: {'PASS' if top_k_pass else 'FAIL'}")
    print(f"Exact global rank proof: {'PASS' if rank_pass else 'FAIL'}")

def main() -> None:
    parser = argparse.ArgumentParser(description="Local leaderboard scaling and correctness lab")
    parser.add_argument("command", choices=("setup", "capacity", "naive", "partitioned", "all"))
    args = parser.parse_args()
    if args.command == "capacity":
        print_capacity_model()
        return
    client = connect()
    if args.command == "setup":
        run_setup(client)
    elif args.command == "naive":
        run_naive(client)
    elif args.command == "partitioned":
        run_partitioned(client)
    else:
        run_setup(client)
        print()
        print_capacity_model()
        print()
        run_naive(client)
        print()
        run_partitioned(client)

if __name__ == "__main__":
    main()

Compare this reference with your saved file. Keep the earlier naïve experiment intact while checking the four new partitioned functions and command route.

The code proves the behavior locally. The architecture record now needs to state which answers are partition-local plus which answers require global fan-out.

• Switch back to `ARCHITECTURE.md` from earlier.
• Replace its contents with this updated decision record:

# 12 Million-Player Leaderboard Architecture

## Requirement boundary

The fixed requirement is 12,000,000 concurrent players. Concurrent users do not directly equal requests per second, so the traffic model is an explicit scenario that must be validated with product telemetry. The baseline is 240,000 score updates/s, 100,000 leaderboard reads/s, 25,000 top-K reads/s, 75,000 around-me reads/s, 12 active partitions, 1,000,000 average players per partition, and a 2x hot-partition planning multiplier.

## Measured local obstacle

The naïve experiment stores all players in `lb:global:all:s1`. It measures 100 percent of writes and reads targeting that one key. Laptop throughput is not an AWS capacity claim. A single Redis Sorted Set is one key, so repeated traffic remains concentrated on the cluster shard that owns that key.

## Partition model

Active keys use `lb:<region>:<mode>:<season>`. Four regions, three game modes, and one active season create 12 active keys. Region, mode, and season preserve meaningful partition-local queries.

## Query semantics

### Partition top K

Reading the first K members from one partition is exact for that region, mode, and season.

### Global top K

Read K candidates from every active partition, merge candidate lists, and retain the best K. This is exact at aggregation time. A player below position K in its own partition already has K players ahead in that partition and cannot be in the global top K. Production materialized global top-K responses must include an `as_of` value.

### Exact global rank

Competition rank is `1 + sum(players with score strictly greater across every active partition)`. Tied scores share rank. The local implementation reads the player score and runs one exclusive score count per partition. It is exact but costs fan-out and needs a separate rate limit and service objective.

### Around me

Partition-local around-me is exact and uses one rank lookup plus one range read on a single key. A high-QPS exact global around-me feed needs another index or relaxed freshness guarantee.

What does the decision record capture?

Partition-local top K plus around-me remain single-key operations. Global top K remains exact at its declared aggregation point.

Exact global rank remains available through fan-out. Its separate cost prevents the design from presenting it as a cheap hot-path query.

• Save `ARCHITECTURE.md`.
• Preview `ARCHITECTURE.md` in Visual Studio Code.
• Confirm the preview contains `Partition model` plus all four query-semantics headings.

Is the Markdown preview incomplete?

Check that every heading begins on its own line. Confirm the fenced text from the code block was not pasted into `ARCHITECTURE.md`.

Use this prompt for a structure check: Help me review the architecture headings.

Before you run the proof, what do you expect to happen to the hottest-key share when 3,000 writes spread across 12 keys?

• Run the partitioned experiment in the terminal from earlier with this command:

python3 leaderboard_lab.py partitioned

What does this command test?

The command rebuilds the board across all 12 partitions. It then tests write distribution plus both global-query proofs against the deterministic truth model.

You should see `Partition distribution: PASS`, `Global top-K proof: PASS`, and `Exact global rank proof: PASS`. The report also shows 12 active keys plus an exact-rank fan-out of 12.

Strong result. Your partitioned board now distributes writes while preserving exact answers for the two global queries you explicitly proved.

Missing one of the PASS results?

A failed distribution check usually means an update still uses `GLOBAL_KEY` instead of `player_keys[player_id]`. A failed top-K check usually points to the candidate sort order.

A failed rank check usually means the score boundary includes tied scores. Use this prompt to isolate the mismatch: Help me debug the partitioned proof.

Your board now has measured distribution plus proven global-query semantics. Next, you will test how the design behaves during duplicate delivery, cache failure, and delayed aggregation.

Drill Failures and Map the AWS Design

Your partitioned leaderboard now spreads writes across twelve Redis keys. It also proves exact global top-K results and exact competition rank.

Production traffic introduces retries, cache outages, and delayed projections. This step makes those failure outcomes testable before mapping them to an architecture that preserves durable scores.

In this step, get ready to:

• Suppress duplicate score events by event ID.
• Replay score projections after a simulated Redis outage.
• Define the AWS data flow and failure contract.

Simulate duplicate delivery and Redis failure

Streaming systems can deliver the same event more than once. Idempotency ensures an identical retry has the same effect as processing the event once.

The processor keeps durable scores separate from the Redis projection. That boundary lets a score remain accepted while the ranking cache is unavailable.

• In leaderboard_lab.py, find the end of run_partitioned().
• Add the event model directly below run_partitioned() by copying this code:

@dataclass(frozen=True)
class ScoreEvent:
    event_id: str
    player_id: str
    new_score: int

What does this event model do?

• The frozen data class gives each score update an immutable event ID.
• The player ID identifies the durable score to replace.
• The new score carries the absolute value projected into Redis.

• Save leaderboard_lab.py.
• Confirm the existing partition experiment still runs by using this command:

python3 leaderboard_lab.py partitioned

What does this check prove?

This regression check proves the new event model has not changed the partitioned experiment. You should still see all three partition proofs report PASS.

Does the partition check fail?

Check that ScoreEvent sits after the complete run_partitioned() function. Confirm the class fields remain indented beneath the class definition.

Ask for help with the placement of the new class.

• Add the processor class directly below ScoreEvent by copying this code:

class DurableScoreProcessor:
    def __init__(self, client: redis.Redis, durable_scores: dict[str, int], player_keys: dict[str, str]) -> None:
        self.client = client
        self.durable_scores = durable_scores
        self.player_keys = player_keys
        self.processed_event_ids: set[str] = set()
        self.projection_backlog: list[ScoreEvent] = []
        self.redis_available = True

    def process(self, event: ScoreEvent) -> str:
        if event.event_id in self.processed_event_ids:
            return "duplicate"
        self.processed_event_ids.add(event.event_id)
        self.durable_scores[event.player_id] = event.new_score
        if not self.redis_available:
            self.projection_backlog.append(event)
            return "durable-only"
        key = self.player_keys[event.player_id]
        self.client.zadd(key, {event.player_id: event.new_score})
        return "applied"

How does duplicate suppression work?

• The processor stores accepted event IDs in processed_event_ids.
• A repeated ID returns duplicate before changing durable state.
• A unique event always updates durable_scores first.
• An unavailable projection queues the event in projection_backlog for recovery.

• Save leaderboard_lab.py.
• Run the partition experiment again to confirm the new processor loads correctly:

python3 leaderboard_lab.py partitioned

Why rerun the partition proof?

The command imports the new class while exercising the existing leaderboard path. Three PASS results confirm the processor definition has not broken earlier behavior.

Seeing an indentation problem?

Confirm that process() is indented inside DurableScoreProcessor. Check that each statement inside the method has one additional indentation level.

Help me inspect the processor indentation.

• Add the recovery method inside DurableScoreProcessor directly below process() by copying this code:

    def recover_projection(self) -> int:
        self.redis_available = True
        replayed = 0
        for event in self.projection_backlog:
            key = self.player_keys[event.player_id]
            latest_score = self.durable_scores[event.player_id]
            self.client.zadd(key, {event.player_id: latest_score})
            replayed += 1
        self.projection_backlog.clear()
        return replayed

How does projection recovery work?

• The method marks Redis as available before replay begins.
• Each queued event resolves the latest durable score for its player.
• The replay updates the correct partition key.
• The cleared backlog prevents a recovered projection from being replayed again.

• Save leaderboard_lab.py.
• Confirm the existing partition proofs remain intact by running this command:

python3 leaderboard_lab.py partitioned

What should remain unchanged?

You should still see Partition distribution: PASS, Global top-K proof: PASS, and Exact global rank proof: PASS.

Does Python reject the recovery method?

Make sure recover_projection() aligns with process() inside the class. The method must not be nested inside process().

Help me place the recovery method correctly.

Demonstrate stale aggregation and replay

A durable write can succeed while its Redis projection remains stale. A cached global top-K result stays stale for an additional period because aggregation refreshes independently.

The failure drill captures each state in sequence. It proves duplicate suppression, durable-only acceptance, projection replay, and delayed aggregation with one deterministic experiment.

• Add run_failures() below DurableScoreProcessor by copying this code:

def run_failures(client: redis.Redis) -> None:
    wait_for_redis(client)
    client.flushdb()
    scores, player_keys = seed_partitioned(client)
    processor = DurableScoreProcessor(client, scores, player_keys)
    retry_player = "player-00010"
    retry_score = scores[retry_player] + 500
    retry_event = ScoreEvent("event-retry-1", retry_player, retry_score)
    first_status = processor.process(retry_event)
    second_status = processor.process(retry_event)
    stored_retry_score = client.zscore(player_keys[retry_player], retry_player)
    duplicate_pass = first_status == "applied" and second_status == "duplicate" and stored_retry_score == float(retry_score)
    cached_top_k = global_top_k(client)
    champion = min(scores, key=scores.get)
    winning_score = max(scores.values()) + 10_000
    processor.redis_available = False
    outage_status = processor.process(ScoreEvent("event-outage-1", champion, winning_score))
    redis_score_during_outage = client.zscore(player_keys[champion], champion)
    replayed = processor.recover_projection()
    redis_score_after_recovery = client.zscore(player_keys[champion], champion)
    refreshed_top_k = global_top_k(client)
    outage_pass = outage_status == "durable-only" and processor.durable_scores[champion] == winning_score and redis_score_during_outage != float(winning_score) and redis_score_after_recovery == float(winning_score) and replayed == 1
    stale_ids = [player_id for player_id, _ in cached_top_k]
    refreshed_ids = [player_id for player_id, _ in refreshed_top_k]
    delayed_aggregation_pass = champion not in stale_ids and refreshed_ids[0] == champion
    print("FAILURE DRILL")
    print(f"Duplicate retry: {'PASS' if duplicate_pass else 'FAIL'}")
    print(f"Redis outage replay: {'PASS' if outage_pass else 'FAIL'}")
    print(f"Delayed aggregation: {'PASS' if delayed_aggregation_pass else 'FAIL'}")
    print("Failure contract: durable writes continue; ranking reads are stale or unavailable until projection recovery.")

What does the failure drill prove?

• The same retry event is submitted twice to prove only its first delivery changes state.
• The lowest-scoring player receives a winning durable score while Redis is marked unavailable.
• The recovery method replays exactly one queued projection.
• The cached top-K excludes the new champion until global_top_k() refreshes the result.

• Find the existing main() function near the bottom of leaderboard_lab.py.
• Replace the complete main() function with this version:

def main() -> None:
    parser = argparse.ArgumentParser(description="Local leaderboard scaling and correctness lab")
    parser.add_argument("command", choices=("setup", "capacity", "naive", "partitioned", "failures", "all"))
    args = parser.parse_args()
    if args.command == "capacity":
        print_capacity_model()
        return
    client = connect()
    if args.command == "setup":
        run_setup(client)
    elif args.command == "naive":
        run_naive(client)
    elif args.command == "partitioned":
        run_partitioned(client)
    elif args.command == "failures":
        run_failures(client)
    else:
        run_setup(client)
        print()
        print_capacity_model()
        print()
        run_naive(client)
        print()
        run_partitioned(client)
        print()
        run_failures(client)

How does the command router change?

• The parser now accepts failures as a command.
• The matching branch calls run_failures().
• The all path includes the failure drill after the partitioned experiment.

• Save leaderboard_lab.py.

Before you run the drill, do you expect the cached top-K to include the new champion before the explicit refresh?

• Run the complete failure experiment with this command:

python3 leaderboard_lab.py failures

What should the drill reveal?

The durable model accepts the winning score during the simulated outage. Redis catches up during replay.

The earlier top-K snapshot stays stale until the explicit refresh. You should see all three checks report PASS.

That is the hard part proven. Your lab now turns duplicate delivery, cache loss, and stale aggregation into observable outcomes.

Does one failure check report FAIL?

For duplicate failure, confirm both calls use the same retry_event. For replay failure, confirm the outage event enters projection_backlog before recovery.

For aggregation failure, confirm cached_top_k is captured before the winning score is processed. Help me diagnose the failed check.

✔️ Awesome, I've got everything!

Great. Double-check that you saved leaderboard_lab.py before continuing.

ⓧ I'd like to double check the full code

import argparse
import random
import time
from collections import Counter
from dataclasses import dataclass
from itertools import product

import redis
from redis.exceptions import RedisError

GLOBAL_KEY = "lb:global:all:s1"
SIM_PLAYERS = 12_000
SIM_UPDATES = 3_000
READ_SAMPLES = 120
RANDOM_SEED = 20_260_906

@dataclass(frozen=True)
class WorkloadAssumptions:
    concurrent_players: int = 12_000_000
    update_fraction_per_second: float = 0.02
    leaderboard_reads_per_player_per_minute: float = 0.5
    top_k_share: float = 0.25
    around_me_share: float = 0.75
    regions: tuple[str, ...] = ("na-east", "na-west", "eu", "apac")
    modes: tuple[str, ...] = ("solo", "duo", "squad")
    active_seasons: tuple[str, ...] = ("s1",)
    hot_partition_multiplier: float = 2.0

    @property
    def score_updates_per_second(self) -> int:
        return int(self.concurrent_players * self.update_fraction_per_second)

    @property
    def leaderboard_reads_per_second(self) -> int:
        return int(self.concurrent_players * self.leaderboard_reads_per_player_per_minute / 60)

    @property
    def top_k_reads_per_second(self) -> int:
        return int(self.leaderboard_reads_per_second * self.top_k_share)

    @property
    def around_me_reads_per_second(self) -> int:
        return int(self.leaderboard_reads_per_second * self.around_me_share)

    @property
    def active_partition_count(self) -> int:
        return len(self.regions) * len(self.modes) * len(self.active_seasons)

    @property
    def average_players_per_partition(self) -> int:
        return self.concurrent_players // self.active_partition_count

    @property
    def hot_partition_players(self) -> int:
        return int(self.average_players_per_partition * self.hot_partition_multiplier)

ASSUMPTIONS = WorkloadAssumptions()
PARTITION_KEYS = tuple(
    f"lb:{region}:{mode}:{season}"
    for region, mode, season in product(
        ASSUMPTIONS.regions,
        ASSUMPTIONS.modes,
        ASSUMPTIONS.active_seasons,
    )
)

def connect() -> redis.Redis:
    return redis.Redis(
        host="localhost",
        port=6379,
        db=0,
        decode_responses=True,
        socket_connect_timeout=2,
        socket_timeout=2,
    )

def wait_for_redis(client: redis.Redis, attempts: int = 10) -> None:
    for _ in range(attempts):
        try:
            if client.ping():
                return
        except RedisError:
            time.sleep(1)
    raise RuntimeError("Redis did not become ready within 10 seconds.")

def run_setup(client: redis.Redis) -> None:
    wait_for_redis(client)
    print(f"Redis ping: {client.ping()}")
    print("Redis image: redis:8.10.1-alpine")
    print("redis-py version: 8.1.0")

def print_capacity_model() -> None:
    a = ASSUMPTIONS
    average_partition_updates = a.score_updates_per_second / a.active_partition_count
    hot_partition_updates = average_partition_updates * a.hot_partition_multiplier
    print("12 MILLION PLAYER WORKLOAD MODEL")
    print("Scenario assumptions, not measured production traffic")
    print(f"Concurrent players: {a.concurrent_players:,}")
    print(f"Peak score-update assumption: {a.update_fraction_per_second:.1%} of players per second")
    print(f"Leaderboard-read assumption: {a.leaderboard_reads_per_player_per_minute} reads/player/minute")
    print(f"Peak score updates/s: {a.score_updates_per_second:,}")
    print(f"Leaderboard reads/s: {a.leaderboard_reads_per_second:,}")
    print(f"Top-K reads/s: {a.top_k_reads_per_second:,}")
    print(f"Around-me reads/s: {a.around_me_reads_per_second:,}")
    print(f"Active partitions: {a.active_partition_count}")
    print(f"Average players/active partition: {a.average_players_per_partition:,}")
    print(f"Hot-partition planning size: {a.hot_partition_players:,} players")
    print(f"Average projected updates/partition/s: {average_partition_updates:,.0f}")
    print(f"Hot projected updates/partition/s: {hot_partition_updates:,.0f}")

def build_dataset(players: int = SIM_PLAYERS) -> tuple[dict[str, int], dict[str, str]]:
    rng = random.Random(RANDOM_SEED)
    scores: dict[str, int] = {}
    player_keys: dict[str, str] = {}
    for index in range(players):
        player_id = f"player-{index:05d}"
        scores[player_id] = rng.randint(0, 10_000)
        player_keys[player_id] = PARTITION_KEYS[index % len(PARTITION_KEYS)]
    return scores, player_keys

def percentile(values: list[float], fraction: float) -> float:
    ordered = sorted(values)
    index = int((len(ordered) - 1) * fraction)
    return ordered[index]

def seed_global(client: redis.Redis) -> dict[str, int]:
    scores, _ = build_dataset()
    pipeline = client.pipeline(transaction=False)
    for player_id, score in scores.items():
        pipeline.zadd(GLOBAL_KEY, {player_id: score})
    pipeline.execute()
    return scores

def run_naive(client: redis.Redis) -> None:
    wait_for_redis(client)
    client.flushdb()
    scores = seed_global(client)
    player_ids = list(scores)
    rng = random.Random(RANDOM_SEED + 1)
    write_key_hits: Counter[str] = Counter()
    write_latencies_ms: list[float] = []
    write_started = time.perf_counter()
    for _ in range(SIM_UPDATES):
        player_id = rng.choice(player_ids)
        new_score = scores[player_id] + rng.randint(1, 20)
        started = time.perf_counter_ns()
        client.zadd(GLOBAL_KEY, {player_id: new_score})
        write_latencies_ms.append((time.perf_counter_ns() - started) / 1_000_000)
        scores[player_id] = new_score
        write_key_hits[GLOBAL_KEY] += 1
    write_seconds = time.perf_counter() - write_started
    read_key_hits: Counter[str] = Counter()
    read_latencies_ms: list[float] = []
    for sample in range(READ_SAMPLES):
        started = time.perf_counter_ns()
        if sample % 4 == 0:
            client.zrevrange(GLOBAL_KEY, 0, 9, withscores=True)
            read_key_hits[GLOBAL_KEY] += 1
        else:
            player_id = rng.choice(player_ids)
            rank = client.zrevrank(GLOBAL_KEY, player_id)
            if rank is not None:
                start = max(0, rank - 2)
                client.zrevrange(GLOBAL_KEY, start, start + 4, withscores=True)
            read_key_hits[GLOBAL_KEY] += 2
        read_latencies_ms.append((time.perf_counter_ns() - started) / 1_000_000)
    hottest_write_share = max(write_key_hits.values()) / sum(write_key_hits.values())
    print("NAIVE GLOBAL LEADERBOARD")
    print(f"Players loaded: {len(scores):,}")
    print(f"Writes measured: {SIM_UPDATES:,}")
    print(f"Unique write keys: {len(write_key_hits)}")
    print(f"Hottest write-key share: {hottest_write_share:.1%}")
    print(f"Unique read keys: {len(read_key_hits)}")
    print(f"Local measured writes/s: {SIM_UPDATES / write_seconds:,.0f}")
    print(f"Write p50: {percentile(write_latencies_ms, 0.50):.3f} ms")
    print(f"Write p95: {percentile(write_latencies_ms, 0.95):.3f} ms")
    print(f"Read p50: {percentile(read_latencies_ms, 0.50):.3f} ms")
    print(f"Read p95: {percentile(read_latencies_ms, 0.95):.3f} ms")
    print(f"Projected peak writes sent to this one key: {ASSUMPTIONS.score_updates_per_second:,}/s")
    passed = len(write_key_hits) == 1 and hottest_write_share == 1.0
    print(f"Designed obstacle: {'PASS' if passed else 'FAIL'} - one key receives every write")

def seed_partitioned(client: redis.Redis, players: int = SIM_PLAYERS) -> tuple[dict[str, int], dict[str, str]]:
    scores, player_keys = build_dataset(players)
    pipeline = client.pipeline(transaction=False)
    for player_id, score in scores.items():
        pipeline.zadd(player_keys[player_id], {player_id: score})
    pipeline.execute()
    return scores, player_keys

def global_top_k(client: redis.Redis, keys: tuple[str, ...] = PARTITION_KEYS, k: int = 10) -> list[tuple[str, float]]:
    candidates: list[tuple[str, float]] = []
    for key in keys:
        candidates.extend(client.zrevrange(key, 0, k - 1, withscores=True))
    return sorted(candidates, key=lambda item: (item[1], item[0]), reverse=True)[:k]

def exact_global_rank(client: redis.Redis, player_id: str, player_key: str, keys: tuple[str, ...] = PARTITION_KEYS) -> int:
    score = client.zscore(player_key, player_id)
    if score is None:
        raise KeyError(player_id)
    players_with_higher_score = sum(client.zcount(key, f"({score}", "+inf") for key in keys)
    return 1 + players_with_higher_score

def run_partitioned(client: redis.Redis) -> None:
    wait_for_redis(client)
    client.flushdb()
    scores, player_keys = seed_partitioned(client)
    player_ids = list(scores)
    rng = random.Random(RANDOM_SEED + 1)
    write_key_hits: Counter[str] = Counter()
    write_latencies_ms: list[float] = []
    write_started = time.perf_counter()
    for _ in range(SIM_UPDATES):
        player_id = rng.choice(player_ids)
        key = player_keys[player_id]
        new_score = scores[player_id] + rng.randint(1, 20)
        started = time.perf_counter_ns()
        client.zadd(key, {player_id: new_score})
        write_latencies_ms.append((time.perf_counter_ns() - started) / 1_000_000)
        scores[player_id] = new_score
        write_key_hits[key] += 1
    write_seconds = time.perf_counter() - write_started
    hottest_write_share = max(write_key_hits.values()) / sum(write_key_hits.values())
    merged_top_k = global_top_k(client)
    expected_top_k = [(player_id, float(score)) for player_id, score in sorted(scores.items(), key=lambda item: (item[1], item[0]), reverse=True)[:10]]
    target_player = "player-00042"
    measured_rank = exact_global_rank(client, target_player, player_keys[target_player])
    target_score = scores[target_player]
    expected_rank = 1 + sum(1 for score in scores.values() if score > target_score)
    local_rank = client.zrevrank(player_keys[target_player], target_player)
    local_neighbors = []
    if local_rank is not None:
        local_neighbors = client.zrevrange(player_keys[target_player], max(0, local_rank - 2), max(0, local_rank - 2) + 4, withscores=True)
    projected_average = ASSUMPTIONS.score_updates_per_second / ASSUMPTIONS.active_partition_count
    projected_hot = projected_average * ASSUMPTIONS.hot_partition_multiplier
    distribution_pass = len(write_key_hits) == len(PARTITION_KEYS)
    top_k_pass = merged_top_k == expected_top_k
    rank_pass = measured_rank == expected_rank
    print("PARTITIONED LEADERBOARD")
    print(f"Active keys touched: {len(write_key_hits)}")
    print(f"Hottest write-key share: {hottest_write_share:.1%}")
    print(f"Local measured writes/s: {SIM_UPDATES / write_seconds:,.0f}")
    print(f"Write p50: {percentile(write_latencies_ms, 0.50):.3f} ms")
    print(f"Write p95: {percentile(write_latencies_ms, 0.95):.3f} ms")
    print(f"Projected average writes/partition/s: {projected_average:,.0f}")
    print(f"Projected hot writes/partition/s: {projected_hot:,.0f}")
    print(f"Global top-K candidate count: {10 * len(PARTITION_KEYS)}")
    print(f"Exact-rank partition fan-out: {len(PARTITION_KEYS)}")
    print(f"Target score: {target_score}")
    print(f"Exact competition rank: {measured_rank:,}")
    print(f"Partition-local around-me sample: {local_neighbors}")
    print(f"Partition distribution: {'PASS' if distribution_pass else 'FAIL'}")
    print(f"Global top-K proof: {'PASS' if top_k_pass else 'FAIL'}")
    print(f"Exact global rank proof: {'PASS' if rank_pass else 'FAIL'}")

@dataclass(frozen=True)
class ScoreEvent:
    event_id: str
    player_id: str
    new_score: int

class DurableScoreProcessor:
    def __init__(self, client: redis.Redis, durable_scores: dict[str, int], player_keys: dict[str, str]) -> None:
        self.client = client
        self.durable_scores = durable_scores
        self.player_keys = player_keys
        self.processed_event_ids: set[str] = set()
        self.projection_backlog: list[ScoreEvent] = []
        self.redis_available = True

    def process(self, event: ScoreEvent) -> str:
        if event.event_id in self.processed_event_ids:
            return "duplicate"
        self.processed_event_ids.add(event.event_id)
        self.durable_scores[event.player_id] = event.new_score
        if not self.redis_available:
            self.projection_backlog.append(event)
            return "durable-only"
        key = self.player_keys[event.player_id]
        self.client.zadd(key, {event.player_id: event.new_score})
        return "applied"

    def recover_projection(self) -> int:
        self.redis_available = True
        replayed = 0
        for event in self.projection_backlog:
            key = self.player_keys[event.player_id]
            latest_score = self.durable_scores[event.player_id]
            self.client.zadd(key, {event.player_id: latest_score})
            replayed += 1
        self.projection_backlog.clear()
        return replayed

def run_failures(client: redis.Redis) -> None:
    wait_for_redis(client)
    client.flushdb()
    scores, player_keys = seed_partitioned(client)
    processor = DurableScoreProcessor(client, scores, player_keys)
    retry_player = "player-00010"
    retry_score = scores[retry_player] + 500
    retry_event = ScoreEvent("event-retry-1", retry_player, retry_score)
    first_status = processor.process(retry_event)
    second_status = processor.process(retry_event)
    stored_retry_score = client.zscore(player_keys[retry_player], retry_player)
    duplicate_pass = first_status == "applied" and second_status == "duplicate" and stored_retry_score == float(retry_score)
    cached_top_k = global_top_k(client)
    champion = min(scores, key=scores.get)
    winning_score = max(scores.values()) + 10_000
    processor.redis_available = False
    outage_status = processor.process(ScoreEvent("event-outage-1", champion, winning_score))
    redis_score_during_outage = client.zscore(player_keys[champion], champion)
    replayed = processor.recover_projection()
    redis_score_after_recovery = client.zscore(player_keys[champion], champion)
    refreshed_top_k = global_top_k(client)
    outage_pass = outage_status == "durable-only" and processor.durable_scores[champion] == winning_score and redis_score_during_outage != float(winning_score) and redis_score_after_recovery == float(winning_score) and replayed == 1
    stale_ids = [player_id for player_id, _ in cached_top_k]
    refreshed_ids = [player_id for player_id, _ in refreshed_top_k]
    delayed_aggregation_pass = champion not in stale_ids and refreshed_ids[0] == champion
    print("FAILURE DRILL")
    print(f"Duplicate retry: {'PASS' if duplicate_pass else 'FAIL'}")
    print(f"Redis outage replay: {'PASS' if outage_pass else 'FAIL'}")
    print(f"Delayed aggregation: {'PASS' if delayed_aggregation_pass else 'FAIL'}")
    print("Failure contract: durable writes continue; ranking reads are stale or unavailable until projection recovery.")

def main() -> None:
    parser = argparse.ArgumentParser(description="Local leaderboard scaling and correctness lab")
    parser.add_argument("command", choices=("setup", "capacity", "naive", "partitioned", "failures", "all"))
    args = parser.parse_args()
    if args.command == "capacity":
        print_capacity_model()
        return
    client = connect()
    if args.command == "setup":
        run_setup(client)
    elif args.command == "naive":
        run_naive(client)
    elif args.command == "partitioned":
        run_partitioned(client)
    elif args.command == "failures":
        run_failures(client)
    else:
        run_setup(client)
        print()
        print_capacity_model()
        print()
        run_naive(client)
        print()
        run_partitioned(client)
        print()
        run_failures(client)

if __name__ == "__main__":
    main()

What does this reference contain?

This file combines the setup, capacity, naïve, partitioned, and failure experiments. Use it to compare structure or find a missing line.

Map the failure contract to AWS

The local durable dictionary represents the source of truth. The Redis Sorted Sets represent a rebuildable ranking projection.

The production design maps ingestion to Amazon Kinesis Data Streams. An idempotent AWS Lambda writer stores durable state in Amazon DynamoDB.

DynamoDB Streams drives a separate projector into Amazon ElastiCache. A read Lambda exposes ranking results through Amazon API Gateway.

• In ARCHITECTURE.md, select all existing content.
• Replace the selected content with the architecture boundary and query model below:

# 12 Million-Player Leaderboard Architecture

## Requirement boundary

The fixed requirement is 12,000,000 concurrent players. The baseline scenario is 240,000 score updates/s, 100,000 leaderboard reads/s, 25,000 top-K reads/s, 75,000 around-me reads/s, 12 active partitions, 1,000,000 average players per partition, and a 2x hot-partition planning multiplier. These are sizing assumptions, not observed production traffic.

## Measured local obstacle

The naïve experiment stores all players in `lb:global:all:s1`; all simulated writes target one Sorted Set key. A laptop benchmark is not a production capacity claim. One Redis Sorted Set remains concentrated on the shard that owns its key.

## Partition and query model

Active keys use `lb:<region>:<mode>:<season>`. Partition top-K and partition-local around-me are exact. Global top-K merges K candidates from every partition and is exact at its aggregation point. Exact global competition rank is `1 + sum(players with strictly greater score across all partitions)`; ties share rank and the query fans out across partitions. Materialized global top-K responses expose an `as_of` freshness marker. Exact global around-me is not the high-QPS hot path.

What does this boundary defend?

• The requirement section separates fixed concurrency from scenario assumptions.
• The obstacle section records key concentration without treating laptop throughput as production capacity.
• The query section states which answers are exact.
• The freshness marker makes delayed aggregation visible to callers.

• Save ARCHITECTURE.md.
• Open its Markdown preview by pressing Shift+Command+V (macOS) or Ctrl+Shift+V (Windows).

You should see the requirement boundary, measured obstacle, and partition query model as three rendered sections.

Does the Markdown preview look incomplete?

Confirm the heading markers remain at the start of their lines. Check that the full query model paragraph appears beneath its heading.

Help me check the Markdown structure.

• Return to ARCHITECTURE.md in Visual Studio Code.
• Append the AWS production mapping below the partition query model:

## AWS production mapping

Trusted game servers publish score events to Amazon Kinesis Data Streams. An idempotent AWS Lambda writer validates events and conditionally writes durable score and event state to Amazon DynamoDB. DynamoDB Streams invokes a projector AWS Lambda that updates the appropriate Amazon ElastiCache partitioned Sorted Set. A read AWS Lambda serves partition reads and materialized global top-K through Amazon API Gateway. A top-K aggregator Lambda creates the materialized global result.

How does the production data flow work?

• Trusted game servers publish score events into the ingestion stream.
• The writer applies idempotent durable updates before projection.
• The stream invokes an independently retryable cache projector.
• The read path serves partition queries and materialized global top-K results.

• Save ARCHITECTURE.md.
• Return to the Markdown preview.

You should see an AWS production mapping section that traces each score from ingestion to durable storage, projection, and reads.

Is the mapping merged into the previous paragraph?

Confirm there is a blank line before the AWS production mapping heading. The heading must begin with two hash characters.

Help me fix the section break.

• Return to ARCHITECTURE.md in Visual Studio Code.
• Append the failure contract below the AWS production mapping:

## Failure contract

| Failure | Expected behavior | Correctness statement |
| --- | --- | --- |
| Producer or consumer retry | The same event may arrive again | Event-ID deduplication makes identical retries idempotent |
| ElastiCache projection fails | Durable write remains in DynamoDB | Projector retries; ranked reads are stale or unavailable until caught up |
| ElastiCache data is lost | Rebuild partition keys from durable scores | Ranking is a rebuildable projection, not the source of truth |
| Global top-K aggregation is delayed | Cached result remains readable | Response exposes `as_of`; exactness applies to that aggregation point |
| One region or mode becomes hot | One partition exceeds average load | Monitor skew and split the product partition if required |
| Exact global rank fan-out is overloaded | Expensive reads amplify across partitions | Rate-limit, queue, or relax freshness |

No AWS resources are deployed by this project.

What does the failure contract guarantee?

• Duplicate delivery changes durable state once.
• Projection failure preserves the durable score for retry or rebuild.
• Delayed global top-K remains readable with an explicit freshness boundary.
• Overloaded global rank queries degrade through rate limits, queues, or relaxed freshness.

• Save ARCHITECTURE.md.
• Return to the Markdown preview.

You should see a six-row failure table followed by confirmation that this project deploys no AWS resources.

Does the failure table render as plain text?

Confirm each table row starts and ends with a vertical bar. Check that the separator row remains directly below the table header.

Help me repair the Markdown table.

✔️ Awesome, I've got everything!

Your architecture record now connects every local failure observation to a production behavior.

ⓧ I'd like to double check the full code

# 12 Million-Player Leaderboard Architecture

## Requirement boundary

The fixed requirement is 12,000,000 concurrent players. The baseline scenario is 240,000 score updates/s, 100,000 leaderboard reads/s, 25,000 top-K reads/s, 75,000 around-me reads/s, 12 active partitions, 1,000,000 average players per partition, and a 2x hot-partition planning multiplier. These are sizing assumptions, not observed production traffic.

## Measured local obstacle

The naïve experiment stores all players in `lb:global:all:s1`; all simulated writes target one Sorted Set key. A laptop benchmark is not a production capacity claim. One Redis Sorted Set remains concentrated on the shard that owns its key.

## Partition and query model

Active keys use `lb:<region>:<mode>:<season>`. Partition top-K and partition-local around-me are exact. Global top-K merges K candidates from every partition and is exact at its aggregation point. Exact global competition rank is `1 + sum(players with strictly greater score across all partitions)`; ties share rank and the query fans out across partitions. Materialized global top-K responses expose an `as_of` freshness marker. Exact global around-me is not the high-QPS hot path.

## AWS production mapping

Trusted game servers publish score events to Amazon Kinesis Data Streams. An idempotent AWS Lambda writer validates events and conditionally writes durable score and event state to Amazon DynamoDB. DynamoDB Streams invokes a projector AWS Lambda that updates the appropriate Amazon ElastiCache partitioned Sorted Set. A read AWS Lambda serves partition reads and materialized global top-K through Amazon API Gateway. A top-K aggregator Lambda creates the materialized global result.

## Failure contract

| Failure | Expected behavior | Correctness statement |
| --- | --- | --- |
| Producer or consumer retry | The same event may arrive again | Event-ID deduplication makes identical retries idempotent |
| ElastiCache projection fails | Durable write remains in DynamoDB | Projector retries; ranked reads are stale or unavailable until caught up |
| ElastiCache data is lost | Rebuild partition keys from durable scores | Ranking is a rebuildable projection, not the source of truth |
| Global top-K aggregation is delayed | Cached result remains readable | Response exposes `as_of`; exactness applies to that aggregation point |
| One region or mode becomes hot | One partition exceeds average load | Monitor skew and split the product partition if required |
| Exact global rank fan-out is overloaded | Expensive reads amplify across partitions | Rate-limit, queue, or relax freshness |

No AWS resources are deployed by this project.

What should you compare?

Use this reference to confirm the workload boundary, production mapping, failure table, and final no-deployment statement appear in the same order.

Before you run the final check, do you expect all three failure guarantees to remain intact after the architecture edits?

• Run the failure drill one final time with this command:

python3 leaderboard_lab.py failures

What confirms the final state?

You should see Duplicate retry: PASS, Redis outage replay: PASS, and Delayed aggregation: PASS.

The final failure-contract line confirms durable writes continue while ranking reads remain stale or unavailable until projection recovery.

Does the final run fail to connect?

Confirm the local Redis container from the earlier step is still running. Then rerun the failure command from the same project terminal with the virtual environment active.

Help me restore the local failure drill.

You now have measured failure evidence plus an architecture contract that explains duplicate delivery, cache recovery, and stale reads. No AWS resources were deployed, so this production design adds no cloud charges.

Secret mission

Reject Out-of-Order Score Events

A unique event can still arrive too late and overwrite the correct score. Add per-player sequence checks so your leaderboard rejects stale updates while preserving the newer durable score and Redis projection.

Clean Up Your Resources

Clean Up Your Resources

Everything you built runs locally on your Mac. No AWS resources were deployed, so there are no cloud charges or ongoing service costs.

Resources you used:

• A local Redis 8.10.1 container.
• A Docker Compose network.
• A disposable .venv virtual environment with redis-py 8.1.0.
• The local leaderboard_lab.py experiment file.
• The local out_of_order_mission.py mission file.
• The local ARCHITECTURE.md design document.
• The local requirements.txt dependency file.
• The local compose.yaml configuration file.

Keep everything running

No action needed. Choose this if you want to rerun the experiments or challenge the workload assumptions later.

• Leave the Redis container running for more leaderboard experiments.
• Keep the virtual environment ready with the pinned Python client.
• Keep the local project files as a system-design reference.
• Expect no AWS charges because the production architecture exists only in ARCHITECTURE.md.

Pause - I'll come back to this later

Shut down the Redis process to free local memory. Your container data and project files stay available for your return.

• Switch back to the project terminal from earlier.
• Stop the Redis container without removing it by running this command:

docker compose stop

What does this command do?

Docker Compose stops the Redis service while preserving its container. Your source files and virtual environment remain untouched.

• Return to Docker Desktop from earlier.
• Confirm the Redis container shows a stopped state.

Your local runtime is paused. The lab remains ready for another experiment.

Container Still Running?

• Confirm Docker Desktop is still running.
• Retry the stop command from the project terminal.
• Get help with the stopped container.

Delete - I don't want to use this again

Remove the local runtime and every project artifact listed above. Deleting the lab feels final because it is, so the commands name each target explicitly.

Deletion Is Permanent

The next commands remove the Redis container plus its network. They also delete the virtual environment and local project files.

• Switch back to the project terminal from earlier.
• Remove the Redis container and Docker Compose network by running this command:

docker compose down

What does this command do?

Docker Compose stops the Redis service. It removes the project container and its Compose network.

• Return to Docker Desktop from earlier.
• Confirm the project Redis container no longer appears.

Runtime Still Listed?

• Wait a few seconds for Docker Desktop to refresh.
• Confirm Docker Desktop is running before retrying the cleanup command.
• Get help removing the local runtime.

The source code and virtual environment remain after the runtime cleanup. The final command removes those local artifacts.

• Return to the project terminal from earlier.
• Delete the virtual environment and named project files by running this command:

rm -rf .venv leaderboard_lab.py out_of_order_mission.py ARCHITECTURE.md requirements.txt compose.yaml

What does this command do?

This removes the virtual environment plus each named project file. Every target is named explicitly, so other files in the workspace remain untouched.

• Return to Visual Studio Code from earlier.
• Confirm the Explorer sidebar no longer lists the deleted files or .venv.

Cleanup complete. Your local Redis runtime and leaderboard project artifacts are now removed.

Files Still Visible?

• Confirm the terminal is still using the project workspace from earlier.
• Refresh the Explorer sidebar in Visual Studio Code.
• Get help removing the remaining files.

Nice Work!

Nice Work!

Strong finish. Your Redis leaderboard lab now gives you measured evidence for defending a production design that serves 12 million concurrent players.

What you learned:

• Built a quantified workload model for 12 million concurrent players. The model turns product behavior into testable traffic assumptions.
• Measured a global hot key receiving 100 percent of simulated writes. Partitioned the leaderboard across 12 product-facing keys. Proved the exact global top-K merge against a truth model. Verified exact competition rank through partition fan-out.
• Proved idempotency under duplicate delivery. Demonstrated Redis projection replay after an outage. Exposed delayed global top-K as stale data. Mapped these outcomes onto an AWS architecture without deploying cloud resources.
• Secret Mission: Added per-player sequence checks that reject a unique delayed event before it can overwrite a newer score.

Ready to quiz yourself?