Pressure-Test Discord Message Storage

Simulate Discord-style message partitions and measure hotspot mitigation.

Introduction

30 Second Summary

One unusually busy conversation can send half of a message service's work to the same place. That crowded destination can slow down people whose conversations are completely unrelated.

In this project, you will build a deterministic Python workload simulator for a Discord-like message service. You will use its before-and-after metrics to create a system design case study about partition keys, time bucketing, and hot partition mitigation.

What You'll Build

Your simulator will print a side-by-side report where the hottest partition drops from 50,000 messages to 5,000, giving you measured evidence to defend the redesigned storage model.

By the end of this project, you'll have:

  • A deterministic hot-channel experiment you can run to expose a dominant partition carrying 50.25% of the workload.
  • A measured strategy comparison that shows how 10 time buckets reduce maximum partition load by a factor of 10.
  • An interview-ready architecture case study connecting your results to ScyllaDB, request coalescing, replication, and a dual-write migration.
  • Secret Mission: Compare 5, 10, and 20 time buckets to defend one bucket size with measured evidence.

Are there any prerequisites?

Familiarity with APIs, databases, replication, and horizontal scaling will help you focus on the design trade-offs.

The hands-on work requires Python 3 plus Visual Studio Code on a Mac. No ScyllaDB cluster or third-party Python packages are required.

Before We Start

A useful system design starts with a testable hypothesis. Before the hands-on work begins, you will define how time bucketing could reduce the absolute impact of a hot Discord channel while preserving efficient channel-local reads.

Set Up the Experiment Workspace

The storage hypothesis now needs a clean local home. A minimal workspace keeps your attention on partition behavior.

Visual Studio Code will hold both project artifacts. Its integrated terminal gives you one place to verify Python 3 before the experiment begins.

In this step, get ready to:
  • Open the discord-message-storage folder as your Visual Studio Code workspace.
  • Create the two empty project files.
  • Verify the Python command you will use for the experiment.
Create the workspace folder

A workspace gives Visual Studio Code one folder to treat as the project boundary. The Explorer stays focused on that folder.

  • Press Cmd+Space on macOS to open Spotlight.
  • Type Visual Studio Code into the search field.
  • Press Return to open Visual Studio Code.

You now have the editor ready for the project workspace.

  • Click File in the top menu bar.
  • Select Open Folder....
  • Select Desktop in the folder dialog.
  • Select New Folder.
  • Enter discord-message-storage as the folder name.
  • Press Return to create the folder.

The discord-message-storage folder now exists on your Desktop.

  • Select the discord-message-storage folder.
  • Click Open.
  • Confirm that you trust this new local folder if the Workspace Trust dialog appears.

Good start. You will see discord-message-storage at the top of the Explorer view.

Create the project files

The partition_experiment.py file will hold the deterministic workload simulation. The Markdown file will turn its measurements into a system design case study.

Both files begin empty. This checkpoint confirms the project structure before you add logic.

  • Select the discord-message-storage folder in the Explorer view.
  • Click the New File... button in the Explorer toolbar.
  • Enter partition_experiment.py as the file name.
  • Press Return to create the file.
  • Click the New File... button again.
  • Enter architecture-case-study.md as the file name.
  • Press Return to create the file.

The project skeleton is ready. You will see both empty files beneath discord-message-storage in the Explorer view.

✔️ Awesome, I've got everything!

Both file names appear in the Explorer view. Keep them empty for this workspace checkpoint.

ⓧ I'd like to double check the full code

The complete contents of partition_experiment.py are blank.

The complete contents of architecture-case-study.md are blank.

Why are both files empty?

This step creates the workspace structure only. The next step adds the first runnable simulation to partition_experiment.py.

Verify Python in the integrated terminal

The integrated terminal starts inside the open workspace folder. This keeps every later experiment command tied to discord-message-storage.

  • Click Terminal in the top menu bar.
  • Select New Terminal.

You will see a terminal panel at the bottom of Visual Studio Code. Its prompt starts inside the open workspace.

Before you run the check, which words do you expect to see at the beginning of the version output?

  • Verify the Python interpreter by running this command:
python3 --version

What does this command do?

  • The python3 command selects the Python 3 interpreter available on your Mac.
  • The --version flag prints the interpreter version before Python exits.

You will see output beginning with Python 3. That confirms your interpreter is ready for the simulation.

Python Version Not Showing?

  • Try the working python3.x command available on your Mac if python3 is unavailable.
  • Confirm that the successful command prints output beginning with Python 3.
  • Use the successful Python command consistently throughout the remaining steps.

Still stuck? Help me identify the working Python 3 command on my Mac. You can also share the command output with the NextWork community.

Your workspace now has the files and interpreter needed for a reproducible storage experiment. Next, you will make the channel-only hot partition visible with measured evidence.

Make the Hot Partition Visible

Your experiment workspace is ready. The next job is to turn a storage assumption into a measurable baseline.

A partition key decides which messages share storage. This step tests a channel-only key against one unusually active channel so you can see the imbalance before adding any mitigation.

In this step, get ready to:
  • Generate a deterministic workload with one unusually active channel.
  • Measure the channel-only distribution across six partition-load metrics.
  • Run the baseline experiment to expose the dominant partition.
Build the deterministic workload

A deterministic workload produces the same messages every time you run it. That repeatability makes each later strategy comparison fair.

  • Add the four workload constants to the top of partition_experiment.py by copying this code:
CHANNEL_COUNT = 100
REGULAR_MESSAGES_PER_CHANNEL = 500
HOT_CHANNEL_MESSAGES = 50_000
BUCKET_COUNT = 10

What do these constants represent?

  • CHANNEL_COUNT sets the workload to 100 channels.
  • REGULAR_MESSAGES_PER_CHANNEL gives each regular channel 500 messages.
  • HOT_CHANNEL_MESSAGES gives the unusually active channel 50_000 messages.
  • BUCKET_COUNT divides the generated messages across 10 simulated time windows.
  • Save partition_experiment.py.
  • Confirm the file begins with CHANNEL_COUNT = 100.
  • Add the deterministic message generator below the constants by copying this code:
def message_stream():
    for bucket_id in range(BUCKET_COUNT):
        messages_in_bucket = HOT_CHANNEL_MESSAGES // BUCKET_COUNT
        for _ in range(messages_in_bucket):
            yield "channel-hot", bucket_id

    for channel_number in range(1, CHANNEL_COUNT):
        channel_id = f"channel-{channel_number:03d}"
        for bucket_id in range(BUCKET_COUNT):
            messages_in_bucket = REGULAR_MESSAGES_PER_CHANNEL // BUCKET_COUNT
            for _ in range(messages_in_bucket):
                yield channel_id, bucket_id

How does the workload generator work?

  • The first loop yields 50_000 messages for channel-hot.
  • The second loop creates the other 99 channel identifiers.
  • Each regular channel yields 500 messages.
  • Each yielded value carries a channel identifier plus its simulated window identifier.
  • Save partition_experiment.py.
  • Confirm message_stream() contains one loop for the hot channel plus one loop for the regular channels.

Does the generator look uneven?

Check that each yield line is nested inside its message loop. Python uses indentation to decide which loop repeats each statement.

Still stuck? Help me check the indentation in my message generator.

Measure partition load

The strategy function converts each message into a partition key. The measurement code counts those keys so the workload becomes a distribution you can inspect.

  • Add the channel-only key function plus the percentile helper below message_stream() by copying this code:
def channel_only_key(channel_id, bucket_id):
    return channel_id


def percentile_95(values):
    rank = (95 * len(values) + 99) // 100
    return values[rank - 1]

What do these helpers calculate?

  • channel_only_key() returns only the channel identifier. Every message from that channel receives the same partition key.
  • bucket_id remains available in the function signature. The channel-only strategy does not use it.
  • percentile_95() selects the nearest-rank p95 value from a sorted list of partition loads.
  • Save partition_experiment.py.
  • Confirm channel_only_key() returns channel_id.

Missing one of the helpers?

Place both functions after message_stream(). Keep each function at the left edge of the file.

Need another pair of eyes? Help me check my partition-key and percentile helpers.

  • Add the measurement function below percentile_95() by copying this code:
def measure(strategy_name, key_function):
    partition_loads = {}

    for channel_id, bucket_id in message_stream():
        partition_key = key_function(channel_id, bucket_id)
        partition_loads[partition_key] = partition_loads.get(partition_key, 0) + 1

    loads = sorted(partition_loads.values())
    total_messages = sum(loads)
    mean_load = total_messages / len(loads)
    max_load = max(loads)

    return {
        "strategy": strategy_name,
        "partitions": len(loads),
        "max_load": max_load,
        "mean_load": mean_load,
        "p95_load": percentile_95(loads),
        "hottest_share": 100 * max_load / total_messages,
        "max_to_mean": max_load / mean_load,
    }

What does the measurement capture?

  • partition_loads counts how many messages map to each generated key.
  • loads sorts those counts for the percentile calculation.
  • total_messages supports the hottest-partition share calculation.
  • The returned dictionary records partition count, maximum load, mean load, p95 load, hottest share, plus the max-to-mean ratio.
  • Save partition_experiment.py.
  • Confirm the returned dictionary contains all six metrics beneath strategy.

Does the measurement block look incomplete?

Check that partition_loads starts as an empty dictionary. Confirm the return dictionary closes with a brace at the same indentation level.

Need help comparing the function? Help me find the mismatch in my measure function.

Expose the channel-only baseline

The final pieces format the metrics as one readable row. The entry point runs only the channel-only strategy so its behavior stands on its own.

  • Add the report formatter below measure() by copying this code:
def print_report(results):
    print(
        f"{'Strategy':<28}"
        f"{'Partitions':>12}"
        f"{'Max':>10}"
        f"{'Mean':>10}"
        f"{'P95':>10}"
        f"{'Top share':>12}"
        f"{'Max/mean':>12}"
    )

    for result in results:
        print(
            f"{result['strategy']:<28}"
            f"{result['partitions']:>12}"
            f"{result['max_load']:>10}"
            f"{result['mean_load']:>10.1f}"
            f"{result['p95_load']:>10}"
            f"{result['hottest_share']:>11.2f}%"
            f"{result['max_to_mean']:>11.2f}x"
        )

How is the report formatted?

  • The first print() call creates aligned column headings.
  • The loop prints one row for each strategy result.
  • mean_load displays with one decimal place.
  • hottest_share displays as a percentage with two decimal places.
  • Save partition_experiment.py.
  • Confirm print_report() contains one heading print plus one result-row print.

Do the formatted strings look misaligned?

Check every opening brace against its closing brace. Keep the quote style exactly as shown around each dictionary key.

Want help checking the formatter? Help me inspect the f-strings in print_report.

  • Add the program entry point below print_report() by copying this code:
def main():
    channel_only = measure("Channel only", channel_only_key)
    print_report([channel_only])


if __name__ == "__main__":
    main()

How does the baseline start?

  • main() measures the workload with channel_only_key.
  • print_report() receives that single result as a list.
  • The __main__ check starts the experiment when you run the file as a script.
  • Save partition_experiment.py.

✔️ Awesome, I've got everything!

Your channel-only experiment is assembled. Keep partition_experiment.py saved before you run it.

ⓧ I'd like to double check the full code

Compare your saved partition_experiment.py with this complete baseline:

CHANNEL_COUNT = 100
REGULAR_MESSAGES_PER_CHANNEL = 500
HOT_CHANNEL_MESSAGES = 50_000
BUCKET_COUNT = 10


def message_stream():
    for bucket_id in range(BUCKET_COUNT):
        messages_in_bucket = HOT_CHANNEL_MESSAGES // BUCKET_COUNT
        for _ in range(messages_in_bucket):
            yield "channel-hot", bucket_id

    for channel_number in range(1, CHANNEL_COUNT):
        channel_id = f"channel-{channel_number:03d}"
        for bucket_id in range(BUCKET_COUNT):
            messages_in_bucket = REGULAR_MESSAGES_PER_CHANNEL // BUCKET_COUNT
            for _ in range(messages_in_bucket):
                yield channel_id, bucket_id


def channel_only_key(channel_id, bucket_id):
    return channel_id


def percentile_95(values):
    rank = (95 * len(values) + 99) // 100
    return values[rank - 1]


def measure(strategy_name, key_function):
    partition_loads = {}

    for channel_id, bucket_id in message_stream():
        partition_key = key_function(channel_id, bucket_id)
        partition_loads[partition_key] = partition_loads.get(partition_key, 0) + 1

    loads = sorted(partition_loads.values())
    total_messages = sum(loads)
    mean_load = total_messages / len(loads)
    max_load = max(loads)

    return {
        "strategy": strategy_name,
        "partitions": len(loads),
        "max_load": max_load,
        "mean_load": mean_load,
        "p95_load": percentile_95(loads),
        "hottest_share": 100 * max_load / total_messages,
        "max_to_mean": max_load / mean_load,
    }


def print_report(results):
    print(
        f"{'Strategy':<28}"
        f"{'Partitions':>12}"
        f"{'Max':>10}"
        f"{'Mean':>10}"
        f"{'P95':>10}"
        f"{'Top share':>12}"
        f"{'Max/mean':>12}"
    )

    for result in results:
        print(
            f"{result['strategy']:<28}"
            f"{result['partitions']:>12}"
            f"{result['max_load']:>10}"
            f"{result['mean_load']:>10.1f}"
            f"{result['p95_load']:>10}"
            f"{result['hottest_share']:>11.2f}%"
            f"{result['max_to_mean']:>11.2f}x"
        )


def main():
    channel_only = measure("Channel only", channel_only_key)
    print_report([channel_only])


if __name__ == "__main__":
    main()

Before you run the experiment, which channel do you think creates the largest partition?

  • Switch back to the integrated terminal from earlier.
  • Run the channel-only experiment with this command:
python3 partition_experiment.py

What does this command do?

python3 runs the saved partition_experiment.py script. The entry point generates the workload before printing its partition metrics.

You'll see 100 partitions. The maximum load is 50000 messages.

The p95 load is 500. The hottest-partition share is 50.25%.

What does the report reveal?

  • The other 99 partitions each hold 500 messages.
  • The channel-hot partition holds 50000 messages.
  • One partition receives just over half of the entire 99500-message workload.
  • This intended shortfall is a hot partition. The channel-only key concentrates a highly active channel in one place.

Does the script fail to print the report?

  • Confirm the integrated terminal still points to the discord-message-storage folder.
  • Check every function name against the full-code tab. Python identifiers are case-sensitive.
  • Check the terminal's reported line in partition_experiment.py for mismatched indentation, quotes, braces, or parentheses.

Still blocked? Help me troubleshoot my channel-only partition experiment.

You now have measured evidence that the channel-only design concentrates the hot channel's traffic. Next, you'll add time buckets to bound the absolute load while tracking the imbalance that remains.

Add Time Buckets and Compare the Distribution

Your channel-only simulation now exposes a hot partition that receives more than half of the workload. Time bucketing is the next pressure test.

Partition size and traffic balance are separate design questions. You will run both strategies against the same 99,500 messages to measure what changes.

In this step, get ready to:
  • Add a bucketed partition key that combines each channel with a time window.
  • Measure both partitioning strategies against the same message stream.
  • Compare all six metrics to identify the improvement and the remaining skew.
Add a bucketed partition key

A partition key determines which partition receives each message. Time bucketing adds the simulated window to that key so one channel can use several bounded partitions.

  • Find the channel_only_key() function in partition_experiment.py.
  • Add the bucketed key function directly below channel_only_key() by pasting this code:
def channel_bucket_key(channel_id, bucket_id):
    return f"{channel_id}:bucket-{bucket_id}"

What does this code do?

  • The channel_bucket_key() function receives the same channel identifier and simulated window identifier produced by message_stream().
  • The formatted string creates keys such as channel-hot:bucket-0.
  • Each channel now maps to 10 bounded partitions across the simulated workload.
Measure both strategies

The existing measure() function accepts any key function. Reusing message_stream() gives both strategies identical channel IDs and bucket IDs.

  • Find the main() function near the bottom of partition_experiment.py.
  • Replace the complete main() function with this code:
def main():
    channel_only = measure("Channel only", channel_only_key)
    bucketed = measure("Channel + 10 buckets", channel_bucket_key)
    print_report([channel_only, bucketed])

    reduction = channel_only["max_load"] / bucketed["max_load"]
    print(f"\nMaximum partition load drops by {reduction:.1f}x.")
    print(
        "The max/mean ratio stays high: bucketing bounds partition size, "
        "but the hot channel is still much busier than a regular channel."
    )

What does this code do?

  • The first measurement keeps the original channel-only key as the baseline.
  • The second measurement uses channel_bucket_key against the same generated messages.
  • The print_report() call places both strategies in one table for direct comparison.
  • The reduction calculation divides the original maximum load by the bucketed maximum load.
  • The final statements summarize the measured reduction and its limit.
  • Save partition_experiment.py.

Before you run the comparison, do you expect every load metric to improve by the same factor?

  • Run the comparison in the integrated terminal with this command:
python3 partition_experiment.py

What should you see?

  • The Channel only row remains at 100 partitions.
  • The Channel + 10 buckets row shows 1000 partitions.
  • The bucketed Max value is 5000.
  • The bucketed P95 value is 50.
  • The bucketed Top share value is 5.03%.
  • The bucketed Max/mean value remains 50.25x.
  • The summary reports a maximum partition-load reduction of 10.0x.

Missing the bucketed results?

  • Confirm that main() passes both channel_only and bucketed to print_report() if the report still shows one row.
  • Check that channel_bucket_key() returns a key containing both channel_id and bucket_id if the partition count remains 100.
  • Compare the indentation in your main() function with the code above if Python reports a syntax problem.

Still stuck? Help me debug why my bucketed partition report is missing or incorrect.

Great work. Your simulator now compares both partition keys against the same 99,500 messages.

Interpret all six metrics

The table separates absolute partition size from relative load imbalance. Reading each column across both rows shows exactly what the bucketed key changes.

  • Locate the six metric columns in the terminal report.
  • Read both rows within one column before moving to the next metric.

What does the comparison prove?

  • The Partitions count increases from 100 to 1000. Ten windows now exist for every channel.
  • The Max load falls from 50000 to 5000. Bucketing caps the largest hot-channel partition at one tenth of its previous load.
  • The Mean load falls from 995.0 to 99.5. The same workload is spread over 10 times as many partitions.
  • The p95 partition load falls from 500 to 50. Most regular-channel partitions are split into smaller windows too.
  • The hottest-partition share falls from 50.25% to 5.03%. One partition now owns about one tenth as much of the total workload.
  • The max-to-mean ratio remains 50.25x. The maximum shrinks by a factor of 10. The mean does too. Relative skew therefore remains visible.

✔️ Awesome, I've got everything!

Your saved partition_experiment.py now measures both strategies and explains the remaining skew.

ⓧ I'd like to double check the full code

  • Compare your complete partition_experiment.py file with this reference:
CHANNEL_COUNT = 100
REGULAR_MESSAGES_PER_CHANNEL = 500
HOT_CHANNEL_MESSAGES = 50_000
BUCKET_COUNT = 10


def message_stream():
    for bucket_id in range(BUCKET_COUNT):
        messages_in_bucket = HOT_CHANNEL_MESSAGES // BUCKET_COUNT
        for _ in range(messages_in_bucket):
            yield "channel-hot", bucket_id

    for channel_number in range(1, CHANNEL_COUNT):
        channel_id = f"channel-{channel_number:03d}"
        for bucket_id in range(BUCKET_COUNT):
            messages_in_bucket = REGULAR_MESSAGES_PER_CHANNEL // BUCKET_COUNT
            for _ in range(messages_in_bucket):
                yield channel_id, bucket_id


def channel_only_key(channel_id, bucket_id):
    return channel_id


def channel_bucket_key(channel_id, bucket_id):
    return f"{channel_id}:bucket-{bucket_id}"


def percentile_95(values):
    rank = (95 * len(values) + 99) // 100
    return values[rank - 1]


def measure(strategy_name, key_function):
    partition_loads = {}

    for channel_id, bucket_id in message_stream():
        partition_key = key_function(channel_id, bucket_id)
        partition_loads[partition_key] = partition_loads.get(partition_key, 0) + 1

    loads = sorted(partition_loads.values())
    total_messages = sum(loads)
    mean_load = total_messages / len(loads)
    max_load = max(loads)

    return {
        "strategy": strategy_name,
        "partitions": len(loads),
        "max_load": max_load,
        "mean_load": mean_load,
        "p95_load": percentile_95(loads),
        "hottest_share": 100 * max_load / total_messages,
        "max_to_mean": max_load / mean_load,
    }


def print_report(results):
    print(
        f"{'Strategy':<28}"
        f"{'Partitions':>12}"
        f"{'Max':>10}"
        f"{'Mean':>10}"
        f"{'P95':>10}"
        f"{'Top share':>12}"
        f"{'Max/mean':>12}"
    )

    for result in results:
        print(
            f"{result['strategy']:<28}"
            f"{result['partitions']:>12}"
            f"{result['max_load']:>10}"
            f"{result['mean_load']:>10.1f}"
            f"{result['p95_load']:>10}"
            f"{result['hottest_share']:>11.2f}%"
            f"{result['max_to_mean']:>11.2f}x"
        )


def main():
    channel_only = measure("Channel only", channel_only_key)
    bucketed = measure("Channel + 10 buckets", channel_bucket_key)
    print_report([channel_only, bucketed])

    reduction = channel_only["max_load"] / bucketed["max_load"]
    print(f"\nMaximum partition load drops by {reduction:.1f}x.")
    print(
        "The max/mean ratio stays high: bucketing bounds partition size, "
        "but the hot channel is still much busier than a regular channel."
    )


if __name__ == "__main__":
    main()

How does this file fit together?

The file generates one deterministic workload before measuring it with two partition-key functions. The shared reporting path keeps the comparison consistent.

Your measurements now show that time bucketing bounds absolute partition load while relative skew stays visible. Next, you turn that evidence into an architecture case study.

Write the Architecture Case Study

Your simulator now proves that time bucketing reduces the absolute load of the hottest partition. Those measurements need an architecture argument before they become useful interview evidence.

The case study begins with query-first data modeling. It then connects your evidence to Discord's documented ScyllaDB architecture.

You will close with operational failure mitigations. You will also document a staged migration plan.

In this step, get ready to:
  • Turn the measured simulator output into a query-first storage argument.
  • Map the request path from the API layer to ScyllaDB.
  • Document failure mitigations plus a five-stage migration plan.
Frame the evidence and data model

Query-first design starts with the reads the system must serve. The partition key groups one channel window while the clustering key preserves chronological message order.

  • Select architecture-case-study.md from the left sidebar in Visual Studio Code.
  • Build the executive summary and logical model by pasting this content into the empty file:
# Discord-Like Message Storage: Quantifying a Hot-Channel Redesign

## Executive summary

This case study tests a narrow design question: how does adding a time bucket to a channel-based message partition change the load created by one unusually active channel?

The experiment models 100 channels and 99,500 messages. One hot channel produces 50,000 messages, while each of the other 99 channels produces 500. A channel-only key places 50.25% of all messages in one partition. Adding 10 time buckets cuts the largest partition from 50,000 to 5,000 messages and its share from 50.25% to 5.03%.

Bucketing does not eliminate skew. The max-to-mean ratio remains 50.25 because the hot channel is still 100 times busier than a regular channel. The redesign bounds absolute partition growth, while upstream request coalescing and concurrency control are still needed for synchronized read spikes.

## Requirements and access patterns

Functional requirements:

- Append a message to a channel.
- Fetch the newest messages for one channel in chronological order.
- Continue pagination from a known message identifier.
- Retrieve older history by moving across time buckets.

Non-functional requirements:

- Keep writes horizontally distributable.
- Preserve low-latency channel-local reads.
- Prevent one large channel from creating an unbounded partition.
- Tolerate node or zone failures through replication.
- Migrate storage engines without planned downtime or silent data loss.

The primary query is channel history, so the logical partition key is `(channel_id, bucket)`. The clustering key is the chronologically sortable `message_id`. This keeps one channel and time window together while preserving ordered range reads inside the partition.

How does this frame the design?

  • The executive summary states the storage question before presenting the measured answer.
  • The requirements define the reads and operational properties that the design must support.
  • The logical key (channel_id, bucket) bounds each partition to one channel window.
  • The clustering key message_id preserves the order needed for history reads.
  • Save architecture-case-study.md.
  • Select the Markdown preview icon in the top-right of the editor.

You should see the case-study title followed by the executive summary. The functional requirements should render as a four-item list.

Does the Markdown look unformatted?

  • Confirm that the file name ends with .md.
  • Check that each heading begins with the exact number of hash characters shown in the code.

Still stuck? Help me fix the Markdown formatting in my architecture case study.

The measured tables make the design claim reproducible. They also preserve the key nuance that lower absolute load does not remove relative skew.

  • Append the workload model and measured comparison by pasting this content below the existing text:


## Quantitative workload model

| Input | Value |
| --- | ---: |
| Total channels | 100 |
| Regular channels | 99 |
| Messages per regular channel | 500 |
| Hot-channel messages | 50,000 |
| Total messages | 99,500 |
| Time buckets | 10 |

## Results

| Strategy | Partitions | Maximum load | Mean load | P95 load | Hottest share | Max/mean |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| Channel only | 100 | 50,000 | 995.0 | 500 | 50.25% | 50.25x |
| Channel plus 10 buckets | 1,000 | 5,000 | 99.5 | 50 | 5.03% | 50.25x |

Interpretation:

- The maximum partition load falls by 10 times.
- The hottest partition's share falls by 10 times.
- P95 falls because most regular-channel partitions are also split into 10 smaller windows.
- Relative skew remains. Time bucketing controls partition size, but it does not make a celebrity channel as quiet as a small channel.

What do the tables prove?

  • The workload table records the assumptions behind the experiment.
  • The results table compares both strategies against the same 99,500 messages.
  • The maximum load and hottest share each fall by a factor of 10.
  • The unchanged max-to-mean ratio shows that uniform bucketing preserves the relative traffic imbalance.
  • Save architecture-case-study.md.
  • Return to the rendered Markdown preview.

You should now see two rendered tables. The Results table should contain one row for each storage strategy.

Are the tables showing as plain text?

  • Check that every table row begins and ends with a vertical bar.
  • Confirm that the alignment row sits directly below its table header.

Need another pair of eyes? Help me debug why my Markdown tables are not rendering.

Map the request path and failure modes

The request path shows where each protective mechanism acts. Request coalescing reduces duplicate reads before they reach storage.

Consistent hash routing brings requests for the same channel to one data-service instance. Replication and quorum consistency shape availability plus tail latency inside the database.

  • Append the proposed request path by pasting this content below the Results interpretation:


## Proposed architecture

```text
Clients
   |
API layer
   |
Consistent hash routing by channel_id
   |
Data service instance
   |  request coalescing for identical concurrent reads
   |
ScyllaDB message store
   |  partition key: (channel_id, bucket)
   |  clustering key: message_id
   |
Replicas serving the configured consistency level
```

The data service is intentionally separate from business logic. Routing all requests for a channel to the same service instance makes coalescing effective: concurrent requests for the same row can share one database query instead of multiplying load during a major announcement.

How does the request path reduce load?

  • The API layer sends each message request toward the data service.
  • Consistent hash routing keeps one channel's requests on the same service instance.
  • Request coalescing lets identical concurrent reads share one database query.
  • ScyllaDB stores messages by channel bucket in chronological clustering order.
  • Save architecture-case-study.md.
  • Return to the rendered Markdown preview.

The request path should now render as a fixed-width diagram. You should be able to trace it from Clients to the replicas.

Is the architecture diagram broken?

  • Confirm that the diagram begins with three backticks followed by text.
  • Confirm that another line of three backticks closes the diagram.

Still seeing misplaced diagram text? Help me repair the fenced architecture diagram in my Markdown file.

Time bucketing limits historical partition growth. The failure analysis now addresses the load that can still concentrate in the active bucket.

  • Append the three failure modes and mitigations by pasting this content below the data-service explanation:


## Failure modes and mitigations

1. Hot current bucket
   - Risk: A live event can still overload the current bucket even when historical partitions are bounded.
   - Mitigation: Coalesce identical reads, apply bounded concurrency, monitor per-partition latency, and shorten the bucket only when measurements justify the extra fan-out.

2. Replica slowdown under quorum
   - Risk: A slow replica can raise tail latency for requests waiting on multiple replicas.
   - Mitigation: Maintain capacity headroom, isolate workloads, and make consistency choices explicit for each query.

3. Oversized history reads
   - Risk: Smaller buckets increase how many partitions an old-history request may traverse.
   - Mitigation: Page from a known message and query only the bucket containing that cursor before moving to the previous bucket.

Why these failure modes?

  • The hot-current-bucket scenario covers synchronized traffic during a live event.
  • The replica scenario covers tail latency when a quorum waits on slower nodes.
  • The history-read scenario covers the extra query fan-out created by smaller buckets.
  • Save architecture-case-study.md.
  • Return to the rendered Markdown preview.

You should see three numbered failure modes. Each one should contain a nested risk plus a nested mitigation.

Are the risks missing their indentation?

  • Keep three spaces before each nested risk or mitigation bullet.
  • Leave one blank line between the numbered failure modes.

Need help with the nested lists? Help me fix the indentation in my failure-mode section.

Document the migration and interview defense

A dual-write migration keeps new data synchronized while historical records move in the background. Sampled read comparisons provide evidence before primary reads cut over.

The interview defense turns each design choice into a concise explanation. It also names the limits of that choice.

  • Append the migration plan and interview defense by pasting this content below the failure modes:


## Migration plan

1. Provision the target ScyllaDB storage path and validate its schema and capacity.
2. Dual-write new messages to the old and new stores.
3. Backfill historical token ranges with checkpoints so interrupted work can resume.
4. Send a sample of reads to both stores and compare results automatically.
5. Cut primary reads to the new store while retaining a rollback window before retiring the old path.

## Interview defense

Why not partition only by channel?

A single large channel grows without a bound and concentrates traffic. The experiment places 50.25% of the workload in one partition.

What does time bucketing solve?

It bounds the amount of data in each partition and lowers the absolute hottest-partition load. In this experiment, 10 buckets produce a 10-times reduction.

What does time bucketing not solve?

It does not remove real-time skew. The active bucket for a large channel can still be far busier than every regular-channel bucket.

Why use request coalescing?

Many users can request the same row after a major announcement. Coalescing converts duplicate concurrent reads into shared database work before the spike reaches storage.

How would you migrate without downtime?

Dual-write, checkpoint the backfill, compare sampled reads between stores, cut over only after validation, and retain a rollback path.

How does the migration control risk?

  • Dual-writing keeps incoming messages available in both storage paths.
  • Checkpointed backfills make interrupted historical migration work resumable.
  • Sampled comparisons detect mismatched reads before the cutover.
  • The rollback window preserves a recovery path after primary reads move.
  • Save architecture-case-study.md.
  • Return to the rendered Markdown preview.

Your case study should now show five numbered migration stages. The interview defense should contain five questions with evidence-based answers.

Are migration stages or questions missing?

  • Confirm that the migration list is numbered from 1 through 5.
  • Check that every interview question has its answer directly below it.

Want help comparing the structure? Help me find the missing migration stage or interview answer.

The evidence section gives readers a path back to the documented architecture. It separates measured project assumptions from externally documented design choices.

  • Complete the case study by appending these evidence links below the interview defense:


## Evidence

- Discord, "How Discord Stores Trillions of Messages": https://discord.com/blog/how-discord-stores-trillions-of-messages/
- Discord, "How Discord Automates ScyllaDB Clusters at Scale": https://discord.com/blog/how-discord-automates-scylladb-clusters-at-scale
- ScyllaDB, "Query Design": https://docs.scylladb.com/stable/get-started/data-modeling/query-design.html

What does the evidence support?

  • The Discord storage source supports the partitioning model plus the data-service design.
  • The Discord automation source provides current ScyllaDB operational context.
  • The ScyllaDB query-design source supports query-first partitioning plus clustering choices.

Before the final check, which sections do you expect to prove that this design handles both data distribution and operational risk?

  • Save architecture-case-study.md.
  • Return to the rendered Markdown preview.

You should see functional and non-functional requirements above the measured Results table.

The request path should run from Clients through the API layer, consistent hash routing, a data service, and ScyllaDB.

The file should also show three numbered failure modes plus five numbered migration stages.

Is a final section missing?

  • Scroll through the rendered preview to locate each second-level heading.
  • Compare the file against the full reference below if a heading or paragraph is absent.

Need help finding the mismatch? Help me compare my architecture case study with the expected final structure.

✔️ Awesome, I've got everything!

Your case study now connects the simulator's measurements to storage design decisions. Make sure architecture-case-study.md is saved.

ⓧ I'd like to double check the full code

Compare architecture-case-study.md with this complete reference:

# Discord-Like Message Storage: Quantifying a Hot-Channel Redesign

## Executive summary

This case study tests a narrow design question: how does adding a time bucket to a channel-based message partition change the load created by one unusually active channel?

The experiment models 100 channels and 99,500 messages. One hot channel produces 50,000 messages, while each of the other 99 channels produces 500. A channel-only key places 50.25% of all messages in one partition. Adding 10 time buckets cuts the largest partition from 50,000 to 5,000 messages and its share from 50.25% to 5.03%.

Bucketing does not eliminate skew. The max-to-mean ratio remains 50.25 because the hot channel is still 100 times busier than a regular channel. The redesign bounds absolute partition growth, while upstream request coalescing and concurrency control are still needed for synchronized read spikes.

## Requirements and access patterns

Functional requirements:

- Append a message to a channel.
- Fetch the newest messages for one channel in chronological order.
- Continue pagination from a known message identifier.
- Retrieve older history by moving across time buckets.

Non-functional requirements:

- Keep writes horizontally distributable.
- Preserve low-latency channel-local reads.
- Prevent one large channel from creating an unbounded partition.
- Tolerate node or zone failures through replication.
- Migrate storage engines without planned downtime or silent data loss.

The primary query is channel history, so the logical partition key is `(channel_id, bucket)`. The clustering key is the chronologically sortable `message_id`. This keeps one channel and time window together while preserving ordered range reads inside the partition.

## Quantitative workload model

| Input | Value |
| --- | ---: |
| Total channels | 100 |
| Regular channels | 99 |
| Messages per regular channel | 500 |
| Hot-channel messages | 50,000 |
| Total messages | 99,500 |
| Time buckets | 10 |

## Results

| Strategy | Partitions | Maximum load | Mean load | P95 load | Hottest share | Max/mean |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| Channel only | 100 | 50,000 | 995.0 | 500 | 50.25% | 50.25x |
| Channel plus 10 buckets | 1,000 | 5,000 | 99.5 | 50 | 5.03% | 50.25x |

Interpretation:

- The maximum partition load falls by 10 times.
- The hottest partition's share falls by 10 times.
- P95 falls because most regular-channel partitions are also split into 10 smaller windows.
- Relative skew remains. Time bucketing controls partition size, but it does not make a celebrity channel as quiet as a small channel.

## Proposed architecture

```text
Clients
   |
API layer
   |
Consistent hash routing by channel_id
   |
Data service instance
   |  request coalescing for identical concurrent reads
   |
ScyllaDB message store
   |  partition key: (channel_id, bucket)
   |  clustering key: message_id
   |
Replicas serving the configured consistency level
```

The data service is intentionally separate from business logic. Routing all requests for a channel to the same service instance makes coalescing effective: concurrent requests for the same row can share one database query instead of multiplying load during a major announcement.

## Failure modes and mitigations

1. Hot current bucket
   - Risk: A live event can still overload the current bucket even when historical partitions are bounded.
   - Mitigation: Coalesce identical reads, apply bounded concurrency, monitor per-partition latency, and shorten the bucket only when measurements justify the extra fan-out.

2. Replica slowdown under quorum
   - Risk: A slow replica can raise tail latency for requests waiting on multiple replicas.
   - Mitigation: Maintain capacity headroom, isolate workloads, and make consistency choices explicit for each query.

3. Oversized history reads
   - Risk: Smaller buckets increase how many partitions an old-history request may traverse.
   - Mitigation: Page from a known message and query only the bucket containing that cursor before moving to the previous bucket.

## Migration plan

1. Provision the target ScyllaDB storage path and validate its schema and capacity.
2. Dual-write new messages to the old and new stores.
3. Backfill historical token ranges with checkpoints so interrupted work can resume.
4. Send a sample of reads to both stores and compare results automatically.
5. Cut primary reads to the new store while retaining a rollback window before retiring the old path.

## Interview defense

Why not partition only by channel?

A single large channel grows without a bound and concentrates traffic. The experiment places 50.25% of the workload in one partition.

What does time bucketing solve?

It bounds the amount of data in each partition and lowers the absolute hottest-partition load. In this experiment, 10 buckets produce a 10-times reduction.

What does time bucketing not solve?

It does not remove real-time skew. The active bucket for a large channel can still be far busier than every regular-channel bucket.

Why use request coalescing?

Many users can request the same row after a major announcement. Coalescing converts duplicate concurrent reads into shared database work before the spike reaches storage.

How would you migrate without downtime?

Dual-write, checkpoint the backfill, compare sampled reads between stores, cut over only after validation, and retain a rollback path.

## Evidence

- Discord, "How Discord Stores Trillions of Messages": https://discord.com/blog/how-discord-stores-trillions-of-messages/
- Discord, "How Discord Automates ScyllaDB Clusters at Scale": https://discord.com/blog/how-discord-automates-scylladb-clusters-at-scale
- ScyllaDB, "Query Design": https://docs.scylladb.com/stable/get-started/data-modeling/query-design.html

This reference contains the exact final case study. Use it to identify missing sections without replacing correct work.

That is the case study finished. Your simulator and architecture argument now tell one evidence-backed story.

Secret mission

Find a Defensible Bucket Size

Your current experiment assumes that 10 time buckets offer the right balance. Compare 5, 10, and 20 buckets to choose a size using measured hotspot reduction and read fan-out costs.

Clean Up Your Resources

Clean Up Your Resources

Everything you created stays on your Mac, so there are no cloud resources or ongoing costs. Decide whether to keep the folder, close Visual Studio Code for now, or move the folder to Trash.

Resources you used:

  • Local Python script at discord-message-storage/partition_experiment.py.
  • Local Markdown case study at discord-message-storage/architecture-case-study.md.

Keep everything running

No action needed. Choose this if you want the simulator and case study ready for more experiments.

  • Keep the discord-message-storage folder in its current location.
  • Retain partition_experiment.py for future bucket-size comparisons.
  • Retain architecture-case-study.md as your written system design case study.

Pause - I'll come back to this later

Pausing clears your workspace while preserving every local artifact for later.

  • Close Visual Studio Code.
  • Leave the discord-message-storage folder in its current location.
  • Reopen discord-message-storage in Visual Studio Code when you are ready to continue.

The simulator exits after printing its report, so no background service remains active.

Delete - I don't want to use this again

Removing the folder clears both project files from their current location. They remain recoverable from Trash until you empty it.

  • Close Visual Studio Code.
  • Use Finder to locate the discord-message-storage folder that was open in Visual Studio Code.
  • Move the discord-message-storage folder to Trash.
  • Confirm the discord-message-storage folder no longer appears in its original Finder location.

Nice Work!

Nice Work!

Outstanding work! Your deterministic Python experiment makes one hot partition visible through reproducible metrics. Your architecture case study turns those measurements into a defensible ScyllaDB storage design.

You've learned how to:

  • Built a deterministic workload simulator that exposes a dominant hot partition. Reported six load-distribution metrics for the same 99,500-message workload.
  • Compared channel-only partitioning with time bucketing. Proved that 10 buckets cut maximum load from 50,000 to 5,000. Identified the unchanged 50.25x max-to-mean ratio as evidence of remaining relative skew.
  • Produced a system design case study grounded in query-first data modeling. Explained how Discord-style data services use request coalescing before ScyllaDB. Mapped a five-stage dual-write migration with rollback.
  • Completed the optional Secret Mission by testing three bucket counts. Used measured maximum loads to defend a bucket-size choice against the cost of wider history-read fan-out.

Ready to quiz yourself?