Break a Distributed System on Purpose

Build a 3-node key-value store and observe the CAP theorem firsthand.

Introduction

30 Second Summary

Every time you withdraw money from an ATM or send a message on a chat app, a distributed system is making a split-second decision: should it risk showing you stale data, or should it refuse the request entirely until it can guarantee accuracy?

In this project, you will build a 3-node distributed key-value store, deliberately break the network between nodes, and observe firsthand why no system can guarantee perfect data consistency and availability at the same time.

What You'll Build

A distributed key-value store consisting of 3 Flask nodes running in Docker containers, connected on a shared network with configurable consistency and availability modes.

By the end of this project, you'll have:

  • A running 3-node distributed cluster built with Python, Flask, and Docker.
  • Firsthand evidence of the CAP theorem by observing how a network partition forces a choice between consistency and availability.
  • A clear mental model of CP vs AP systems that maps to real-world databases like PostgreSQL and Cassandra.
  • Secret Mission: Add timestamp-based conflict resolution so nodes can automatically merge diverged data when the partition heals.

Are there any prerequisites?

You should be comfortable with basic Python (variables, functions, dictionaries) and have some familiarity with typing commands in the terminal.

No prior knowledge of distributed systems or Docker networking is required.

Set Up the Project and Build the Node

Every distributed system starts with a single node that knows how to store data and talk to its peers. In this step, you will build that node as a Flask application, then package it into a Docker container so you can run three identical copies on one machine.

The node you build will support two operating modes: CP (consistency-first) and AP (availability-first). You will not test those modes yet. This step focuses on getting the code written and the image built cleanly.

In this step, get ready to:
  • Write the Flask application that acts as a distributed key-value store node.
  • Create the Dockerfile, requirements file, and Docker Compose configuration.
  • Build the Docker image and confirm it compiles without errors.
Create the Flask application

Your node is a Python file called app.py that stores key-value pairs in memory and exposes REST endpoints for reading, writing, and replicating data between peers.

  • In VS Code, right-click the empty Explorer panel on the left and select New File.
  • Name the file app.py and press Enter to create it.
  • Paste the following imports and configuration code into app.py:
import os
import time

from flask import Flask, request, jsonify
import requests as http_requests

app = Flask(__name__)

# In-memory key-value store
store = {}

# Configuration from environment
NODE_NAME = os.environ.get("NODE_NAME", "node1")
PEERS = [p for p in os.environ.get("PEERS", "").split(",") if p]
MODE = os.environ.get("MODE", "CP")

What does this code do?

store is a plain Python dictionary that holds all key-value data in memory. When the container restarts, the data disappears. That is fine for learning purposes.

NODE_NAME, PEERS, and MODE are pulled from environment variables so each container can have its own identity and know where its peers live. Docker Compose will inject these values later.

  • Below the configuration block, add the GET endpoints that let you read data from the node:

@app.route("/data", methods=["GET"])
def get_all_data():
    return jsonify({"node": NODE_NAME, "mode": MODE, "store": store})


@app.route("/data/<key>", methods=["GET"])
def get_data(key):
    if key in store:
        return jsonify({"node": NODE_NAME, "key": key, "value": store[key]})
    return jsonify({"error": "Key not found"}), 404

What do these endpoints do?

/data returns everything the node currently stores, along with its name and mode. /data/<key> returns a single value by key, or a 404 if the key does not exist.

  • Below the GET endpoints, add the PUT endpoint that handles writes in both CP and AP modes:

@app.route("/data/<key>", methods=["PUT"])
def put_data(key):
    global MODE
    value = request.json.get("value")

    if MODE == "CP":
        # CP Mode: Replicate to ALL peers before confirming
        for peer in PEERS:
            try:
                resp = http_requests.post(
                    f"http://{peer}/replicate",
                    json={"key": key, "value": value},
                    timeout=2,
                )
                if resp.status_code != 200:
                    return jsonify({
                        "error": f"Replication to {peer} failed",
                        "reason": "Write rejected to maintain consistency (CP mode)",
                    }), 503
            except http_requests.exceptions.RequestException:
                return jsonify({
                    "error": f"Cannot reach {peer}",
                    "reason": "Write rejected to maintain consistency (CP mode)",
                }), 503

        # All peers confirmed, write locally
        store[key] = value
        return jsonify({
            "status": "ok",
            "node": NODE_NAME,
            "key": key,
            "value": value,
            "mode": "CP",
            "message": "All nodes consistent",
        })

    else:
        # AP Mode: Write locally first, replicate best-effort
        store[key] = value

        replication_results = []
        for peer in PEERS:
            try:
                http_requests.post(
                    f"http://{peer}/replicate",
                    json={"key": key, "value": value},
                    timeout=1,
                )
                replication_results.append({"peer": peer, "status": "replicated"})
            except http_requests.exceptions.RequestException:
                replication_results.append({"peer": peer, "status": "unreachable"})

        return jsonify({
            "status": "ok",
            "node": NODE_NAME,
            "key": key,
            "value": value,
            "mode": "AP",
            "replication": replication_results,
        })

What does this code do?

This is the heart of the project. In CP mode, the node contacts every peer and waits for confirmation before saving locally. If any peer is unreachable, the write is rejected with a 503 error. Consistency is protected, but availability is sacrificed.

In AP mode, the node saves locally first, then tries to replicate. If a peer is unreachable, the write still succeeds. Availability is preserved, but the nodes can end up with different data.

  • Finally, add the replication endpoint, mode-switching endpoints, and the main block at the bottom of app.py:

@app.route("/replicate", methods=["POST"])
def replicate():
    key = request.json.get("key")
    value = request.json.get("value")
    store[key] = value
    return jsonify({"status": "ok", "node": NODE_NAME})


@app.route("/mode", methods=["GET"])
def get_mode():
    return jsonify({"node": NODE_NAME, "mode": MODE})


@app.route("/mode", methods=["POST"])
def set_mode():
    global MODE
    new_mode = request.json.get("mode", "").upper()
    if new_mode in ("CP", "AP"):
        MODE = new_mode
        return jsonify({"status": "ok", "node": NODE_NAME, "mode": MODE})
    return jsonify({"error": "Mode must be CP or AP"}), 400


if __name__ == "__main__":
    port = int(os.environ.get("PORT", 5000))
    app.run(host="0.0.0.0", port=port)

What does this code do?

/replicate is the endpoint that peers call to push data to this node. It blindly accepts whatever it receives and saves it to the local store.

/mode lets you switch a running node between CP and AP mode on the fly using a POST request. The main block starts the Flask server on port 5000 inside the container.

  • Save app.py by pressing Cmd+S (macOS) or Ctrl+S (Windows).

✔️ Awesome, I've got everything!

Great. Double check you have saved app.py and move on to creating the supporting files.

ⓧ I'd like to double check the full code

import os
import time

from flask import Flask, request, jsonify
import requests as http_requests

app = Flask(__name__)

# In-memory key-value store
store = {}

# Configuration from environment
NODE_NAME = os.environ.get("NODE_NAME", "node1")
PEERS = [p for p in os.environ.get("PEERS", "").split(",") if p]
MODE = os.environ.get("MODE", "CP")


@app.route("/data", methods=["GET"])
def get_all_data():
    return jsonify({"node": NODE_NAME, "mode": MODE, "store": store})


@app.route("/data/<key>", methods=["GET"])
def get_data(key):
    if key in store:
        return jsonify({"node": NODE_NAME, "key": key, "value": store[key]})
    return jsonify({"error": "Key not found"}), 404


@app.route("/data/<key>", methods=["PUT"])
def put_data(key):
    global MODE
    value = request.json.get("value")

    if MODE == "CP":
        # CP Mode: Replicate to ALL peers before confirming
        for peer in PEERS:
            try:
                resp = http_requests.post(
                    f"http://{peer}/replicate",
                    json={"key": key, "value": value},
                    timeout=2,
                )
                if resp.status_code != 200:
                    return jsonify({
                        "error": f"Replication to {peer} failed",
                        "reason": "Write rejected to maintain consistency (CP mode)",
                    }), 503
            except http_requests.exceptions.RequestException:
                return jsonify({
                    "error": f"Cannot reach {peer}",
                    "reason": "Write rejected to maintain consistency (CP mode)",
                }), 503

        # All peers confirmed, write locally
        store[key] = value
        return jsonify({
            "status": "ok",
            "node": NODE_NAME,
            "key": key,
            "value": value,
            "mode": "CP",
            "message": "All nodes consistent",
        })

    else:
        # AP Mode: Write locally first, replicate best-effort
        store[key] = value

        replication_results = []
        for peer in PEERS:
            try:
                http_requests.post(
                    f"http://{peer}/replicate",
                    json={"key": key, "value": value},
                    timeout=1,
                )
                replication_results.append({"peer": peer, "status": "replicated"})
            except http_requests.exceptions.RequestException:
                replication_results.append({"peer": peer, "status": "unreachable"})

        return jsonify({
            "status": "ok",
            "node": NODE_NAME,
            "key": key,
            "value": value,
            "mode": "AP",
            "replication": replication_results,
        })


@app.route("/replicate", methods=["POST"])
def replicate():
    key = request.json.get("key")
    value = request.json.get("value")
    store[key] = value
    return jsonify({"status": "ok", "node": NODE_NAME})


@app.route("/mode", methods=["GET"])
def get_mode():
    return jsonify({"node": NODE_NAME, "mode": MODE})


@app.route("/mode", methods=["POST"])
def set_mode():
    global MODE
    new_mode = request.json.get("mode", "").upper()
    if new_mode in ("CP", "AP"):
        MODE = new_mode
        return jsonify({"status": "ok", "node": NODE_NAME, "mode": MODE})
    return jsonify({"error": "Mode must be CP or AP"}), 400


if __name__ == "__main__":
    port = int(os.environ.get("PORT", 5000))
    app.run(host="0.0.0.0", port=port)
Create the supporting files

Your node needs three more files: a dependency list, a container recipe, and an orchestration config that spins up three copies on one network.

  • Create a new file called requirements.txt in the same folder as app.py.
  • Paste the following pinned dependencies:
flask==3.1.3
requests==2.34.2
  • Save requirements.txt.
  • Create another new file called Dockerfile (no extension) in the same folder.
  • Paste the following container definition:
FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

CMD ["python", "app.py"]

What does the Dockerfile do?

It starts from the official python:3.12-slim image, copies your dependency list, installs Flask and requests, then copies your application code. The CMD line tells Docker what command to run when the container starts.

  • Save Dockerfile.

File has a .txt extension?

Some editors automatically add .txt to new files. Make sure your file is named exactly Dockerfile with no extension. In VS Code, check the file name in the Explorer panel on the left.

Help me rename my Dockerfile

✔️ Awesome, I've got everything!

Great. You should now have requirements.txt and Dockerfile saved alongside app.py.

ⓧ I'd like to double check the full code

Here is requirements.txt:

flask==3.1.3
requests==2.34.2

Here is Dockerfile:

FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

CMD ["python", "app.py"]
  • Create one more file called docker-compose.yml in the same folder.
  • Paste the following Docker Compose configuration that defines three identical nodes on a shared network:
services:
  node1:
    build: .
    container_name: node1
    ports:
      - "5001:5000"
    environment:
      - NODE_NAME=node1
      - PEERS=node2:5000,node3:5000
      - MODE=CP
    networks:
      - capnet

  node2:
    build: .
    container_name: node2
    ports:
      - "5002:5000"
    environment:
      - NODE_NAME=node2
      - PEERS=node1:5000,node3:5000
      - MODE=CP
    networks:
      - capnet

  node3:
    build: .
    container_name: node3
    ports:
      - "5003:5000"
    environment:
      - NODE_NAME=node3
      - PEERS=node1:5000,node2:5000
      - MODE=CP
    networks:
      - capnet

networks:
  capnet:
    name: capnet
    driver: bridge

What does this configuration do?

Each service (node1, node2, node3) builds from the same Dockerfile but gets its own NODE_NAME and list of PEERS. Ports 5001, 5002, and 5003 on your machine map to port 5000 inside each container.

The capnet network gives all three nodes a private network where they can reach each other by container name. Using name: capnet gives the network a clean, predictable name you can reference in later commands.

  • Save docker-compose.yml.
  • Confirm your project folder contains exactly 4 files by running this in the VS Code terminal (Ctrl+` to open it):
ls

You should see app.py, docker-compose.yml, Dockerfile, and requirements.txt listed.

Missing a file?

Make sure you created each file in the same folder. If ls shows fewer than 4 files, check that VS Code's Explorer panel is open to your project folder and re-create the missing file there.

Help me find where my files went

✔️ Awesome, I've got everything!

All 4 files are in place. Time to build the image.

ⓧ I'd like to double check the full code

services:
  node1:
    build: .
    container_name: node1
    ports:
      - "5001:5000"
    environment:
      - NODE_NAME=node1
      - PEERS=node2:5000,node3:5000
      - MODE=CP
    networks:
      - capnet

  node2:
    build: .
    container_name: node2
    ports:
      - "5002:5000"
    environment:
      - NODE_NAME=node2
      - PEERS=node1:5000,node3:5000
      - MODE=CP
    networks:
      - capnet

  node3:
    build: .
    container_name: node3
    ports:
      - "5003:5000"
    environment:
      - NODE_NAME=node3
      - PEERS=node1:5000,node2:5000
      - MODE=CP
    networks:
      - capnet

networks:
  capnet:
    name: capnet
    driver: bridge
Build the Docker image

Before you launch the cluster, confirm everything compiles cleanly by building the image.

  • Build all three service images by running this command in your terminal:
docker compose build
  • Wait for the build to finish. You should see output ending with FINISHED and no red error text.

What should I see?

Docker pulls the python:3.12-slim base image (first time only), installs your pip dependencies, and copies app.py into the image. The final lines will show each build step completing, with a FINISHED status at the top of the output summary.

Seeing a build error?

If you see a "file not found" error, make sure requirements.txt, Dockerfile, and app.py are all in the same folder, and that your terminal is in that folder.

If you see a pip install failure, double check that requirements.txt contains exactly flask==3.1.3 and requests==2.34.2 on separate lines with no typos.

Help me debug my Docker build error

Your image is built and ready. Next up, you will launch all three nodes and watch data replicate across the cluster in real time.

Launch the Cluster and Test Normal Operation

Your Docker image is built and ready. Now it's time to bring your 3-node distributed cluster to life and see it work as a unified system.

In this step, you'll launch all three nodes and prove that data written to one node automatically replicates to the others. This is what a healthy distributed system looks like when there is no network failure.

In this step, get ready to:
  • Start the 3-node cluster and confirm all nodes are reachable.
  • Write data to one node and verify it replicates to all others.
  • Build confidence that the cluster operates as a unified system.
Start the cluster and verify all nodes are running
  • Launch all 3 nodes in the background by running this command:
docker compose up -d
  • Confirm all containers are running by running:
docker compose ps
  • You should see all 3 containers listed with a running status. Each maps to a different port: node1 on 5001, node2 on 5002, node3 on 5003.

Containers not starting?

If you see an error about port conflicts, another process is already using one of the ports (5001, 5002, or 5003). Stop the conflicting process or change the port mapping in docker-compose.yml.

If a container exits immediately, check the logs with docker compose logs node1 to see what went wrong.

help me debug my Docker containers that won't start

  • Query each node's data endpoint to confirm they respond. Run these three commands one at a time:
curl http://localhost:5001/data
curl http://localhost:5002/data
curl http://localhost:5003/data
  • Each node should respond with a JSON object showing its name, mode set to CP, and an empty store.

What does the response tell you?

Each response includes the node's name (node1, node2, or node3), its current mode (CP), and an empty store object. This confirms each container is running its own independent instance of app.py, all connected on the same capnet network.

Connection refused?

If curl returns a connection error, the container might still be starting up. Wait a few seconds and try again.

Make sure Docker Desktop is running and the containers show as active in docker compose ps.

help me troubleshoot why curl cannot connect to my Docker containers

Write data to one node and confirm replication

Your cluster is running. Now you'll write data to a single node and verify it automatically replicates to all other nodes.

  • Write a key-value pair to node1 by running this command:
curl -X PUT http://localhost:5001/data/color -H "Content-Type: application/json" -d '{"value": "blue"}'
  • You should see a JSON response with "status": "ok", "message": "All nodes consistent", and "mode": "CP".

What just happened behind the scenes?

In CP mode, node1 does not save the data locally until it has confirmed that ALL peer nodes accepted it first. It sent the key color with value blue to node2 and node3 via the /replicate endpoint. Both responded with success, so only then did node1 save it locally.

This guarantees all three nodes have the same data at the same time. That is consistency.

  • Read the data back from all 3 nodes to prove replication worked. Run each command one at a time:
curl http://localhost:5001/data/color
curl http://localhost:5002/data/color
curl http://localhost:5003/data/color
  • All three nodes should return "value": "blue". The data is consistent across the entire cluster.

Why does the CAP theorem not apply yet?

Right now, all three properties of the CAP theorem are satisfied: data is consistent (same value on all nodes), the system is available (every request gets a response), and the network is intact (no partition).

The CAP theorem only forces a trade-off when a network partition occurs. In the next steps, you will break the network and see which property gets sacrificed.

Write multiple pairs to build confidence in the cluster

One key-value pair is a good start. Let's write a few more to prove the cluster consistently replicates across all nodes.

  • Write two more key-value pairs to node1 by running these commands:
curl -X PUT http://localhost:5001/data/city -H "Content-Type: application/json" -d '{"value": "Tokyo"}'
curl -X PUT http://localhost:5001/data/language -H "Content-Type: application/json" -d '{"value": "Python"}'
  • Both should return "status": "ok" with "message": "All nodes consistent".
  • Now read the full data store from node2 and node3 to confirm everything replicated:
curl http://localhost:5002/data
curl http://localhost:5003/data
  • Both nodes should return a store object containing all three keys: color, city, and language with the values blue, Tokyo, and Python.

What does this prove?

You wrote all data to node1, but node2 and node3 have it too. The cluster acts as a unified system. No matter which node you query, you get the same answer.

This is the ideal state of a distributed system. In the real world, systems like PostgreSQL replicas and DynamoDB tables work the same way when the network is healthy.

Seeing 'Key not found' on some nodes?

If node2 or node3 returns a 404 for a key you wrote to node1, the replication request may have timed out. Check node1's write response for any error messages.

Try writing the key again. If the problem persists, restart the cluster with docker compose down followed by docker compose up -d.

help me debug why data is not replicating between my Docker nodes

Your cluster is healthy and replicating perfectly. In the next step, you will break the network between nodes and watch the CAP theorem force a painful choice.

Simulate a Network Partition in CP Mode

Your cluster is humming along perfectly. Every write replicates to all three nodes, and every read returns the same data no matter which node you ask.

Now it's time to break things. In a real distributed system, network failures happen constantly. Cables get cut, data centers lose connectivity, cloud regions go down. The CAP theorem says that when a network partition occurs, a system must choose between consistency (all nodes agree) and availability (all nodes accept requests). Your cluster is in CP mode, so let's see which one it sacrifices.

In this step, get ready to:
  • Disconnect a node from the network to simulate a partition.
  • Observe CP mode rejecting a write to protect consistency.
  • Reconnect the node and confirm no data was corrupted.
Disconnect node3 from the network

A network partition is when some nodes in a cluster can no longer communicate with each other. The system splits into isolated groups that cannot coordinate.

  • Simulate a partition by disconnecting node3 from the capnet network by running this command:
docker network disconnect capnet node3

What does this command do?

This physically removes node3 from the Docker bridge network. It's not a fake flag or a software toggle. The container genuinely loses its network interface on capnet.

From node1 and node2's perspective, node3 has vanished. Any HTTP request to node3:5000 will time out. This is exactly what happens during a real network partition between data centers.

  • Confirm node3 is unreachable by trying to read from it:
curl http://localhost:5003/data

You should see a connection refused or timeout error. The node's container is still running, but it has no network path to the outside world.

Still getting a response from node3?

The disconnect command produces no output on success. If you still get a JSON response from node3, double check you typed capnet (not a different network name) and node3 (not node-3 or another variant).

help me troubleshoot why node3 is still responding after docker network disconnect

Attempt a write and observe CP mode reject it

Your cluster is now partitioned. Node1 and node2 can still talk to each other, but neither can reach node3. In CP mode, a write must replicate to ALL peers before it's confirmed. Let's see what happens when that's impossible.

  • Attempt to write a new key-value pair to node1 by running:
curl -X PUT http://localhost:5001/data/city \
  -H "Content-Type: application/json" \
  -d '{"value": "Tokyo"}'

You should see a 503 error response with a message like this:

{
  "error": "Cannot reach node3:5000",
  "reason": "Write rejected to maintain consistency (CP mode)"
}

What just happened?

Node1 tried to replicate your write to both node2 and node3. It reached node2 successfully, but the request to node3 timed out after 2 seconds. Because the system is in CP mode, node1 refused to confirm the write.

This is the CP trade-off in action. Consistency is preserved because no node stores data that other nodes don't have. But availability is sacrificed because the system cannot accept new writes until ALL nodes are reachable again.

  • Verify that node1 did NOT store the rejected write by running:
curl http://localhost:5001/data/city

You should see a 404 "Key not found" response. The key was never committed because replication failed.

Why is this important?

In a CP system, a failed write means ZERO nodes have the data. There's no situation where node1 says one thing and node3 says another. This eliminates the split-brain problem entirely, at the cost of temporarily refusing service.

Real-world CP databases like PostgreSQL (with synchronous replication) and etcd make this same choice. They'd rather say "sorry, try again later" than risk data corruption.

Reconnect node3 and verify consistency

The partition is over. Time to reconnect node3 and confirm that no inconsistent data crept in while it was isolated.

  • Reconnect node3 to the network by running:
docker network connect capnet node3
  • Verify node3 is reachable again by running:
curl http://localhost:5003/data

You should see a JSON response listing node3's store with the same data it had before the partition.

  • Now retry the write that failed earlier:
curl -X PUT http://localhost:5001/data/city \
  -H "Content-Type: application/json" \
  -d '{"value": "Tokyo"}'

This time you should see a success response with "message": "All nodes consistent".

  • Confirm all three nodes have identical data by querying each one:
curl http://localhost:5001/data/city
curl http://localhost:5002/data/city
curl http://localhost:5003/data/city

All three should return {"value": "Tokyo"}. No split-brain, no conflicting data, no inconsistency. That's the guarantee CP mode provides.

Node3 returning different data?

If node3 shows a different store than node1 and node2, it may not have fully reconnected. Try waiting a few seconds and querying again. The docker network connect command restores the network interface, but DNS resolution may take a moment.

help me troubleshoot why node3 has different data after reconnecting

What did CP mode actually protect?

During the entire partition, zero writes succeeded. That means every node's data store remained identical throughout. When the partition healed, there was nothing to reconcile. No conflicts, no merge logic, no data loss.

The cost was clear: for about as long as the partition lasted, the system was unable to accept new data. In the next step, you'll see what happens when a system chooses the opposite trade-off.

You've just witnessed the CP side of the CAP theorem firsthand. Your system chose to refuse writes rather than risk inconsistency. Next up, you'll flip to AP mode and see what happens when the system chooses availability instead.

Switch to AP Mode and Observe Split-Brain

In the previous step, you proved that CP mode protects data accuracy by rejecting writes during a partition. But what if your application needs to stay available no matter what?

Now you will flip all three nodes into AP mode and run the same partition experiment. This time writes will succeed even when a node is unreachable. The price you pay is called a split-brain: different nodes end up with different data for the same key.

In this step, get ready to:
  • Switch all nodes from CP to AP mode at runtime.
  • Write data during a partition and see it succeed (but fail to replicate).
  • Create a true split-brain by writing conflicting values to partitioned nodes.
Switch all nodes to AP mode and partition node3

AP stands for Availability + Partition tolerance. In this mode, the system accepts every write regardless of whether it can reach all peers.

  • Switch node1 to AP mode by running this command:
curl -X POST http://localhost:5001/mode -H "Content-Type: application/json" -d '{"mode": "AP"}'
  • Switch node2 to AP mode by running:
curl -X POST http://localhost:5002/mode -H "Content-Type: application/json" -d '{"mode": "AP"}'
  • Switch node3 to AP mode by running:
curl -X POST http://localhost:5003/mode -H "Content-Type: application/json" -d '{"mode": "AP"}'
  • You should see a response from each node confirming "mode": "AP".

Why can you change modes at runtime?

The /mode endpoint updates the global MODE variable in each node's running process. The docker-compose.yml still says MODE=CP, but the runtime value now overrides it until the containers restart.

  • Now disconnect node3 from the network to create a partition:
docker network disconnect capnet node3
  • Node3 is now isolated from node1 and node2, just like in the CP experiment.

Seeing an error on disconnect?

If you see an error saying the container is not connected to the network, node3 may still be disconnected from the previous step. Run docker network connect capnet node3 first, then disconnect it again.

Help me reconnect my containers to the Docker network.

Write data during the partition and observe missing replication

In CP mode, this same scenario rejected the write entirely. AP mode takes the opposite approach: accept the write, replicate where possible, and hope for the best.

  • Write a new key-value pair to node1 while node3 is partitioned:
curl -X PUT http://localhost:5001/data/city -H "Content-Type: application/json" -d '{"value": "Sydney"}'

What should I see?

The response shows "status": "ok" with "mode": "AP". The write succeeded. Look at the replication array in the response. Node2 shows "status": "replicated" but node3 shows "status": "unreachable". The system accepted the write anyway.

  • Reconnect node3 and query all three nodes to compare their data:
docker network connect capnet node3
  • Read the city key from node1:
curl http://localhost:5001/data/city
  • Read the same key from node3:
curl http://localhost:5003/data/city

Node1 returns "value": "Sydney" but node3 returns a 404 error. The data never made it to node3 during the partition.

What is the split-brain problem?

A split-brain occurs when nodes in a distributed system hold different versions of the truth. In CP mode, the system prevents this by refusing writes. In AP mode, the system allows the divergence to happen.

This is the fundamental trade-off of the CAP Theorem: you cannot avoid it, you can only choose which side to land on.

Create a true split-brain with conflicting writes

Missing data on one node is bad, but the real nightmare is two nodes having different values for the same key. This is exactly what happens in production AP databases like Cassandra and DynamoDB when partitions occur.

  • Disconnect node3 from the network again:
docker network disconnect capnet node3
  • Write a value for the key leader to node1:
curl -X PUT http://localhost:5001/data/leader -H "Content-Type: application/json" -d '{"value": "node1-was-here"}'

The write succeeds on node1 and replicates to node2, but cannot reach node3.

  • Reconnect node3 and disconnect node1 to flip the partition:
docker network connect capnet node3
docker network disconnect capnet node1
  • Now write a DIFFERENT value for the same key to node3:
curl -X PUT http://localhost:5003/data/leader -H "Content-Type: application/json" -d '{"value": "node3-was-here"}'

This write also succeeds. Node3 writes locally and replicates to node2, overwriting node1's value on node2.

  • Reconnect node1 so all nodes are back on the network:
docker network connect capnet node1
  • Query node1 for the leader key:
curl http://localhost:5001/data/leader
  • Query node3 for the same key:
curl http://localhost:5003/data/leader

Node1 returns "value": "node1-was-here" and node3 returns "value": "node3-was-here". Two nodes, same key, different values. This is a confirmed split-brain.

Why does this matter in the real world?

Imagine the key is a bank account balance and the two values are different dollar amounts. Neither node is wrong. Both accepted valid writes from legitimate clients. The system just has no way to know which value is correct.

Real AP databases like DynamoDB and Cassandra solve this with conflict resolution strategies (timestamps, vector clocks, application-level merges). Without one of these, the split-brain stays forever.

Both nodes returning the same value?

If both nodes show the same value for leader, the partition may not have been active when you wrote. Make sure you disconnect node3 BEFORE writing to node1, and disconnect node1 BEFORE writing to node3. The order matters.

Help me debug why my split-brain experiment is not showing different values.

You now have direct evidence of why every distributed database must pick a side. PostgreSQL and etcd choose CP (reject writes, stay consistent). Cassandra and DynamoDB choose AP (accept writes, deal with conflicts later). Next up, you will build the conflict resolution that AP systems need to heal a split-brain.

Secret mission

Add Conflict Resolution When Partitions Heal

You've seen the split-brain problem firsthand. Nodes hold different values for the same key with no way to reconcile. In this mission, you'll implement timestamp-based conflict resolution so nodes can automatically merge their diverged data when the partition heals.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally using Docker, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Docker containers: node1, node2, node3.
  • Docker network: capnet.
  • Docker images built from the project Dockerfile.
  • Local project files: app.py, requirements.txt, Dockerfile, docker-compose.yml.

Keep everything running

No action needed. Choose this if you're still actively building or experimenting with network partitions and replication modes.

  • Your 3-node cluster continues running on ports 5001, 5002, and 5003.
  • Docker Desktop will consume some memory while the containers are active.
  • You can stop and restart the cluster anytime with docker compose up -d from your project folder.

Pause - I'll come back to this later

Shut down the running containers to free up memory. Your project files stay on disk so you can restart later.

  • Open your terminal and navigate to your project folder (the one containing docker-compose.yml).
  • Stop the containers by running:
docker compose down
  • This stops and removes the containers and the capnet network, but keeps your images and source files intact.
  • To restart later, run docker compose up -d from the same folder.

Delete - I don't want to use this again

Remove all Docker resources and project files to start completely fresh.

  • Open your terminal and navigate to your project folder (the one containing docker-compose.yml).
  • Stop the containers, remove the network, and delete the built images by running:
docker compose down --rmi all
  • Navigate up one level from your project folder by running:
cd ..
  • Delete the project folder and all its contents.

macOS

  • Remove the project folder by running:
rm -rf cap-distributed-system

Windows

  • Remove the project folder by running:
rmdir /s /q cap-distributed-system

Folder name might be different

Replace cap-distributed-system with whatever you named your project folder when you created it in Step 1.

  • Confirm everything is removed by running ls (macOS) or dir (Windows). Your project folder should no longer appear.

Nice Work!

Nice Work!

You just built a distributed system, broke it on purpose, and watched it fail in two fundamentally different ways. That hands-on experience is something most engineers only read about in textbooks.

You've learned how to:

  • Build a 3-node distributed key-value store from scratch using Python, Flask, and Docker.
  • Simulate network partitions using Docker network commands and observe the CAP theorem force a trade-off between consistency and availability.
  • Demonstrate both CP mode (writes rejected during partition to protect data accuracy) and AP mode (writes accepted during partition, causing split-brain divergence).
  • Secret Mission: Implement last-write-wins conflict resolution to automatically merge diverged data when a partition heals.

Ready to quiz yourself?