Explore Distributed Databases

Build a local multi-node CockroachDB cluster and prove fault tolerance.

Introduction

30 Second Summary

Every time you check your bank balance, stream a show, or send a message, the app behind it needs to stay online no matter what. Traditional databases run on a single server. If that server goes down, everything stops. Distributed databases solve this by spreading your data across multiple machines that work together as one system.

In this project, you will build a multi-node database cluster on your own machine, watch it replicate data automatically, scale it by adding new nodes, and then deliberately break it to prove it keeps running.

What You'll Build

A local multi-node CockroachDB cluster that demonstrates automatic replication, horizontal scaling with rebalancing, and fault tolerance under node failure.

By the end of this project, you'll have:

  • A 3-node CockroachDB cluster running locally with data replicated across all nodes.
  • Evidence of automatic rebalancing in the DB Console after scaling the cluster to 5 nodes.
  • Proof of fault tolerance by killing a node and watching the cluster continue serving queries without interruption.
  • Secret Mission: Increase replication to 5x and prove your cluster can survive two simultaneous node failures.

Are there any prerequisites?

Basic familiarity with SQL is helpful (SELECT, INSERT, CREATE TABLE). No distributed systems experience is required.

Install CockroachDB

Before you can run a distributed database cluster, you need the database software itself installed on your machine.

In this step, you will install CockroachDB using Homebrew and verify it works by launching a temporary demo cluster.

In this step, get ready to:
  • Install CockroachDB via the official Homebrew tap.
  • Verify the installation by running a temporary in-memory cluster.
Install CockroachDB via Homebrew

CockroachDB provides an official Homebrew tap that installs the correct binary for your Mac's architecture (Intel or Apple Silicon).

  • Check that Homebrew is installed by running this command:
brew --version

✔️ I see a version number

Homebrew is ready. Move on to the install command below.

ⓧ Command not found

Homebrew is not installed yet. Install it by running this command:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Follow the prompts in your terminal to complete the installation. Once it finishes, close and reopen your terminal, then run brew --version again to confirm it works.

  • Install CockroachDB by running this command:
brew install cockroachdb/tap/cockroach

What does this command do?

This command tells Homebrew to add the official CockroachDB tap (a third-party repository of formulas) and install the cockroach binary from it. Once installed, the cockroach command becomes available in your terminal.

  • The install may take a minute or two. Once it finishes, confirm the binary is available by running:
cockroach version

You should see output showing the CockroachDB version number, build information, and platform details.

Seeing command not found?

Make sure the Homebrew install completed without errors. Scroll up in your terminal and look for any red error text during the install.

Try closing and reopening your terminal, then run cockroach version again. Some shell environments need a fresh session to pick up new binaries.

Help me troubleshoot why cockroach is not found after installing via Homebrew.

Verify the installation

The quickest way to confirm CockroachDB works is to start a temporary in-memory cluster. This spins up a single-node cluster that disappears the moment you close it.

  • Start a temporary demo cluster by running:
cockroach demo

What does this command do?

The cockroach demo command starts a temporary, in-memory CockroachDB cluster and opens an interactive SQL shell. It preloads a sample dataset so you can experiment immediately.

Nothing is saved to disk. The cluster only exists while the shell is open. Exiting the shell destroys all data.

You should see a welcome message with cluster details, followed by an interactive SQL prompt.

  • Type a quick SQL query to confirm the shell is working:
SELECT 1;

You should see a result table with the value 1 returned.

  • Exit the demo shell by typing:
\q

Why is CockroachDB experimental on macOS?

CockroachDB on macOS is experimental and not suitable for production deployments. However, it works perfectly for local development and learning.

For this project, the macOS build gives us everything we need to explore distributed database concepts hands-on.

Demo cluster not starting?

If you see an error about ports being in use, another process may be listening on port 26257. Close any other database processes and try again.

If you see a permissions error, make sure your user has write access to the temporary directory your system uses.

Help me troubleshoot why cockroach demo is failing to start on my Mac.

CockroachDB is installed and ready. Next up, you will start a real multi-node cluster and see distributed replication in action.

Start a 3-Node Cluster

A distributed database spreads data across multiple machines called nodes. Each node holds a copy of the data, so if one goes down, the others keep serving requests without interruption.

In this step, you'll launch three CockroachDB nodes on your local machine, each running on a different port to simulate separate servers. Once all three are up and connected, you'll initialize the cluster and see them come to life in a visual dashboard.

In this step, get ready to:
  • Start three CockroachDB nodes in separate terminal windows.
  • Initialize the cluster with a one-time command.
  • Explore the DB Console to see your live nodes.
Start three nodes in separate terminals

Each CockroachDB node runs as a foreground process that occupies a terminal window. You'll need three terminals for the three nodes, plus one more terminal for running commands against the cluster.

  • Return to the terminal you used in Step 1.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create a project folder and move into it by running these commands:
mkdir cockroachdb-cluster
cd cockroachdb-cluster
  • Confirm the folder exists by running:
ls

You should see the cockroachdb-cluster folder listed in the output (it will be empty for now).

  • Start node 1 by running this command:
cockroach start --insecure --store=node1 --listen-addr=localhost:26257 --http-addr=localhost:8080 --join=localhost:26257,localhost:26258,localhost:26259

You'll see a warning message about insecure mode, followed by a line that says the node will wait for cockroach init. The terminal will NOT return to a prompt. This is expected.

What do these flags mean?

  • --insecure disables TLS encryption, which simplifies local development.
  • --store=node1 tells CockroachDB to save this node's data in a node1/ folder.
  • --listen-addr=localhost:26257 sets the SQL and RPC port for this node.
  • --http-addr=localhost:8080 sets the port for the web-based DB Console.
  • --join lists all the nodes in the cluster so they know how to find each other.

Seeing an error about the port already in use?

If you see a message about port 26257 or 8080 being in use, another process is already occupying that port. Close any other CockroachDB instances or services using those ports and try again.

If you previously ran this step and the node1/ directory already exists, delete it first with rm -rf node1 and re-run the start command.

Help me troubleshoot my cockroach start command failing on node 1.

  • Open a new Terminal window by pressing Cmd+N (macOS) or opening a new terminal instance (Windows/Linux).
  • Navigate to your project folder by running:
cd ~/Desktop/cockroachdb-cluster
  • Start node 2 by running this command:
cockroach start --insecure --store=node2 --listen-addr=localhost:26258 --http-addr=localhost:8081 --join=localhost:26257,localhost:26258,localhost:26259

You'll see the same insecure-mode warning. The terminal stays occupied, just like node 1.

  • Open a third Terminal window by pressing Cmd+N (macOS) or opening another terminal instance (Windows/Linux).
  • Navigate to your project folder by running:
cd ~/Desktop/cockroachdb-cluster
  • Start node 3 by running this command:
cockroach start --insecure --store=node3 --listen-addr=localhost:26259 --http-addr=localhost:8082 --join=localhost:26257,localhost:26258,localhost:26259

You now have three nodes running, each in its own terminal window. All three are waiting to be initialized into a cluster.

Initialize the cluster

The nodes are running, but they don't form a cluster yet. CockroachDB requires a one-time initialization command that tells the nodes to connect and start replicating data between each other.

  • Open a fourth Terminal window by pressing Cmd+N (macOS) or opening another terminal instance (Windows/Linux).
  • Navigate to your project folder by running:
cd ~/Desktop/cockroachdb-cluster
  • Initialize the cluster by running this command:
cockroach init --insecure --host=localhost:26257

You should see the message:

Cluster successfully initialized

What just happened?

The cockroach init command performed a one-time handshake that connected all three nodes into a single cluster. Each node now knows about the others and will automatically replicate data between them using the Raft consensus protocol.

If you look at the other three terminal windows, you'll see that each node has printed startup details confirming it joined the cluster.

Not seeing the success message?

Make sure all three nodes are still running in their respective terminal windows. If any of them stopped, go back to that terminal and re-run its cockroach start command.

Double check that you're running cockroach init from the fourth terminal (not one of the terminals occupied by a running node).

Help me troubleshoot my cockroach init command not working.

Explore the DB Console

CockroachDB includes a built-in web dashboard called the DB Console. It shows you a live view of your cluster's health, node status, and replica distribution.

  • Open your web browser.
  • Navigate to http://localhost:8080 in the address bar.
  • On the Cluster Overview page, look at the Node List section.

You should see three live nodes listed, each showing a healthy status with roughly equal replica counts.

What are replicas?

CockroachDB stores data in chunks called ranges. Each range is copied (replicated) across multiple nodes. The default replication factor is 3, meaning every piece of data exists on all three of your nodes.

Equal replica counts across nodes means your data is evenly distributed. This is the foundation of fault tolerance. If one node goes down, the other two still have complete copies.

Your 3-node cluster is alive and replicating data automatically. Next up, you'll write data to the cluster and prove that every node can serve it.

Write Data and Verify Replication

Your 3-node cluster is running, but it has no user data yet. The real power of a distributed database reveals itself when you write data to one node and it automatically appears on every other node.

In this step, you'll create a database, insert sample data through node1, then connect to node2 and prove the same data is already there. No manual copying or syncing required.

In this step, get ready to:
  • Create a database and table with sample data on node1.
  • Read the same data from node2 to prove automatic replication.
  • Confirm replica distribution in the DB Console.
Create a database and table

Before you can prove replication works, you need data in the cluster. You'll connect to node1's SQL shell and create a simple banking database with account records.

  • Open a new terminal tab and connect to node1's SQL shell by running this command:
cockroach sql --insecure --host=localhost:26257

You should see a welcome message that includes "Welcome to the CockroachDB SQL shell" followed by a > prompt.

Not seeing the SQL prompt?

Make sure your cluster nodes are still running in their terminal tabs. If a node stopped, restart it with the same cockroach start command from the previous step.

Also confirm you typed --host=localhost:26257 exactly. A typo in the port number will cause a connection refused error.

Help me troubleshoot my CockroachDB SQL shell connection

  • Create the bank database, an accounts table, and insert sample data by running these SQL commands:
CREATE DATABASE bank;

CREATE TABLE bank.accounts (id INT PRIMARY KEY, name STRING, balance DECIMAL);

INSERT INTO bank.accounts VALUES (1, 'Alice', 1000.50);
INSERT INTO bank.accounts VALUES (2, 'Bob', 250.00);
INSERT INTO bank.accounts VALUES (3, 'Carol', 5000.75);

SELECT * FROM bank.accounts;

What do these commands do?

  • CREATE DATABASE bank; creates a new database called bank on your cluster.
  • CREATE TABLE bank.accounts (...) defines a table with three columns: an integer ID, a name, and a decimal balance.
  • Each INSERT adds one account record. CockroachDB automatically replicates each row across all three nodes.
  • SELECT * FROM bank.accounts; reads back all the data to confirm it was inserted successfully.

You should see a table with 3 rows showing Alice, Bob, and Carol with their balances.

Seeing a syntax error?

Make sure every SQL statement ends with a semicolon. The SQL shell waits for more input until it sees one. If you're stuck at a continuation prompt (->), type a semicolon and press Enter to execute.

Help me fix my SQL syntax error in CockroachDB

Read from a different node

The data exists on node1. Now for the key question: can node2 read it without you doing anything? In a traditional single-server database, you would need to manually set up replication. In CockroachDB, it happens automatically.

  • Exit the current SQL shell by typing:
\q
  • Connect to node2 by running this command:
cockroach sql --insecure --host=localhost:26258

You should see the same welcome message. You are now connected to a completely different node in your cluster.

  • Run a SELECT query to check if the data is already here:
SELECT * FROM bank.accounts;

You should see the exact same 3 rows (Alice, Bob, Carol) that you inserted through node1.

Why does this work?

CockroachDB replicates each range of data 3 times by default and stores each replica on a different node. When you inserted those rows through node1, the cluster automatically created copies on node2 and node3. Every node can serve reads for any data in the cluster.

This is the Raft consensus protocol in action. All three replicas agree on the data before the write is considered complete.

Not seeing the bank database?

Double check that you connected to port 26258 (node2) and not another port. If you see "relation does not exist", make sure you typed bank.accounts with the database prefix.

Help me read data from a different CockroachDB node

Check replica distribution in the DB Console

You proved replication works through SQL queries. Now visualize it in the DB Console to see exactly how data is spread across your cluster.

  • Switch to your web browser with the DB Console open at http://localhost:8080.
  • Look at the **Overview** page (the default landing page).
  • Observe the replica counts next to each node.

You should see roughly equal replica counts across all three nodes. This confirms the default replication factor of 3 is working.

What is the replication factor?

The replication factor determines how many copies of your data exist across the cluster. With a default replication factor of 3, every piece of data has exactly 3 copies, one on each node. This means the cluster can survive 1 node failing without losing any data.

You will test this fault tolerance yourself in a later step.

Your data is replicated and readable from any node in the cluster. Next up, you'll scale the cluster to 5 nodes and watch CockroachDB automatically rebalance data to use the new capacity.

Scale the Cluster and Watch Rebalancing

You proved that CockroachDB replicates data automatically across all three nodes. Any node can serve reads for the same data without you lifting a finger.

But replication alone is only part of the story. What happens when your database needs more capacity? Traditional databases force you to buy a bigger machine. Distributed databases scale horizontally by adding more nodes. In this step, you will add two new nodes and watch CockroachDB automatically rebalance data to use the extra capacity.

In this step, get ready to:
  • Add two new nodes to your running cluster.
  • Observe automatic rebalancing in the DB Console as replicas spread to the new nodes.
Add two more nodes to your cluster

Adding nodes to a CockroachDB cluster is the same cockroach start command you used before, just with different port numbers and store directories. The --join flag tells each new node where to find the existing cluster.

  • Open a new terminal tab by pressing Cmd+T (macOS) or Ctrl+Shift+T (Windows/Linux).
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Start node4 by running this command:
cockroach start --insecure --store=node4 --listen-addr=localhost:26260 --http-addr=localhost:8083 --join=localhost:26257,localhost:26258,localhost:26259
  • You should see an insecure mode warning followed by a message confirming the node is attempting to join the cluster.

What does this command do?

This is the same pattern you used for nodes 1 through 3, with two differences:

  • --store=node4 creates a new data directory called node4/ on your Desktop.
  • --listen-addr=localhost:26260 and --http-addr=localhost:8083 give this node unique ports so it does not conflict with the other three.
  • The --join flag still points to the original three nodes so the new node discovers and joins the existing cluster.
  • Open another new terminal tab by pressing Cmd+T (macOS) or Ctrl+Shift+T (Windows/Linux).
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Start node5 by running this command:
cockroach start --insecure --store=node5 --listen-addr=localhost:26261 --http-addr=localhost:8084 --join=localhost:26257,localhost:26258,localhost:26259
  • You should see the same insecure mode warning and startup messages confirming node5 is joining the cluster.
  • Confirm both new node directories exist by running this in either node4 or node5's terminal:
ls ~/Desktop | grep node
  • You should see node1 through node5 listed in the output.

Node not starting?

  • If you see a port conflict error, double-check that you used port 26260 for node4 and 26261 for node5. Each node must have unique --listen-addr and --http-addr values.
  • Make sure you are in the same directory (~/Desktop) where your other node directories live. The --store flag creates a relative path from your current location.
  • If the node hangs without printing anything, verify your original 3-node cluster is still running in its terminal tabs.

Still stuck? Help me troubleshoot why my new CockroachDB node won't start or join the cluster.

Watch automatic rebalancing in the DB Console

Your cluster now has 5 nodes, but all the data still lives on the original 3. CockroachDB detects the new capacity and begins moving replicas to even out the load. You can watch this happen in real time.

  • Switch back to your browser with the DB Console at http://localhost:8080.
  • Click Cluster Overview in the left sidebar (or refresh the page if you are already there).
  • Confirm you see 5 nodes listed as Live.
  • Look at the Replicas column for each node.

At first, the replica count for node4 and node5 will be lower than the others. Within a minute or two, those numbers will even out across all five nodes as data rebalances automatically.

What is automatic rebalancing?

CockroachDB stores data in chunks called ranges. Each range is replicated (by default, 3 copies). When new nodes appear, the cluster detects uneven distribution and starts splitting ranges and moving replicas to the new nodes.

No manual sharding. No data migration scripts. The cluster handles it as a background process while continuing to serve reads and writes.

Why does this matter?

In production, this means you can handle traffic spikes by adding machines. The database redistributes itself without downtime or manual intervention. This is horizontal scaling in action.

Your cluster just scaled from 3 to 5 nodes without a single line of migration code. Next up, you will put the cluster's fault tolerance to the test by killing a node mid-flight.

Simulate Node Failure and Prove Fault Tolerance

Your 5-node cluster is humming along with data evenly distributed. Now comes the real test: what happens when a node crashes?

In production, servers fail all the time. Hardware dies, networks drop, processes crash. A distributed database earns its keep by continuing to serve traffic even when parts of the system go down. In this step, you will kill a node, prove the cluster keeps working, and then watch it heal itself.

In this step, get ready to:
  • Kill a node to simulate a server crash.
  • Prove the cluster continues serving reads and writes without interruption.
  • Restart the failed node and watch it rejoin the cluster.
Kill a node

To simulate a real-world failure, you will gracefully shut down node3 using a terminal signal. First you need to find node3's process ID (PID), then send it a termination signal.

  • Find the process IDs of all running CockroachDB nodes by running this command in any available terminal:
ps -ef | grep cockroach | grep -v grep

What does this output show?

Each line represents one running CockroachDB process. Look for the line containing --store=node3 to identify node3's process. The number in the second column is the PID you need.

  • Identify the PID for the process with --store=node3 in the command.
  • Gracefully shut down node3 by running this command, replacing the PID with the one you found:
kill -TERM [[NODE3_PID="PID of the node3 process"]]

Why kill -TERM instead of kill -9?

The -TERM signal tells the process to shut down gracefully. It drains active connections and transfers range leases to other nodes before stopping. This simulates a realistic server failure rather than an instant crash.

  • Switch to your browser and return to the DB Console at http://localhost:8080.
  • On the Cluster Overview page, you should see node3's status change to SUSPECT. After 5 minutes it transitions to DEAD.

Don't see node3 marked as suspect?

It can take a few seconds for the DB Console to detect the failure. Refresh the page. If node3 still shows as LIVE, double check that you sent the kill signal to the correct PID.

help me figure out why node3 still shows as live after I ran kill -TERM

Prove the cluster still works

Node3 is down. If this were a traditional single-server database, your application would be offline. Let's prove that a distributed database handles this without breaking a sweat.

  • Connect to node1 (which is still running) by opening a new terminal and running:
cockroach sql --insecure --host=localhost:26257
  • Insert a new row into the bank.accounts table to prove writes still work:
INSERT INTO bank.accounts VALUES (2, 500.00);
  • Read all rows to confirm both the original and new data are accessible:
SELECT * FROM bank.accounts;

What does this prove?

You just wrote new data and read it back while a node was dead. The cluster has 5 nodes but only needs a majority of replicas (2 out of 3) to be available for any given range. With node3 down, the remaining 4 nodes still hold enough replicas to maintain quorum and serve all traffic.

This is the Raft consensus protocol in action. As long as a majority of replicas are alive, reads and writes proceed normally.

  • You should see both rows returned with no errors. The cluster served your INSERT and SELECT without interruption.
  • Exit the SQL shell by typing:
\q

Getting a connection error?

Make sure you are connecting to a node that is still running. If you accidentally try localhost:26259 (node3's port), you will get a connection refused error because node3 is down. Use localhost:26257 (node1) or localhost:26258 (node2) instead.

help me connect to my CockroachDB cluster after killing a node

Restart the failed node

In the real world, a failed server eventually comes back online. When it does, a distributed database should automatically bring it back into the fold. Let's restart node3 and watch it rejoin.

  • Switch to the terminal where node3 was originally running (it should now show a shell prompt since the process exited).
  • Restart node3 using the same command you used to start it originally:
cockroach start --insecure --store=node3 --listen-addr=localhost:26259 --http-addr=localhost:8082 --join=localhost:26257,localhost:26258,localhost:26259

Why the same command?

CockroachDB stores all of node3's state in the node3/ directory (specified by --store=node3). When the node restarts, it reads its existing data from that directory, reconnects to the cluster, and picks up where it left off.

  • Return to the DB Console at http://localhost:8080 and refresh the page.
  • On the Cluster Overview page, you should see 5 live nodes again. Node3's status returns to LIVE and replicas rebalance back to it.

What about the data written while node3 was down?

The row you inserted (account 2) was replicated to the surviving nodes while node3 was offline. Once node3 rejoins, the cluster automatically re-replicates any missing data back to it. You don't need to do anything manually.

If a node stays offline for 5 minutes, CockroachDB considers it dead and begins re-replicating its ranges to other live nodes to restore the replication factor.

Node3 not rejoining?

Make sure you ran the cockroach start command from the same directory where the node3/ folder exists. If you get a store directory error, cd back to your cockroachdb-cluster folder on the Desktop and try again.

help me restart my CockroachDB node after it was killed

You just proved fault tolerance in action. Your cluster lost a node and kept serving traffic without a hiccup. Next up, you will push this even further in the Secret Mission by surviving two simultaneous failures.

Secret mission

Survive Two Simultaneous Failures

Your cluster currently replicates data 3 times. With 3 replicas, it can survive 1 node failure (2 out of 3 still form a majority). But what if you need to survive 2 simultaneous failures?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your cluster running, pause it to come back later, or delete everything entirely.

Resources you used:

  • Five CockroachDB node data directories (node1/ through node5/) on your Desktop.
  • Five running CockroachDB processes (one per node).
  • CockroachDB binary installed via Homebrew.

Keep everything running

No action needed. Choose this if you're still actively experimenting with the cluster or plan to continue right away.

  • Your 5-node cluster continues running in the background.
  • Restart any stopped nodes anytime with the same cockroach start commands you used earlier.
  • The DB Console remains available at http://localhost:8080 as long as the cluster is running.

Pause - I'll come back to this later

Stop all running CockroachDB processes to free up memory, but keep your data directories so you can restart the cluster later.

  • Find all running CockroachDB processes by running:
ps -ef | grep cockroach | grep -v grep
  • For each process ID (the number in the second column), stop the node gracefully by running:
kill -TERM <PID>
  • Repeat for all five node processes.
  • Your data directories (node1/ through node5/) remain on disk. To restart the cluster later, run the same cockroach start commands from Step 2 and Step 4.

Delete - I don't want to use this again

Remove all project data and optionally uninstall CockroachDB entirely.

  • Find all running CockroachDB processes by running:
ps -ef | grep cockroach | grep -v grep
  • For each process ID (the number in the second column), stop the node gracefully by running:
kill -TERM <PID>
  • Repeat for all five node processes.
  • Once all processes are stopped, delete the node data directories by running:
rm -rf node1 node2 node3 node4 node5
  • Optionally, uninstall the CockroachDB binary from your machine by running:
brew uninstall cockroachdb/tap/cockroach
  • This removes the cockroach command from your system. You can always reinstall it later with brew install cockroachdb/tap/cockroach.

That's a Wrap!

That's a Wrap!

Nice work! You just experienced the core mechanics of distributed databases firsthand by building, scaling, breaking, and healing a CockroachDB cluster on your own machine.

You learned how to:

  • Built a 3-node CockroachDB cluster and proved automatic replication by reading the same data from any node.
  • Scaled horizontally to 5 nodes and watched automatic rebalancing redistribute data across the cluster in real time.
  • Killed a node and proved fault tolerance. The cluster served queries without interruption.
  • Increased replication to 5x and survived two simultaneous node failures using the Raft consensus protocol.

Ready to quiz yourself?