Build a Distributed Key-Value Store

Build a multi-node key-value store in Go with replication and fault detection.

Introduction

30 Second Summary

Every time you look something up on Google or add an item to your cart on Amazon, your request is handled by one of thousands of servers. If one server crashes, everything keeps working because the data is copied across multiple machines.

In this project, you will build a distributed key-value store in Go. You'll run 3 server nodes on your machine, replicate data between them, and watch the system keep running when a node goes down.

What You'll Build

A distributed in-memory key-value store using only Go's standard library, with HTTP APIs for reading and writing data, automatic write replication across nodes, and heartbeat-based failure detection.

By the end of this project, you'll have:

  • A multi-node key-value store running 3 independent server nodes that replicate writes to each other automatically.
  • A heartbeat health checker that detects when a node stops responding and removes it from the cluster.
  • Firsthand experience with the CAP theorem by killing nodes and observing how consistency and availability trade off.
  • A consistent hashing upgrade that distributes keys across nodes like a real distributed database (Secret Mission).

Are there any prerequisites?

This project assumes basic Go experience with functions, structs, goroutines, and HTTP handlers. You should also be comfortable with REST APIs and using curl from the terminal.

Set Up the Project

Distributed systems power the services you use every day. From databases like DynamoDB to caching layers like Redis, these systems replicate data across machines, detect failures, and keep running when things go wrong.

In this step, you'll create a project folder, initialize a Go module, and write a starter main.go file. This gives you a clean foundation for the key-value store you'll build across the rest of the project.

In this step, get ready to:
  • Create a project directory and initialize a Go module.
  • Set up a main.go file with the basic package structure.
  • Verify the project compiles with go build.
Create the project directory and initialize a Go module

Every Go project starts with a module. A module is a collection of Go source files that are versioned and distributed together.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal and press Enter to open it.

Before creating the project, confirm that Go is installed on your machine.

  • Check your Go installation by running this command:
go version

I see a version number

You should see something like go version go1.26.2. Any version 1.22 or higher works for this project.

Command not found

Go is not installed yet. Follow the official installation guide for your operating system.

  • Download and install Go from the official Go downloads page.
  • After installing, close and reopen your terminal.
  • Run go version again to confirm the installation succeeded.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir distributed-kv
cd distributed-kv
  • Initialize a Go module by running this command:
go mod init distributed-kv

What does go mod init do?

This command creates a go.mod file in your project folder. go.mod tracks your module name and any external dependencies. This project uses only Go's standard library, so the file stays minimal.

  • Confirm the module was created by running:
ls

You should see go.mod in the output.

Don't see go.mod?

  • Make sure you ran go mod init distributed-kv from inside the distributed-kv folder. Run pwd to check your current directory.
  • If you see an error about Go not being found, revisit the installation step above.

Help me troubleshoot my Go module setup

Create the starter main.go file

Go programs start from a main.go file with a main() function. This is the entry point for your entire application.

  • Create the main.go file by running this command:
touch main.go
  • Confirm the file was created by running:
ls

You should see both go.mod and main.go in the output.

Now open this project in your code editor so you can add code to the file.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type VS Code (or the name of your code editor) and press Enter to open it.
  • Click File in the top menu bar.
  • Click Open Folder.
  • Navigate to your Desktop and select the distributed-kv folder, then click Open.
  • Open main.go from the file explorer sidebar on the left.
  • Add the following starter code to main.go:
package main

import (
	"fmt"
)

func main() {
	// Entry point for the distributed key-value store
	fmt.Println("Distributed KV Store starting...")
}

What does this code do?

  • package main tells Go this is an executable program, not a library.
  • import "fmt" brings in Go's formatting package for printing to the console.
  • func main() is the function Go calls when you run the program. Every executable Go program needs one.
  • Save main.go by pressing Cmd+S (macOS) or Ctrl+S (Windows).
Verify the project compiles

A quick build check confirms that your Go module, imports, and code syntax are all correct before you start adding real logic.

  • Back in your terminal, compile the project by running this command:
go build

What does go build do?

go build compiles your Go source files into a binary executable. If the build succeeds, you see no output in the terminal. That silence means everything compiled correctly.

  • Verify the binary was created by running:
ls

You should see a distributed-kv binary file alongside your go.mod and main.go.

  • Test the binary by running it:
./distributed-kv

You should see Distributed KV Store starting... printed in the terminal.

Seeing a compile error?

  • Make sure you saved main.go before running go build. Unsaved changes won't be compiled.
  • Check that your terminal is inside the distributed-kv folder by running pwd. You should see a path ending in /distributed-kv.
  • Look for typos in your code. Go is strict about syntax. Even a missing parenthesis or bracket causes a compile failure.

Help me fix my Go compile error

Awesome, I've got everything!

Double check that you've saved main.go and that go build completed without errors.

I'd like to double check the full code

Here's the complete main.go at this point:

package main

import (
	"fmt"
)

func main() {
	// Entry point for the distributed key-value store
	fmt.Println("Distributed KV Store starting...")
}

Your project scaffolding is ready. Next up, you'll build the actual key-value store with a thread-safe in-memory map and a REST API to read, write, and delete keys.

Build the Single-Node Key-Value Store

Your Go module is initialized and compiling. Now it is time to build the thing it is going to run.

In this step, you will create a single-node key-value store. It is an in-memory data store that lets you save, retrieve, and delete values through a REST API. By the end, you will have a working server you can test from your terminal.

In this step, get ready to:
  • Create a thread-safe in-memory store with Get, Set, and Delete methods.
  • Build HTTP endpoints for storing and retrieving key-value pairs.
  • Run the server and test it with curl.
Create the thread-safe Store struct

Your key-value store needs to handle multiple HTTP requests arriving at the same time. If two requests try to write to the same Go map simultaneously, your program crashes.

A mutex solves this by letting only one goroutine access the map at a time. You will wrap your map in a struct with a read-write mutex so reads can happen concurrently while writes get exclusive access.

  • In main.go, replace the existing code with the following imports and Store struct:
package main

import (
	"flag"
	"fmt"
	"io"
	"log"
	"net/http"
	"sync"
)

// --- Store ---

// Store holds key-value pairs in a thread-safe map
type Store struct {
	mu   sync.RWMutex
	data map[string]string
}

// NewStore creates an empty store
func NewStore() *Store {
	return &Store{data: make(map[string]string)}
}

// Get retrieves a value by key (concurrent-safe read)
func (s *Store) Get(key string) (string, bool) {
	s.mu.RLock()
	defer s.mu.RUnlock()
	val, ok := s.data[key]
	return val, ok
}

// Set stores a key-value pair (exclusive write)
func (s *Store) Set(key, value string) {
	s.mu.Lock()
	defer s.mu.Unlock()
	s.data[key] = value
}

// Delete removes a key from the store (exclusive write)
func (s *Store) Delete(key string) {
	s.mu.Lock()
	defer s.mu.Unlock()
	delete(s.data, key)
}

What does this code do?

  • sync.RWMutex is a read-write mutex. It allows multiple goroutines to read at the same time, but only one goroutine can write.
  • Get uses RLock (read lock) so multiple lookups can happen concurrently without blocking each other.
  • Set and Delete use Lock (write lock) for exclusive access. No other goroutine can read or write while a write is in progress.
  • defer ensures the lock is always released when the function returns, even if something goes wrong.

Pro tip: unused imports

The import block includes packages like flag and net/http that the Store struct does not use yet. Go will not compile with unused imports. That is expected. The file will compile once you add the main function in the next substep.

Create the HTTP handler functions

Go's net/http package includes a built-in router called ServeMux. Since Go 1.22, ServeMux supports method-based routing. This means you can write route patterns like "PUT /kv/{key}" that match both the HTTP method and the URL path.

  • Below the Store code in main.go, add the following main function with the PUT handler:
// --- HTTP Handlers ---

func main() {
	// Define a --port flag so we can run multiple nodes later
	port := flag.String("port", "8080", "Port to listen on")
	flag.Parse()

	addr := fmt.Sprintf("localhost:%s", *port)
	store := NewStore()
	mux := http.NewServeMux()

	// PUT /kv/{key} - store a value
	mux.HandleFunc("PUT /kv/{key}", func(w http.ResponseWriter, r *http.Request) {
		key := r.PathValue("key")
		body, err := io.ReadAll(r.Body)
		if err != nil {
			http.Error(w, "Failed to read body", http.StatusBadRequest)
			return
		}
		value := string(body)
		store.Set(key, value)
		log.Printf("[STORE] SET %s = %s", key, value)

		w.WriteHeader(http.StatusOK)
		fmt.Fprintf(w, "OK")
	})
}

What does this code do?

  • flag.String defines a --port flag with a default of "8080". This lets you run multiple nodes on different ports later.
  • NewStore() creates a fresh in-memory store. http.NewServeMux() creates the HTTP router that maps URL patterns to handler functions.
  • The route pattern "PUT /kv/{key}" tells the ServeMux to match only PUT requests to /kv/something.
  • r.PathValue("key") extracts the {key} wildcard from the URL. io.ReadAll(r.Body) reads the full request body to use as the value.

You will not see a change yet. The next chunk adds the remaining handlers and the server startup.

  • Inside your main function, find the closing }) of the PUT handler. Add the following GET and DELETE handlers below it, before the final closing } of main:
	// GET /kv/{key} - retrieve a value
	mux.HandleFunc("GET /kv/{key}", func(w http.ResponseWriter, r *http.Request) {
		key := r.PathValue("key")
		value, ok := store.Get(key)
		if !ok {
			http.Error(w, "Key not found", http.StatusNotFound)
			return
		}
		w.WriteHeader(http.StatusOK)
		fmt.Fprint(w, value)
	})

	// DELETE /kv/{key} - remove a key
	mux.HandleFunc("DELETE /kv/{key}", func(w http.ResponseWriter, r *http.Request) {
		key := r.PathValue("key")
		store.Delete(key)
		log.Printf("[STORE] DELETE %s", key)

		w.WriteHeader(http.StatusOK)
		fmt.Fprint(w, "OK")
	})

What does this code do?

  • The GET handler looks up a key using store.Get(key). If the key does not exist, it returns a 404 Not Found response.
  • The DELETE handler removes a key from the store and returns OK. It logs the deletion so you can track what is happening on each node.
  • Both handlers use the same r.PathValue("key") pattern to extract the key from the URL.

You will not see a change yet. The next substep adds the server startup so you can run and test everything.

Start the server and test with curl

Your handlers are registered, but the server is not listening yet. You need to call http.ListenAndServe to start accepting connections on the configured port.

  • At the bottom of your main function, just before the closing }, add the server startup code:
	// Start the HTTP server
	log.Printf("[SERVER] Node starting on %s", addr)
	if err := http.ListenAndServe(addr, mux); err != nil {
		log.Fatalf("Server failed: %v", err)
	}

What does this code do?

  • http.ListenAndServe(addr, mux) starts the HTTP server on the given address and routes all incoming requests through your ServeMux.
  • The log.Printf line prints a startup message so you know which port the node is listening on.
  • log.Fatalf exits the program immediately if the server fails to start (for example, if the port is already in use).
  • Save main.go.

Awesome, I've got everything!

Double check you have saved main.go before continuing.

I'd like to double check the full code

Here is the full main.go file you should have at this point.

package main

import (
	"flag"
	"fmt"
	"io"
	"log"
	"net/http"
	"sync"
)

// --- Store ---

type Store struct {
	mu   sync.RWMutex
	data map[string]string
}

func NewStore() *Store {
	return &Store{data: make(map[string]string)}
}

func (s *Store) Get(key string) (string, bool) {
	s.mu.RLock()
	defer s.mu.RUnlock()
	val, ok := s.data[key]
	return val, ok
}

func (s *Store) Set(key, value string) {
	s.mu.Lock()
	defer s.mu.Unlock()
	s.data[key] = value
}

func (s *Store) Delete(key string) {
	s.mu.Lock()
	defer s.mu.Unlock()
	delete(s.data, key)
}

// --- HTTP Handlers ---

func main() {
	port := flag.String("port", "8080", "Port to listen on")
	flag.Parse()

	addr := fmt.Sprintf("localhost:%s", *port)
	store := NewStore()
	mux := http.NewServeMux()

	// Key-value endpoints
	mux.HandleFunc("PUT /kv/{key}", func(w http.ResponseWriter, r *http.Request) {
		key := r.PathValue("key")
		body, err := io.ReadAll(r.Body)
		if err != nil {
			http.Error(w, "Failed to read body", http.StatusBadRequest)
			return
		}
		value := string(body)
		store.Set(key, value)
		log.Printf("[STORE] SET %s = %s", key, value)

		w.WriteHeader(http.StatusOK)
		fmt.Fprintf(w, "OK")
	})

	mux.HandleFunc("GET /kv/{key}", func(w http.ResponseWriter, r *http.Request) {
		key := r.PathValue("key")
		value, ok := store.Get(key)
		if !ok {
			http.Error(w, "Key not found", http.StatusNotFound)
			return
		}
		w.WriteHeader(http.StatusOK)
		fmt.Fprint(w, value)
	})

	mux.HandleFunc("DELETE /kv/{key}", func(w http.ResponseWriter, r *http.Request) {
		key := r.PathValue("key")
		store.Delete(key)
		log.Printf("[STORE] DELETE %s", key)

		w.WriteHeader(http.StatusOK)
		fmt.Fprint(w, "OK")
	})

	log.Printf("[SERVER] Node starting on %s", addr)
	if err := http.ListenAndServe(addr, mux); err != nil {
		log.Fatalf("Server failed: %v", err)
	}
}

Time to see it in action. You will start the server in one terminal window and send requests from another.

  • Start the server by running this command in your project terminal:
go run main.go

You should see a log line that says [SERVER] Node starting on localhost:8080. The terminal stays open because the server is running.

Pro tip: macOS firewall

macOS may show a firewall dialog asking if you want to allow incoming connections. Click Allow. This is harmless and only appears the first time.

  • Open a second terminal window. Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal and press Enter to open a new terminal window.
  • Store a key-value pair by running this command:
curl -X PUT -d "hello" http://localhost:8080/kv/greeting

You should see OK printed in your second terminal. In your first terminal (the server), you should see a log line like [STORE] SET greeting = hello.

  • Read the value back by running:
curl http://localhost:8080/kv/greeting

You should see hello printed back.

  • Delete the key by running:
curl -X DELETE http://localhost:8080/kv/greeting

You should see OK printed.

  • Verify the key is gone by running the GET request again:
curl http://localhost:8080/kv/greeting

You should see Key not found printed. The key was deleted successfully.

Seeing "Connection refused"?

  • Make sure your server is still running in the first terminal. If you see the command prompt instead of the server log, run go run main.go again.
  • Check that your curl URL uses port 8080 and localhost. A typo in the URL is the most common cause.
  • If you see address already in use in the server terminal, another process is using port 8080. Stop it or run with a different port: go run main.go --port 9090.

Still stuck? Help me debug why my Go HTTP server is returning Connection refused when I use curl.

  • Stop the server by pressing Ctrl+C in the server terminal. You will restart it in the next step.

You have a working single-node key-value store. It can store, retrieve, and delete data through HTTP. Next up, you will make multiple nodes aware of each other so they can form a cluster.

Add Node Registration and Cluster Awareness

Your key-value store works on a single node. If that node crashes, every key is gone.

A distributed system solves this by running multiple copies of the same service. For that to work, each node needs to know who else is in the cluster. In this step, you'll build the cluster management layer that lets nodes find each other and join together.

In this step, get ready to:
  • Create a thread-safe Cluster struct that tracks peer addresses.
  • Add HTTP endpoints for joining the cluster and listing peers.
  • Implement automatic cluster joining on startup.
Create the Cluster struct

The Cluster struct is the brain of your node's cluster awareness. It tracks who the current node is and which peers it knows about.

  • In main.go, add the Cluster struct and its constructor below your Store code:
// --- Cluster ---

type Cluster struct {
	mu       sync.RWMutex
	self     string
	peers    []string
	failures map[string]int
}

func NewCluster(self string) *Cluster {
	return &Cluster{
		self:     self,
		peers:    []string{},
		failures: make(map[string]int),
	}
}

What does this code do?

  • self stores this node's own address (e.g. localhost:8080).
  • peers is the list of other nodes this node knows about.
  • failures tracks how many consecutive health check failures each peer has had. You'll use this in Step 5 when you build the health checker.
  • mu is an RWMutex that protects the peer list from concurrent access, just like the one in your Store.

You won't see a change yet. The next chunk adds the methods that make this struct useful.

  • Add the AddPeer method below NewCluster:
func (c *Cluster) AddPeer(addr string) {
	c.mu.Lock()
	defer c.mu.Unlock()
	// Don't add ourselves as a peer
	if addr == c.self {
		return
	}
	// Don't add duplicates
	for _, p := range c.peers {
		if p == addr {
			return
		}
	}
	c.peers = append(c.peers, addr)
	c.failures[addr] = 0
	log.Printf("[CLUSTER] Added peer: %s", addr)
}

What does AddPeer do?

AddPeer has two safety guards. First, it ignores the node's own address so a node never adds itself as a peer. Second, it checks for duplicates so the same peer is never listed twice. Both guards prevent bugs that would surface later during replication.

You won't see a change yet. The remaining utility methods come next.

  • Add the remaining Cluster methods below AddPeer:
func (c *Cluster) RemovePeer(addr string) {
	c.mu.Lock()
	defer c.mu.Unlock()
	for i, p := range c.peers {
		if p == addr {
			c.peers = append(c.peers[:i], c.peers[i+1:]...)
			delete(c.failures, addr)
			log.Printf("[CLUSTER] Removed peer: %s", addr)
			return
		}
	}
}

func (c *Cluster) GetPeers() []string {
	c.mu.RLock()
	defer c.mu.RUnlock()
	// Return a copy so callers can't modify the original
	result := make([]string, len(c.peers))
	copy(result, c.peers)
	return result
}

func (c *Cluster) RecordFailure(addr string) int {
	c.mu.Lock()
	defer c.mu.Unlock()
	c.failures[addr]++
	return c.failures[addr]
}

func (c *Cluster) ResetFailure(addr string) {
	c.mu.Lock()
	defer c.mu.Unlock()
	c.failures[addr] = 0
}

What do these methods do?

  • RemovePeer finds a peer by address and removes it from both the peers slice and the failures map.
  • GetPeers returns a copy of the peer list. Returning a copy prevents callers from accidentally modifying the original slice.
  • RecordFailure and ResetFailure track consecutive health check failures per peer. You'll wire these into a health checker in Step 5.
  • Save main.go.
  • Verify the file compiles by running:
go build

You should see no output, which means the build succeeded.

Seeing a compilation error?

  • Check that the Cluster struct fields match exactly: mu sync.RWMutex, self string, peers []string, failures map[string]int.
  • Make sure every method receiver uses (c *Cluster) with a pointer, not (c Cluster).
  • help me fix this Go compilation error
Add the join and peers endpoints

The Cluster struct manages the peer list in memory. Now you need HTTP endpoints that let other nodes interact with it. A POST /join endpoint lets a new node announce itself, and a GET /peers endpoint lets anyone inspect the cluster state.

Before adding the handlers, you need to update your imports and wire the Cluster into main().

  • In your import block at the top of main.go, find this line:
"flag"
  • Add "encoding/json" directly above it, so your imports look like this:
"encoding/json"
"flag"
  • Now scroll to the top of your main() function. Find these lines:
port := flag.String("port", "8080", "Port to listen on")
flag.Parse()
  • Add a join flag between those two lines, so the block looks like this:
port := flag.String("port", "8080", "Port to listen on")
join := flag.String("join", "", "Address of a node to join (e.g. localhost:8080)")
flag.Parse()
  • Next, find this line below flag.Parse():
store := NewStore()
  • Add the Cluster initialization directly below it:
store := NewStore()
cluster := NewCluster(addr)

What's the join flag for?

The -join flag tells a new node which existing node to contact when starting up. The very first node in the cluster starts without -join. Every subsequent node uses it to announce itself to the cluster.

  • Add the POST /join handler to your mux, below the DELETE /kv/{key} handler:
	// Cluster endpoints
	mux.HandleFunc("POST /join", func(w http.ResponseWriter, r *http.Request) {
		var payload map[string]string
		if err := json.NewDecoder(r.Body).Decode(&payload); err != nil {
			http.Error(w, "Invalid body", http.StatusBadRequest)
			return
		}
		peerAddr := payload["addr"]
		cluster.AddPeer(peerAddr)

		// Return our current peer list (plus ourselves)
		peers := cluster.GetPeers()
		w.Header().Set("Content-Type", "application/json")
		json.NewEncoder(w).Encode(peers)
	})

What does the /join endpoint do?

When a new node POSTs its address to /join, this handler adds it to the local peer list. It also responds with the current list of peers. This is how the joining node discovers everyone else in the cluster with a single request.

You won't see a change yet. The next handler makes the peer list inspectable.

  • Add the GET /peers handler directly below the POST /join handler:
	mux.HandleFunc("GET /peers", func(w http.ResponseWriter, r *http.Request) {
		peers := cluster.GetPeers()
		w.Header().Set("Content-Type", "application/json")
		json.NewEncoder(w).Encode(map[string]interface{}{
			"self":  cluster.self,
			"peers": peers,
		})
	})

What does the /peers endpoint do?

This endpoint returns a JSON object with the node's own address and its list of known peers. You'll use it throughout the project to check cluster state.

  • Save main.go.
  • Stop any running server with Ctrl+C, then restart it by running:
go run main.go -port 8080
  • Open a second terminal window. Test the join endpoint by running:
curl -X POST http://localhost:8080/join -d '{"addr":"localhost:9999"}'

You should see [] as the response. The peer list is empty because this is the first peer being added. In the server terminal, you should see a log line saying [CLUSTER] Added peer: localhost:9999.

  • Now check the peer list by running:
curl http://localhost:8080/peers

You should see a JSON response that includes "self":"localhost:8080" and localhost:9999 in the peers array.

Getting a 404 or connection refused?

  • Make sure you restarted the server after saving your changes. Code changes don't take effect until you stop and re-run.
  • Check that your mux.HandleFunc calls use the exact patterns "POST /join" and "GET /peers". The method prefix is required.
  • help me debug my /join or /peers endpoint
Implement cluster joining on startup

Right now, nodes can accept join requests. But a new node doesn't automatically reach out to anyone. You need a joinCluster function that runs on startup to introduce the new node to the cluster.

  • First, add "bytes" to your import block at the top of main.go. Place it before "encoding/json" to keep imports in alphabetical order.
  • Add the joinCluster function above the main() function:
// --- Join Cluster ---

func joinCluster(cluster *Cluster, seedAddr string) {
	if seedAddr == "" {
		return
	}

	// POST our address to the seed node
	payload, _ := json.Marshal(map[string]string{"addr": cluster.self})
	url := fmt.Sprintf("http://%s/join", seedAddr)
	resp, err := http.Post(url, "application/json", bytes.NewReader(payload))
	if err != nil {
		log.Printf("[JOIN] Failed to join cluster via %s: %v", seedAddr, err)
		return
	}
	defer resp.Body.Close()

	// Read the seed node's peer list
	var peers []string
	if err := json.NewDecoder(resp.Body).Decode(&peers); err != nil {
		log.Printf("[JOIN] Failed to decode peer list: %v", err)
		return
	}

	// Add the seed node and all its peers
	cluster.AddPeer(seedAddr)
	for _, peer := range peers {
		cluster.AddPeer(peer)
		// Notify each existing peer about us
		go func(peer string) {
			notifyURL := fmt.Sprintf("http://%s/join", peer)
			payload, _ := json.Marshal(map[string]string{"addr": cluster.self})
			resp, err := http.Post(notifyURL, "application/json", bytes.NewReader(payload))
			if err == nil {
				resp.Body.Close()
			}
		}(peer)
	}
	log.Printf("[JOIN] Successfully joined cluster via %s", seedAddr)
}

What does joinCluster do?

This function runs a three-phase join process:

  • Phase 1: POST our address to the seed node's /join endpoint.
  • Phase 2: Read the peer list that comes back in the response. This tells us who else is in the cluster.
  • Phase 3: For each existing peer, fire off a goroutine to notify them about us. This way every node in the cluster learns about the new node, not just the seed.

Why use goroutines for peer notifications?

Each notification is an HTTP request that could be slow or fail. Using go func() fires all the notifications concurrently so the joining node doesn't wait for each one to finish sequentially.

  • Now wire the function call into main(). Find this line near the bottom:
	log.Printf("[SERVER] Node starting on %s", addr)
  • Add the joinCluster call directly above it, so the block looks like this:
	// Start systems
	joinCluster(cluster, *join)

	log.Printf("[SERVER] Node starting on %s", addr)
  • Save main.go.
  • Stop any running server with Ctrl+C.
  • Start the first node by running:
go run main.go -port 8080
  • Open a second terminal window and start a second node that joins the first by running:
go run main.go -port 8081 -join localhost:8080

In the second terminal, you should see log lines showing [CLUSTER] Added peer: localhost:8080 and [JOIN] Successfully joined cluster via localhost:8080.

  • Open a third terminal window. Verify both nodes know about each other by running:
curl http://localhost:8080/peers
curl http://localhost:8081/peers

Both nodes should show each other in their peer lists. Node 8080 lists localhost:8081 as a peer, and node 8081 lists localhost:8080.

Nodes not finding each other?

  • Make sure the first node is fully started before launching the second. Look for the [SERVER] Node starting on log line in the first terminal.
  • Check that the -join flag uses the exact address localhost:8080, not http://localhost:8080. The function adds the http:// prefix automatically.
  • If macOS shows a firewall dialog asking to allow incoming connections, click Allow. This is normal when Go starts listening on a port.
  • help me debug cluster joining

Awesome, I've got everything!

Double check you've saved main.go and both nodes show each other in their peer lists.

I'd like to double check the full code

Here is the complete main.go as it should look at the end of this step.

package main

import (
	"bytes"
	"encoding/json"
	"flag"
	"fmt"
	"io"
	"log"
	"net/http"
	"sync"
)

// --- Store ---

type Store struct {
	mu   sync.RWMutex
	data map[string]string
}

func NewStore() *Store {
	return &Store{data: make(map[string]string)}
}

func (s *Store) Get(key string) (string, bool) {
	s.mu.RLock()
	defer s.mu.RUnlock()
	val, ok := s.data[key]
	return val, ok
}

func (s *Store) Set(key, value string) {
	s.mu.Lock()
	defer s.mu.Unlock()
	s.data[key] = value
}

func (s *Store) Delete(key string) {
	s.mu.Lock()
	defer s.mu.Unlock()
	delete(s.data, key)
}

// --- Cluster ---

type Cluster struct {
	mu       sync.RWMutex
	self     string
	peers    []string
	failures map[string]int
}

func NewCluster(self string) *Cluster {
	return &Cluster{
		self:     self,
		peers:    []string{},
		failures: make(map[string]int),
	}
}

func (c *Cluster) AddPeer(addr string) {
	c.mu.Lock()
	defer c.mu.Unlock()
	if addr == c.self {
		return
	}
	for _, p := range c.peers {
		if p == addr {
			return
		}
	}
	c.peers = append(c.peers, addr)
	c.failures[addr] = 0
	log.Printf("[CLUSTER] Added peer: %s", addr)
}

func (c *Cluster) RemovePeer(addr string) {
	c.mu.Lock()
	defer c.mu.Unlock()
	for i, p := range c.peers {
		if p == addr {
			c.peers = append(c.peers[:i], c.peers[i+1:]...)
			delete(c.failures, addr)
			log.Printf("[CLUSTER] Removed peer: %s", addr)
			return
		}
	}
}

func (c *Cluster) GetPeers() []string {
	c.mu.RLock()
	defer c.mu.RUnlock()
	result := make([]string, len(c.peers))
	copy(result, c.peers)
	return result
}

func (c *Cluster) RecordFailure(addr string) int {
	c.mu.Lock()
	defer c.mu.Unlock()
	c.failures[addr]++
	return c.failures[addr]
}

func (c *Cluster) ResetFailure(addr string) {
	c.mu.Lock()
	defer c.mu.Unlock()
	c.failures[addr] = 0
}

// --- Join Cluster ---

func joinCluster(cluster *Cluster, seedAddr string) {
	if seedAddr == "" {
		return
	}

	// POST our address to the seed node
	payload, _ := json.Marshal(map[string]string{"addr": cluster.self})
	url := fmt.Sprintf("http://%s/join", seedAddr)
	resp, err := http.Post(url, "application/json", bytes.NewReader(payload))
	if err != nil {
		log.Printf("[JOIN] Failed to join cluster via %s: %v", seedAddr, err)
		return
	}
	defer resp.Body.Close()

	// Read the seed node's peer list
	var peers []string
	if err := json.NewDecoder(resp.Body).Decode(&peers); err != nil {
		log.Printf("[JOIN] Failed to decode peer list: %v", err)
		return
	}

	// Add the seed node and all its peers
	cluster.AddPeer(seedAddr)
	for _, peer := range peers {
		cluster.AddPeer(peer)
		// Notify each existing peer about us
		go func(peer string) {
			notifyURL := fmt.Sprintf("http://%s/join", peer)
			payload, _ := json.Marshal(map[string]string{"addr": cluster.self})
			resp, err := http.Post(notifyURL, "application/json", bytes.NewReader(payload))
			if err == nil {
				resp.Body.Close()
			}
		}(peer)
	}
	log.Printf("[JOIN] Successfully joined cluster via %s", seedAddr)
}

// --- HTTP Handlers ---

func main() {
	port := flag.String("port", "8080", "Port to listen on")
	join := flag.String("join", "", "Address of a node to join (e.g. localhost:8080)")
	flag.Parse()

	addr := fmt.Sprintf("localhost:%s", *port)
	store := NewStore()
	cluster := NewCluster(addr)
	mux := http.NewServeMux()

	// Key-value endpoints
	mux.HandleFunc("PUT /kv/{key}", func(w http.ResponseWriter, r *http.Request) {
		key := r.PathValue("key")
		body, err := io.ReadAll(r.Body)
		if err != nil {
			http.Error(w, "Failed to read body", http.StatusBadRequest)
			return
		}
		value := string(body)
		store.Set(key, value)
		log.Printf("[STORE] SET %s = %s", key, value)
		w.WriteHeader(http.StatusOK)
		fmt.Fprintf(w, "OK")
	})

	mux.HandleFunc("GET /kv/{key}", func(w http.ResponseWriter, r *http.Request) {
		key := r.PathValue("key")
		value, ok := store.Get(key)
		if !ok {
			http.Error(w, "Key not found", http.StatusNotFound)
			return
		}
		w.WriteHeader(http.StatusOK)
		fmt.Fprint(w, value)
	})

	mux.HandleFunc("DELETE /kv/{key}", func(w http.ResponseWriter, r *http.Request) {
		key := r.PathValue("key")
		store.Delete(key)
		log.Printf("[STORE] DELETE %s", key)
		w.WriteHeader(http.StatusOK)
		fmt.Fprint(w, "OK")
	})

	// Cluster endpoints
	mux.HandleFunc("POST /join", func(w http.ResponseWriter, r *http.Request) {
		var payload map[string]string
		if err := json.NewDecoder(r.Body).Decode(&payload); err != nil {
			http.Error(w, "Invalid body", http.StatusBadRequest)
			return
		}
		peerAddr := payload["addr"]
		cluster.AddPeer(peerAddr)

		// Return our current peer list (plus ourselves)
		peers := cluster.GetPeers()
		w.Header().Set("Content-Type", "application/json")
		json.NewEncoder(w).Encode(peers)
	})

	mux.HandleFunc("GET /peers", func(w http.ResponseWriter, r *http.Request) {
		peers := cluster.GetPeers()
		w.Header().Set("Content-Type", "application/json")
		json.NewEncoder(w).Encode(map[string]interface{}{
			"self":  cluster.self,
			"peers": peers,
		})
	})

	// Start systems
	joinCluster(cluster, *join)

	log.Printf("[SERVER] Node starting on %s", addr)
	if err := http.ListenAndServe(addr, mux); err != nil {
		log.Fatalf("Server failed: %v", err)
	}
}

Your nodes can now discover each other and form a cluster. Next up, you'll make them share data by replicating writes across the cluster.

Implement Write Replication

Your nodes can discover each other and form a cluster. But there's a problem: if you write a key to Node A, only Node A has it.

In this step, you'll build a replication engine that broadcasts every write to all peers in the cluster. When you save a key on Node A, Nodes B and C get it too.

In this step, get ready to:
  • Build a replicate function that broadcasts writes to all peers.
  • Connect the replicate function to the PUT and DELETE handlers.
  • Add loop prevention with a custom HTTP header.
Implement the replicate function

Right now, each node is an island. A write to one node stays on that one node forever. The replicate function fixes this by sending the same write to every peer in the cluster.

  • In main.go, add "strings" and "time" to your import block.
  • Below your Cluster methods (after the GetPeers function), add the replication section by pasting this code:
// --- Replication ---

// replicate sends a write operation to all peers in the cluster
func replicate(cluster *Cluster, method, key, value string) {
	peers := cluster.GetPeers()
	for _, peer := range peers {
		// Send each replication request in its own goroutine
		go func(peer string) {
			url := fmt.Sprintf("http://%s/kv/%s", peer, key)
			var req *http.Request
			var err error

			// Build the appropriate request based on the HTTP method
			if method == http.MethodDelete {
				req, err = http.NewRequest(http.MethodDelete, url, nil)
			} else {
				req, err = http.NewRequest(http.MethodPut, url, strings.NewReader(value))
			}
			if err != nil {
				log.Printf("[REPLICATION] Error creating request for %s: %v", peer, err)
				return
			}

			// Mark this as a replicated request to prevent infinite loops
			req.Header.Set("X-Replicated", "true")
			client := &http.Client{Timeout: 2 * time.Second}
			resp, err := client.Do(req)
			if err != nil {
				log.Printf("[REPLICATION] Failed to replicate to %s: %v", peer, err)
				return
			}
			resp.Body.Close()
			log.Printf("[REPLICATION] Replicated %s /kv/%s to %s", method, key, peer)
		}(peer)
	}
}

What does this code do?

  • GetPeers() grabs a snapshot of the current peer list. This is safe to call from multiple goroutines because GetPeers uses a read lock.
  • Each peer gets its own goroutine (go func(peer string)), so replication happens asynchronously. The main handler returns immediately without waiting for peers to confirm.
  • The function builds either a PUT or DELETE request depending on the method argument, then sends it to the peer's /kv/{key} endpoint.
  • req.Header.Set("X-Replicated", "true") marks the outgoing request with a custom header. You'll use this in the next substep to prevent infinite replication loops.
  • resp.Body.Close() releases the HTTP connection. Skipping this leaks resources over time.

Why async replication?

This design makes your system eventually consistent. The client gets a response as soon as the local write succeeds, without waiting for peers to confirm. There is a brief window where peers might have stale data.

The alternative is synchronous replication, where you wait for all peers to confirm before responding. That gives you stronger consistency but slower writes and lower availability if any peer is down.

  • Save main.go.

You won't see a change yet. The replicate function exists but nothing calls it. You'll connect it to the handlers next.

Add replication to the PUT and DELETE handlers

Now you need to call replicate from your handlers so that every write gets broadcast to peers. But there's a catch you need to handle first.

When Node A gets a write, it replicates to Node B. Node B's PUT handler fires and receives the data. Without a guard, Node B would replicate the write back to Node A. Node A would replicate to B again. This creates an infinite replication loop.

How does the X-Replicated header prevent loops?

The replicate function already sets X-Replicated: true on every outgoing request. Now the handlers need to check for that header. If it is present, the write came from a peer, and the handler should not replicate it further.

Client writes (no header) trigger replication. Peer writes (header present) do not. One hop, no loops.

  • In main.go, scroll to your PUT handler and find this line:
log.Printf("[STORE] SET %s = %s", key, value)
  • Add the following code between that line and w.WriteHeader(http.StatusOK):
// Only replicate if this is an original client request
if r.Header.Get("X-Replicated") != "true" {
	replicate(cluster, http.MethodPut, key, value)
}

What does this code do?

r.Header.Get("X-Replicated") checks whether this request was sent by a peer's replicate function. If it was, the handler skips replication and just stores the value locally. If it wasn't (meaning a client sent it), the handler replicates the write to all peers.

  • Now scroll to your DELETE handler and find this line:
log.Printf("[STORE] DELETE %s", key)
  • Add the same replication guard between that line and w.WriteHeader(http.StatusOK):
// Only replicate if this is an original client request
if r.Header.Get("X-Replicated") != "true" {
	replicate(cluster, http.MethodDelete, key, "")
}

Why is the value an empty string for DELETE?

The replicate function accepts a value parameter, but DELETE requests don't carry a body. Passing an empty string satisfies the function signature while the function internally skips the body for DELETE requests.

  • Save main.go.
  • Verify the project compiles by running:
go build

If it compiles with no output, your replication logic is wired up correctly.

Seeing a compile error?

  • If you see undefined: replicate, make sure the replicate function is defined outside the main function, not inside it.
  • If you see imported and not used: "strings" or the same for "time", double check that your replicate function uses strings.NewReader and time.Second. Go does not allow unused imports.
  • If you see undefined: cluster inside a handler, make sure the replication if-block is inside the handler's anonymous function (where cluster is captured from the enclosing scope).
  • Still stuck? Help me debug this compile error in my Go replication code.

Awesome, I've got everything!

Double check you've saved main.go and that go build compiles with no errors.

I'd like to double check the full code

package main

import (
	"bytes"
	"encoding/json"
	"flag"
	"fmt"
	"io"
	"log"
	"net/http"
	"strings"
	"sync"
	"time"
)

// --- Store ---

type Store struct {
	mu   sync.RWMutex
	data map[string]string
}

func NewStore() *Store {
	return &Store{data: make(map[string]string)}
}

func (s *Store) Get(key string) (string, bool) {
	s.mu.RLock()
	defer s.mu.RUnlock()
	val, ok := s.data[key]
	return val, ok
}

func (s *Store) Set(key, value string) {
	s.mu.Lock()
	defer s.mu.Unlock()
	s.data[key] = value
}

func (s *Store) Delete(key string) {
	s.mu.Lock()
	defer s.mu.Unlock()
	delete(s.data, key)
}

// --- Cluster ---

type Cluster struct {
	mu   sync.RWMutex
	self string
	peers []string
}

func NewCluster(self string) *Cluster {
	return &Cluster{
		self:  self,
		peers: []string{},
	}
}

func (c *Cluster) AddPeer(addr string) {
	c.mu.Lock()
	defer c.mu.Unlock()
	if addr == c.self {
		return
	}
	for _, p := range c.peers {
		if p == addr {
			return
		}
	}
	c.peers = append(c.peers, addr)
	log.Printf("[CLUSTER] Added peer: %s", addr)
}

func (c *Cluster) RemovePeer(addr string) {
	c.mu.Lock()
	defer c.mu.Unlock()
	for i, p := range c.peers {
		if p == addr {
			c.peers = append(c.peers[:i], c.peers[i+1:]...)
			log.Printf("[CLUSTER] Removed peer: %s", addr)
			return
		}
	}
}

func (c *Cluster) GetPeers() []string {
	c.mu.RLock()
	defer c.mu.RUnlock()
	result := make([]string, len(c.peers))
	copy(result, c.peers)
	return result
}

// --- Replication ---

func replicate(cluster *Cluster, method, key, value string) {
	peers := cluster.GetPeers()
	for _, peer := range peers {
		go func(peer string) {
			url := fmt.Sprintf("http://%s/kv/%s", peer, key)
			var req *http.Request
			var err error

			if method == http.MethodDelete {
				req, err = http.NewRequest(http.MethodDelete, url, nil)
			} else {
				req, err = http.NewRequest(http.MethodPut, url, strings.NewReader(value))
			}
			if err != nil {
				log.Printf("[REPLICATION] Error creating request for %s: %v", peer, err)
				return
			}

			req.Header.Set("X-Replicated", "true")
			client := &http.Client{Timeout: 2 * time.Second}
			resp, err := client.Do(req)
			if err != nil {
				log.Printf("[REPLICATION] Failed to replicate to %s: %v", peer, err)
				return
			}
			resp.Body.Close()
			log.Printf("[REPLICATION] Replicated %s /kv/%s to %s", method, key, peer)
		}(peer)
	}
}

// --- Join Cluster ---

func joinCluster(cluster *Cluster, seedAddr string) {
	if seedAddr == "" {
		return
	}

	// POST our address to the seed node
	payload, _ := json.Marshal(map[string]string{"addr": cluster.self})
	url := fmt.Sprintf("http://%s/join", seedAddr)
	resp, err := http.Post(url, "application/json", bytes.NewReader(payload))
	if err != nil {
		log.Printf("[JOIN] Failed to join cluster via %s: %v", seedAddr, err)
		return
	}
	defer resp.Body.Close()

	// Read the seed node's peer list
	var peers []string
	if err := json.NewDecoder(resp.Body).Decode(&peers); err != nil {
		log.Printf("[JOIN] Failed to decode peer list: %v", err)
		return
	}

	// Add the seed node and all its peers
	cluster.AddPeer(seedAddr)
	for _, peer := range peers {
		cluster.AddPeer(peer)
		// Notify each existing peer about us
		go func(peer string) {
			notifyURL := fmt.Sprintf("http://%s/join", peer)
			payload, _ := json.Marshal(map[string]string{"addr": cluster.self})
			resp, err := http.Post(notifyURL, "application/json", bytes.NewReader(payload))
			if err == nil {
				resp.Body.Close()
			}
		}(peer)
	}
	log.Printf("[JOIN] Successfully joined cluster via %s", seedAddr)
}

// --- HTTP Handlers ---

func main() {
	port := flag.String("port", "8080", "Port to listen on")
	join := flag.String("join", "", "Address of a node to join (e.g. localhost:8080)")
	flag.Parse()

	addr := fmt.Sprintf("localhost:%s", *port)
	store := NewStore()
	cluster := NewCluster(addr)
	mux := http.NewServeMux()

	// Key-value endpoints
	mux.HandleFunc("PUT /kv/{key}", func(w http.ResponseWriter, r *http.Request) {
		key := r.PathValue("key")
		body, err := io.ReadAll(r.Body)
		if err != nil {
			http.Error(w, "Failed to read body", http.StatusBadRequest)
			return
		}
		value := string(body)
		store.Set(key, value)
		log.Printf("[STORE] SET %s = %s", key, value)

		if r.Header.Get("X-Replicated") != "true" {
			replicate(cluster, http.MethodPut, key, value)
		}

		w.WriteHeader(http.StatusOK)
		fmt.Fprintf(w, "OK")
	})

	mux.HandleFunc("GET /kv/{key}", func(w http.ResponseWriter, r *http.Request) {
		key := r.PathValue("key")
		value, ok := store.Get(key)
		if !ok {
			http.Error(w, "Key not found", http.StatusNotFound)
			return
		}
		w.WriteHeader(http.StatusOK)
		fmt.Fprint(w, value)
	})

	mux.HandleFunc("DELETE /kv/{key}", func(w http.ResponseWriter, r *http.Request) {
		key := r.PathValue("key")
		store.Delete(key)
		log.Printf("[STORE] DELETE %s", key)

		if r.Header.Get("X-Replicated") != "true" {
			replicate(cluster, http.MethodDelete, key, "")
		}

		w.WriteHeader(http.StatusOK)
		fmt.Fprint(w, "OK")
	})

	// Cluster endpoints
	mux.HandleFunc("POST /join", func(w http.ResponseWriter, r *http.Request) {
		var payload map[string]string
		if err := json.NewDecoder(r.Body).Decode(&payload); err != nil {
			http.Error(w, "Invalid body", http.StatusBadRequest)
			return
		}
		peerAddr := payload["addr"]
		cluster.AddPeer(peerAddr)

		// Return our current peer list (plus ourselves)
		peers := cluster.GetPeers()
		w.Header().Set("Content-Type", "application/json")
		json.NewEncoder(w).Encode(peers)
	})

	mux.HandleFunc("GET /peers", func(w http.ResponseWriter, r *http.Request) {
		peers := cluster.GetPeers()
		w.Header().Set("Content-Type", "application/json")
		json.NewEncoder(w).Encode(map[string]interface{}{
			"self":  cluster.self,
			"peers": peers,
		})
	})

	// Start systems
	joinCluster(cluster, *join)

	log.Printf("[SERVER] Node starting on %s", addr)
	if err := http.ListenAndServe(addr, mux); err != nil {
		log.Fatalf("Server failed: %v", err)
	}
}

Your nodes can now replicate writes to each other. Next up, you'll build a heartbeat health checker that detects when a node goes down and removes it from the cluster automatically.

Add Heartbeat Health Checking

Your nodes can now replicate writes to each other. But what happens when a node crashes? The remaining nodes have no idea it is gone. They keep trying to replicate to a dead address, wasting time and logging errors.

In this step, you will build a heartbeat health checking system. Each node will ping its peers every few seconds. If a peer stops responding, the node removes it from the cluster automatically.

In this step, get ready to:
  • Create a background health check goroutine that pings peers every 3 seconds.
  • Add a health endpoint for peers to ping.
  • Track consecutive failures and automatically remove unresponsive peers.
Create the health check goroutine

A heartbeat is a periodic signal that one node sends to another to confirm it is still alive. Your health checker will run as a background goroutine that loops forever, pinging every peer on a timer.

  • In main.go, find the // --- Replication --- section. Add the following startHealthCheck function below the replicate function:
// --- Health Check ---

func startHealthCheck(cluster *Cluster) {
	// Create a ticker that fires every 3 seconds
	ticker := time.NewTicker(3 * time.Second)
	// Use a short timeout so a dead peer doesn't block the loop
	client := &http.Client{Timeout: 1 * time.Second}

	go func() {
		for range ticker.C {
			peers := cluster.GetPeers()
			for _, peer := range peers {
				url := fmt.Sprintf("http://%s/health", peer)
				resp, err := client.Get(url)
				if err != nil || resp.StatusCode != http.StatusOK {
					// Peer didn't respond or returned an error
					count := cluster.RecordFailure(peer)
					log.Printf("[HEALTH] Peer %s failed health check (%d/3)", peer, count)
					if count >= 3 {
						cluster.RemovePeer(peer)
						log.Printf("[HEALTH] Peer %s removed from cluster", peer)
					}
				} else {
					// Peer is healthy, reset its failure count
					cluster.ResetFailure(peer)
					resp.Body.Close()
				}
			}
		}
	}()
}

What does this code do?

  • time.NewTicker(3 * time.Second) creates a channel that sends a value every 3 seconds. The for range ticker.C loop blocks until each tick arrives.
  • The http.Client has a 1-second timeout. If a peer does not respond within 1 second, the request fails immediately instead of hanging.
  • On each tick, the function loops through all peers and sends a GET /health request. If the request fails or returns a non-200 status, the peer's failure count goes up. Three consecutive failures trigger removal.
  • If the peer responds successfully, its failure count resets to zero. This prevents a single network hiccup from killing a healthy peer.

Why heartbeats instead of on-demand checks?

You could check if a peer is alive only when you need to replicate a write. But that means you would not discover a dead peer until the next write happens. Heartbeats run continuously in the background, so your cluster detects failures within seconds regardless of traffic.

This function references RecordFailure and ResetFailure methods that do not exist yet. You will add them in the third substep. Your code will not compile until then.

Add the /health endpoint

Your health checker sends GET /health requests to peers. Each node needs a handler that responds to those requests.

  • In the main function, find the GET /peers handler. Add this new handler below it:
	mux.HandleFunc("GET /health", func(w http.ResponseWriter, r *http.Request) {
		w.WriteHeader(http.StatusOK)
		fmt.Fprint(w, "OK")
	})

What does this endpoint do?

This is the simplest possible endpoint. It returns a 200 OK status with the text OK. If the node is running, the response succeeds. If the node is down, the request times out. That binary signal is all the health checker needs.

Track consecutive failures and remove dead peers

The startHealthCheck function calls RecordFailure and ResetFailure on the cluster. You need to add a failures counter to the Cluster struct and implement those methods.

  • In main.go, scroll to the Cluster struct definition.
  • Find this line:
type Cluster struct {
	mu       sync.RWMutex
	self     string
	peers    []string
}
  • Add a failures field to the struct, so it looks like this:
type Cluster struct {
	mu       sync.RWMutex
	self     string
	peers    []string
	failures map[string]int
}
  • Next, find the NewCluster function. It currently looks like this:
func NewCluster(self string) *Cluster {
	return &Cluster{
		self:     self,
		peers:    []string{},
	}
}
  • Add the failures map initialization, so it looks like this:
func NewCluster(self string) *Cluster {
	return &Cluster{
		self:     self,
		peers:    []string{},
		failures: make(map[string]int),
	}
}
  • Now find the AddPeer method. Find this line inside it:
	c.peers = append(c.peers, addr)
	log.Printf("[CLUSTER] Added peer: %s", addr)
  • Add c.failures[addr] = 0 between those two lines, so it looks like this:
	c.peers = append(c.peers, addr)
	c.failures[addr] = 0
	log.Printf("[CLUSTER] Added peer: %s", addr)
  • Find the RemovePeer method. Find this line inside the if p == addr block:
			c.peers = append(c.peers[:i], c.peers[i+1:]...)
			log.Printf("[CLUSTER] Removed peer: %s", addr)
  • Add delete(c.failures, addr) between those two lines, so it looks like this:
			c.peers = append(c.peers[:i], c.peers[i+1:]...)
			delete(c.failures, addr)
			log.Printf("[CLUSTER] Removed peer: %s", addr)

Why track failures in the Cluster struct?

The failures map stores a running count of consecutive failed health checks per peer address. Initializing it to 0 in AddPeer means every new peer starts with a clean slate. Cleaning it up in RemovePeer prevents stale entries from accumulating.

Now you need the two methods that the health checker calls to update those failure counts.

  • Add the RecordFailure and ResetFailure methods below the GetPeers method:
func (c *Cluster) RecordFailure(addr string) int {
	c.mu.Lock()
	defer c.mu.Unlock()
	// Increment the failure counter and return the new count
	c.failures[addr]++
	return c.failures[addr]
}

func (c *Cluster) ResetFailure(addr string) {
	c.mu.Lock()
	defer c.mu.Unlock()
	// Peer responded successfully, so clear its failure count
	c.failures[addr] = 0
}

What do these methods do?

  • RecordFailure increments the failure counter for a peer and returns the new count. The health checker uses this return value to decide whether the peer has hit 3 failures.
  • ResetFailure sets the counter back to zero. A single successful health check erases all previous failures for that peer.
  • Both methods lock the mutex before modifying the map, keeping the cluster data thread-safe.

The last piece is to actually start the health checker when the node boots up.

  • In the main function, find the // Start systems comment near the bottom of the file. It currently looks like this:
	// Start systems
	joinCluster(cluster, *join)
  • Add the startHealthCheck(cluster) call above the joinCluster line, so it looks like this:
	// Start systems
	startHealthCheck(cluster)
	joinCluster(cluster, *join)
  • Save main.go.
  • Verify your code compiles by running:
go build -o kvstore

You should see no output and no errors. The command produces a kvstore binary in your project folder.

Seeing a compile error?

  • If you see cluster.RecordFailure undefined, make sure you added both the RecordFailure and ResetFailure methods to the Cluster section of your file.
  • If you see c.failures undefined, check that you added the failures map[string]int field to the Cluster struct definition.
  • If you see assignment to entry in nil map at runtime, make sure NewCluster initializes the failures map with make(map[string]int).
  • Help me fix my compile error.

Awesome, I've got everything!

Double check you have saved main.go before moving on.

I'd like to double check the full code

package main

import (
	"bytes"
	"encoding/json"
	"flag"
	"fmt"
	"io"
	"log"
	"net/http"
	"strings"
	"sync"
	"time"
)

// --- Store ---

type Store struct {
	mu   sync.RWMutex
	data map[string]string
}

func NewStore() *Store {
	return &Store{data: make(map[string]string)}
}

func (s *Store) Get(key string) (string, bool) {
	s.mu.RLock()
	defer s.mu.RUnlock()
	val, ok := s.data[key]
	return val, ok
}

func (s *Store) Set(key, value string) {
	s.mu.Lock()
	defer s.mu.Unlock()
	s.data[key] = value
}

func (s *Store) Delete(key string) {
	s.mu.Lock()
	defer s.mu.Unlock()
	delete(s.data, key)
}

// --- Cluster ---

type Cluster struct {
	mu       sync.RWMutex
	self     string
	peers    []string
	failures map[string]int
}

func NewCluster(self string) *Cluster {
	return &Cluster{
		self:     self,
		peers:    []string{},
		failures: make(map[string]int),
	}
}

func (c *Cluster) AddPeer(addr string) {
	c.mu.Lock()
	defer c.mu.Unlock()
	if addr == c.self {
		return
	}
	for _, p := range c.peers {
		if p == addr {
			return
		}
	}
	c.peers = append(c.peers, addr)
	c.failures[addr] = 0
	log.Printf("[CLUSTER] Added peer: %s", addr)
}

func (c *Cluster) RemovePeer(addr string) {
	c.mu.Lock()
	defer c.mu.Unlock()
	for i, p := range c.peers {
		if p == addr {
			c.peers = append(c.peers[:i], c.peers[i+1:]...)
			delete(c.failures, addr)
			log.Printf("[CLUSTER] Removed peer: %s", addr)
			return
		}
	}
}

func (c *Cluster) GetPeers() []string {
	c.mu.RLock()
	defer c.mu.RUnlock()
	result := make([]string, len(c.peers))
	copy(result, c.peers)
	return result
}

func (c *Cluster) RecordFailure(addr string) int {
	c.mu.Lock()
	defer c.mu.Unlock()
	c.failures[addr]++
	return c.failures[addr]
}

func (c *Cluster) ResetFailure(addr string) {
	c.mu.Lock()
	defer c.mu.Unlock()
	c.failures[addr] = 0
}

// --- Replication ---

func replicate(cluster *Cluster, method, key, value string) {
	peers := cluster.GetPeers()
	for _, peer := range peers {
		go func(peer string) {
			url := fmt.Sprintf("http://%s/kv/%s", peer, key)
			var req *http.Request
			var err error

			if method == http.MethodDelete {
				req, err = http.NewRequest(http.MethodDelete, url, nil)
			} else {
				req, err = http.NewRequest(http.MethodPut, url, strings.NewReader(value))
			}
			if err != nil {
				log.Printf("[REPLICATION] Error creating request for %s: %v", peer, err)
				return
			}

			req.Header.Set("X-Replicated", "true")
			client := &http.Client{Timeout: 2 * time.Second}
			resp, err := client.Do(req)
			if err != nil {
				log.Printf("[REPLICATION] Failed to replicate to %s: %v", peer, err)
				return
			}
			resp.Body.Close()
			log.Printf("[REPLICATION] Replicated %s /kv/%s to %s", method, key, peer)
		}(peer)
	}
}

// --- Health Check ---

func startHealthCheck(cluster *Cluster) {
	ticker := time.NewTicker(3 * time.Second)
	client := &http.Client{Timeout: 1 * time.Second}

	go func() {
		for range ticker.C {
			peers := cluster.GetPeers()
			for _, peer := range peers {
				url := fmt.Sprintf("http://%s/health", peer)
				resp, err := client.Get(url)
				if err != nil || resp.StatusCode != http.StatusOK {
					count := cluster.RecordFailure(peer)
					log.Printf("[HEALTH] Peer %s failed health check (%d/3)", peer, count)
					if count >= 3 {
						cluster.RemovePeer(peer)
						log.Printf("[HEALTH] Peer %s removed from cluster", peer)
					}
				} else {
					cluster.ResetFailure(peer)
					resp.Body.Close()
				}
			}
		}
	}()
}

// --- Join Cluster ---

func joinCluster(cluster *Cluster, seedAddr string) {
	if seedAddr == "" {
		return
	}

	// POST our address to the seed node
	payload, _ := json.Marshal(map[string]string{"addr": cluster.self})
	url := fmt.Sprintf("http://%s/join", seedAddr)
	resp, err := http.Post(url, "application/json", bytes.NewReader(payload))
	if err != nil {
		log.Printf("[JOIN] Failed to join cluster via %s: %v", seedAddr, err)
		return
	}
	defer resp.Body.Close()

	// Read the seed node's peer list
	var peers []string
	if err := json.NewDecoder(resp.Body).Decode(&peers); err != nil {
		log.Printf("[JOIN] Failed to decode peer list: %v", err)
		return
	}

	// Add the seed node and all its peers
	cluster.AddPeer(seedAddr)
	for _, peer := range peers {
		cluster.AddPeer(peer)
		// Notify each existing peer about us
		go func(peer string) {
			notifyURL := fmt.Sprintf("http://%s/join", peer)
			payload, _ := json.Marshal(map[string]string{"addr": cluster.self})
			resp, err := http.Post(notifyURL, "application/json", bytes.NewReader(payload))
			if err == nil {
				resp.Body.Close()
			}
		}(peer)
	}
	log.Printf("[JOIN] Successfully joined cluster via %s", seedAddr)
}

// --- HTTP Handlers ---

func main() {
	port := flag.String("port", "8080", "Port to listen on")
	join := flag.String("join", "", "Address of a node to join (e.g. localhost:8080)")
	flag.Parse()

	addr := fmt.Sprintf("localhost:%s", *port)
	store := NewStore()
	cluster := NewCluster(addr)
	mux := http.NewServeMux()

	// Key-value endpoints
	mux.HandleFunc("PUT /kv/{key}", func(w http.ResponseWriter, r *http.Request) {
		key := r.PathValue("key")
		body, err := io.ReadAll(r.Body)
		if err != nil {
			http.Error(w, "Failed to read body", http.StatusBadRequest)
			return
		}
		value := string(body)
		store.Set(key, value)
		log.Printf("[STORE] SET %s = %s", key, value)

		if r.Header.Get("X-Replicated") != "true" {
			replicate(cluster, http.MethodPut, key, value)
		}

		w.WriteHeader(http.StatusOK)
		fmt.Fprintf(w, "OK")
	})

	mux.HandleFunc("GET /kv/{key}", func(w http.ResponseWriter, r *http.Request) {
		key := r.PathValue("key")
		value, ok := store.Get(key)
		if !ok {
			http.Error(w, "Key not found", http.StatusNotFound)
			return
		}
		w.WriteHeader(http.StatusOK)
		fmt.Fprint(w, value)
	})

	mux.HandleFunc("DELETE /kv/{key}", func(w http.ResponseWriter, r *http.Request) {
		key := r.PathValue("key")
		store.Delete(key)
		log.Printf("[STORE] DELETE %s", key)

		if r.Header.Get("X-Replicated") != "true" {
			replicate(cluster, http.MethodDelete, key, "")
		}

		w.WriteHeader(http.StatusOK)
		fmt.Fprint(w, "OK")
	})

	// Cluster endpoints
	mux.HandleFunc("POST /join", func(w http.ResponseWriter, r *http.Request) {
		var payload map[string]string
		if err := json.NewDecoder(r.Body).Decode(&payload); err != nil {
			http.Error(w, "Invalid body", http.StatusBadRequest)
			return
		}
		peerAddr := payload["addr"]
		cluster.AddPeer(peerAddr)

		// Return our current peer list (plus ourselves)
		peers := cluster.GetPeers()
		w.Header().Set("Content-Type", "application/json")
		json.NewEncoder(w).Encode(peers)
	})

	mux.HandleFunc("GET /peers", func(w http.ResponseWriter, r *http.Request) {
		peers := cluster.GetPeers()
		w.Header().Set("Content-Type", "application/json")
		json.NewEncoder(w).Encode(map[string]interface{}{
			"self":  cluster.self,
			"peers": peers,
		})
	})

	mux.HandleFunc("GET /health", func(w http.ResponseWriter, r *http.Request) {
		w.WriteHeader(http.StatusOK)
		fmt.Fprint(w, "OK")
	})

	// Start systems
	startHealthCheck(cluster)
	joinCluster(cluster, *join)

	log.Printf("[SERVER] Node starting on %s", addr)
	if err := http.ListenAndServe(addr, mux); err != nil {
		log.Fatalf("Server failed: %v", err)
	}
}

Your distributed system now has a self-healing superpower. When a node goes down, the rest of the cluster detects it and moves on. Next up, you will put the entire system to the test by running three nodes and breaking things on purpose.

Test the Distributed System

Your distributed key-value store now has every component in place. You have a thread-safe store, cluster joining, write replication, and heartbeat health checks.

Time to put it all together. You'll spin up three nodes, write data across them, and then break the cluster on purpose to see how it responds.

In this step, get ready to:
  • Start 3 nodes on different ports and form a cluster.
  • Write a key to one node and read it from another to verify replication.
  • Kill a node and watch the health checker detect the failure and remove it.
Start 3 nodes and form a cluster

Each node runs as its own go run process on a separate port. Since each server keeps its terminal occupied, you need a separate terminal window for each node.

  • Open a new terminal tab by pressing Cmd+T (macOS) or Ctrl+Shift+T (Windows).
  • Navigate to your project folder (the one containing main.go) by running:
cd ~/Desktop/distributed-kv
  • Start the first node on port 8080 by running:
go run main.go -port 8080

You should see [SERVER] Node starting on localhost:8080 in the terminal. This node is now running and waiting for requests.

Pro tip: macOS firewall dialog

The first time Go listens on a port, macOS may show a dialog asking to allow incoming connections. Click Allow. This is normal and only appears once.

  • Open a second terminal tab by pressing Cmd+T (macOS) or Ctrl+Shift+T (Windows).
  • Navigate to your project folder again by running:
cd ~/Desktop/distributed-kv
  • Start the second node on port 8081 and join it to the first node by running:
go run main.go -port 8081 -join localhost:8080

You should see join and cluster messages in this terminal. In the first terminal, you should also see [CLUSTER] Added peer: localhost:8081.

  • Open a third terminal tab by pressing Cmd+T (macOS) or Ctrl+Shift+T (Windows).
  • Navigate to your project folder and start the third node on port 8082 by running:
cd ~/Desktop/distributed-kv
go run main.go -port 8082 -join localhost:8080

All three terminals should now show cluster activity. Each node knows about the other two.

  • Open a fourth terminal tab. This one stays free for running curl commands.
  • Verify the cluster formed correctly by running:
curl http://localhost:8080/peers

You should see a JSON response listing localhost:8081 and localhost:8082 as peers of node 1.

Seeing "connection refused"?

  • Make sure all three go run processes are still running. If any terminal shows an error or exited, restart that node.
  • Check that you used different -port values for each node. Two nodes on the same port causes a conflict.
  • If you see address already in use, a previous run may still be holding the port. Stop it with Ctrl+C in the relevant terminal and try again.
  • My nodes won't connect to each other
Write a key and read it from another node

The cluster is running. Now you'll prove that write replication actually works by storing a key on one node and reading it from a different one.

  • In your fourth terminal (the curl terminal), write a key to node 1 by running:
curl -X PUT -d "distributed-systems" http://localhost:8080/kv/topic

You should see OK in the terminal. Check the first terminal window (node 1). You should see log lines showing the store operation and replication to both peers.

  • Read the key from node 2 to verify it was replicated by running:
curl http://localhost:8081/kv/topic

You should see distributed-systems printed in the terminal. The key was written to node 1 but node 2 has it too.

  • Confirm node 3 also has the key by running:
curl http://localhost:8082/kv/topic

Same result. All three nodes have the same data. This is full replication in action.

  • Test a delete operation by removing the key from node 2 and reading from node 1:
curl -X DELETE http://localhost:8081/kv/topic
curl http://localhost:8080/kv/topic

The first command returns OK. The second command returns Key not found. The delete replicated from node 2 to all other nodes.

What just happened?

When you wrote a key to node 1, the PUT handler stored it locally, then called the replicate function. That function sent the same PUT request to nodes 2 and 3 with the X-Replicated: true header.

When nodes 2 and 3 received those requests, they saw the header and stored the value without replicating again. That header is what prevents an infinite replication loop.

Key not found on the other node?

  • Make sure all three nodes are still running. If a node crashed, it couldn't receive the replicated write.
  • Check the node 1 terminal for [REPLICATION] log lines. If you see Failed to replicate, the target node was unreachable.
  • Try writing a new key and reading it again. Replication is async, so there is a tiny delay, but it should arrive within a second.
  • Why isn't my key replicating to the other nodes?
Kill a node and observe failure detection

Replication works when everything is healthy. The real test of a distributed system is what happens when something breaks.

  • First, write a new key so all three nodes have data. Run this in your curl terminal:
curl -X PUT -d "hello-world" http://localhost:8080/kv/greeting
  • Switch to the third terminal (node 3 on port 8082).
  • Press Ctrl+C to kill the node.

Node 3 is dead. Now watch the first and second terminal windows closely.

  • Wait about 10 seconds. The health checker runs every 3 seconds and removes a peer after 3 consecutive failures.

In the node 1 and node 2 terminals, you should see log lines like this appear:

[HEALTH] Peer localhost:8082 failed health check (1/3)
[HEALTH] Peer localhost:8082 failed health check (2/3)
[HEALTH] Peer localhost:8082 failed health check (3/3)
[HEALTH] Peer localhost:8082 removed from cluster

What does this log output mean?

The startHealthCheck goroutine pings each peer every 3 seconds. When node 3 stopped responding, the failure counter incremented each cycle. After 3 consecutive failures (about 9 seconds), the health checker called RemovePeer to drop it from the cluster.

  • Confirm node 3 was removed by checking the peer list on node 1:
curl http://localhost:8080/peers

You should see only localhost:8081 in the peers list. Node 3 is gone.

  • Verify the remaining nodes still work by reading the key from node 2:
curl http://localhost:8081/kv/greeting

You should see hello-world. The cluster lost a node, but the surviving nodes still serve data. The system stayed available.

  • Try writing a new key to node 1 while node 3 is still down:
curl -X PUT -d "only-two-nodes" http://localhost:8080/kv/status
  • Read that key from node 2 to confirm replication still works between the surviving nodes:
curl http://localhost:8081/kv/status

You should see only-two-nodes. Replication keeps working with the remaining peers. Node 3, if it ever comes back, will not have this key. That is the trade-off.

What is the CAP theorem?

The CAP theorem states that a distributed system can guarantee at most two of three properties: Consistency (every read returns the latest write), Availability (every request gets a response), and Partition Tolerance (the system works despite network splits).

Your key-value store is an AP system. When node 3 went down (a partition), the remaining nodes stayed available and kept serving requests. But they sacrificed consistency because node 3 missed the latest writes.

In practice, network partitions always happen. The real design question is whether you favor consistency or availability. This is one of the most common topics in system design interviews.

What is eventual consistency?

Your replication is asynchronous. There is a brief window after a write where different nodes can return different values for the same key. Given enough time with no new writes, all nodes converge to the same state. This model is called eventual consistency. Systems like DynamoDB and Cassandra use this same approach.

Secret mission

Add Consistent Hashing for Key Distribution

Right now, every node stores every key. That works for 3 nodes, but what about 3,000? Real distributed databases like DynamoDB and Cassandra use consistent hashing to distribute keys across nodes. Can you add it to yours?

Before you go

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally, so there are no ongoing costs.

Resources you used:

  • Go project directory containing main.go and go.mod.
  • Three terminal processes running nodes on ports 8080, 8081, and 8082.
  • Compiled Go binary (if generated by go build).

Keep everything running

No action needed. Choose this if you are still actively building or want to experiment more with the cluster.

  • Leave your terminal windows open with the nodes running.
  • Keep the project directory as-is for future reference.

Pause - I'll come back to this later

Shut down the running nodes to free up your terminal sessions. Your project files stay on disk for next time.

  • Press Ctrl+C in each terminal window to stop the running nodes.
  • Close the terminal windows.
  • Your main.go and go.mod files will still be in your project directory when you come back.

Delete - I don't want to use this again

Remove all project resources and start fresh.

  • Press Ctrl+C in each terminal window to stop all running nodes.
  • Delete the entire project directory.
rm -rf distributed-kv
  • That is it. There are no cloud resources or accounts to clean up.

Mission accomplished

That's a Wrap!

You just built a distributed system from scratch. No frameworks, no magic. Just Go, HTTP, and a map.

You've learned how to:

  • Build a thread-safe in-memory key-value store with a REST API using only Go's standard library.
  • Implement node discovery, cluster joining, and write replication with loop prevention using custom HTTP headers.
  • Build a heartbeat-based health checker that detects and removes failed nodes, and observe the CAP theorem in action by killing nodes and watching the cluster adapt.
  • Add consistent hashing to distribute keys across nodes like a real distributed database.

Ready to quiz yourself?