Build a ZooKeeper Leader Election

Run a 3-node ZooKeeper ensemble and implement leader election locally.

Introduction

30 Second Summary

When multiple services need to agree on who's in charge or who gets exclusive access to a shared resource, things get complicated fast. Without a dedicated coordinator, distributed systems descend into race conditions and split-brain scenarios.

In this project, you will build a 3-node ZooKeeper ensemble using Docker Compose, explore its hierarchical data model through the interactive CLI, and implement a leader election that automatically recovers when the leader crashes.

What You'll Build

Picture three ZooKeeper nodes running on your machine, with you connected from multiple terminals, watching in real time as nodes compete for leadership and the cluster heals itself the moment one goes down.

By the end of this project, you'll have:

A running 3-node ZooKeeper cluster you can spin up and tear down with a single docker compose command.
Real-time watch notifications firing across terminals the instant you change coordination data from a separate session.
A leader election simulation where you see exactly which node wins, then watch automatic failover happen when the leader disappears.
Secret Mission: Build a distributed lock using ephemeral sequential nodes and understand why watching only your predecessor prevents a stampede.

Are there any prerequisites?

You need Docker Desktop installed on your machine. No local Java or ZooKeeper installation is required because everything runs inside Docker containers.

Before We Start

Before We Start...

Before we dive in, let's lock in what you're about to build and why it matters.

Launch a 3-Node ZooKeeper Ensemble

To explore how ZooKeeper coordinates distributed systems, you first need a running cluster to connect to.

Docker Compose lets you define all three ZooKeeper nodes in a single configuration file and launch them with one command. In this step, you will create that file and bring your ensemble to life.

In this step, get ready to:

Create your project folder and Docker Compose configuration.
Start the 3-node ensemble and verify all nodes are healthy.
Connect to the ZooKeeper CLI and confirm the namespace is live.

Create a project folder and Docker Compose file

A ZooKeeper ensemble needs at least three nodes to tolerate a failure (two out of three still form a majority). Your Docker Compose file will define all three.

Press the Windows key to open your search bar.
Type PowerShell and press Enter to open it.

Move to your Desktop by running this command:

cd ~/Desktop

Create your project folder and move into it by running these commands:

mkdir zk-lab
cd zk-lab

Confirm the folder was created by running:

ls

You should see an empty folder (no output). Your terminal prompt now shows you are inside zk-lab.

Create a new file called compose.yaml inside the zk-lab folder by running:

New-Item compose.yaml

Open compose.yaml in a text editor (Notepad, VS Code, or any editor you prefer).
Paste the following configuration for the first ZooKeeper node:

services:
  zoo1:
    image: zookeeper:3.9.5
    restart: always
    hostname: zoo1
    container_name: zoo1
    ports:
      - 2181:2181
    environment:
      ZOO_MY_ID: 1
      ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181

What does this configuration do?

image: zookeeper:3.9.5 pulls the official ZooKeeper Docker image pinned to version 3.9.5.
ZOO_MY_ID: 1 gives this node a unique identity within the ensemble.
ZOO_SERVERS lists every node in the ensemble so they can find each other. Port 2888 is for follower-to-leader communication. Port 3888 is for leader election.
ports: 2181:2181 exposes the client port on your machine so you can connect to this node directly.

Add a blank line after the last line of zoo1, then paste the remaining two nodes below:


  zoo2:
    image: zookeeper:3.9.5
    restart: always
    hostname: zoo2
    container_name: zoo2
    ports:
      - 2182:2181
    environment:
      ZOO_MY_ID: 2
      ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181

  zoo3:
    image: zookeeper:3.9.5
    restart: always
    hostname: zoo3
    container_name: zoo3
    ports:
      - 2183:2181
    environment:
      ZOO_MY_ID: 3
      ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181

What is different about zoo2 and zoo3?

Each node has a unique ZOO_MY_ID (2 and 3) and a unique host port (2182 and 2183). Inside each container, ZooKeeper still listens on port 2181.

The ZOO_SERVERS string is identical across all three. Every node needs to know the full membership of the ensemble.

Save compose.yaml.
Back in your PowerShell terminal, confirm the file exists by running:

ls

You should see compose.yaml listed in the output.

✔️ Awesome, I've got everything!

Great. Make sure you have saved the file before moving on.

ⓧ I'd like to double check the full code

Your compose.yaml should contain three service blocks. Scroll up and confirm:

The file starts with services: followed by the zoo1 block (chunk 1 above).
Below zoo1, a blank line separates the zoo2 and zoo3 blocks (chunk 2 above).
Each node has a unique ZOO_MY_ID (1, 2, 3) and a unique host port (2181, 2182, 2183).
The ZOO_SERVERS line is identical in all three blocks.

Don't see compose.yaml in the output?

Make sure your terminal is inside the zk-lab folder. Run pwd to check your current location.
Confirm you saved the file with the exact name compose.yaml (not compose.yaml.txt). Some editors add a hidden extension.

Help me find my compose.yaml file

Start the ensemble and verify health

Your configuration is ready. Time to pull the ZooKeeper image and start all three nodes.

Start the ensemble in detached mode by running:

docker compose up -d

What does this command do?

Docker Compose reads your compose.yaml, pulls the zookeeper:3.9.5 image (if not already downloaded), creates a shared network, and starts all three containers. The -d flag runs them in the background so your terminal stays free.

First run takes longer

The first time you run this, Docker downloads the ZooKeeper image (around 300 MB). Subsequent runs start in seconds because the image is cached locally.

Verify all three containers are running by running:

docker compose ps

You should see zoo1, zoo2, and zoo3 all showing a status of running.

Check which node is the ensemble leader by running:

docker exec zoo1 zkServer.sh status

The output tells you whether zoo1 is a leader or a follower. Either result is fine. ZooKeeper automatically elects one of the three nodes as leader on startup.

What is a quorum?

A quorum is the minimum number of nodes that must agree before ZooKeeper processes a write. With 3 nodes, the quorum is 2 (a majority). This is why ZooKeeper ensembles use odd numbers. You will test this directly in a later step.

Containers not showing as running?

Make sure Docker Desktop is running. Look for the Docker whale icon in your system tray.
Check the logs for errors by running docker compose logs in the zk-lab folder.
If you see a port conflict error, another service on your machine is using port 2181, 2182, or 2183. Stop that service or change the host port in your compose.yaml.

Help me debug my ZooKeeper containers

Connect to the ZooKeeper CLI

ZooKeeper ships with a built-in command-line client called zkCli.sh. You will use it to interact with the cluster for the rest of this project.

Open an interactive CLI session connected to zoo1 by running:

docker exec -it zoo1 zkCli.sh

What does this command do?

docker exec -it runs a command inside a running container with an interactive terminal. zkCli.sh is ZooKeeper's built-in CLI tool. It automatically connects to the local ZooKeeper instance on port 2181.

After a moment, you should see a connection confirmation and a [zk: localhost:2181(CONNECTED) 0] prompt.

Before you run your first command, what do you think lives at the root of ZooKeeper's namespace right now?

List the root namespace by typing this into the ZooKeeper CLI:

ls /

You should see [zookeeper] in the output. This is the default internal node that ZooKeeper creates for its own bookkeeping. Your namespace is live and ready for you to populate.

Seeing a connection refused error?

ZooKeeper may still be starting up. Wait 10 seconds and try the docker exec -it zoo1 zkCli.sh command again.
Run docker compose ps to make sure zoo1 is still running. If it restarted, check docker compose logs zoo1 for errors.

Help me connect to the ZooKeeper CLI

Your 3-node ensemble is running and you have a live CLI session. Next up, you will explore ZooKeeper's data model by creating different types of nodes and watching them behave.

Explore ZooKeeper's Data Model

Your 3-node ensemble is running and you've confirmed the CLI connects to it. Now it's time to understand what makes ZooKeeper useful as a coordination service.

ZooKeeper stores all coordination data in a hierarchical namespace of nodes called znodes. Each znode can hold a small piece of data and have children, much like a file system. In this step, you'll create different znode types and observe the behaviors that make distributed coordination possible.

In this step, get ready to:

Create persistent znodes and build a namespace hierarchy.
Observe how ephemeral nodes disappear when sessions end.
Create sequential nodes and observe automatic ordering.

Create persistent znodes and a namespace hierarchy

Persistent znodes are the default node type. They survive session disconnects and remain in the namespace until explicitly deleted. This makes them ideal for storing configuration data that should always be available.

In your Terminal 1 CLI session (still connected to zoo1 from the previous step), create a parent node by running:

create /services ""

Create a child node with data representing a service endpoint:

create /services/app1 "host1:8080"

You should see Created /services/app1 in the output.

What are these znodes?

You just built a two-level hierarchy: /services is a parent with empty data (a namespace container), and /services/app1 is a child storing a service address.

In a real distributed system, each application instance would register itself under /services so that other processes can discover available endpoints.

Verify the hierarchy by listing the children of /services:

ls /services

You should see [app1] in the output.

Read back the data stored in the child node:

get /services/app1

You should see host1:8080 printed.

Update the node's data to simulate a service changing its address:

set /services/app1 "host2:9090"

Confirm the update and inspect the node's metadata by running:

get -s /services/app1

You should see host2:9090 as the data, plus a stat block showing fields like cZxid, dataVersion, and ephemeralOwner.

What does the stat output tell me?

The dataVersion field is now 1 because you updated the data once since creation. Each set increments this counter.

The ephemeralOwner field shows 0x0, confirming this is a persistent node (not tied to any session).

Seeing 'Node does not exist'?

ZooKeeper requires you to create parent nodes before children. If you tried to create /services/app1 without creating /services first, it fails. Create the parent first, then retry.

help me fix a 'Node does not exist' error in ZooKeeper CLI

Create ephemeral nodes and observe session behavior

Ephemeral nodes are tied to the session that created them. When that session ends, the node automatically disappears. This is how distributed systems detect when a process has crashed or disconnected.

Create an ephemeral node representing a running instance by running:

create -e /services/app1/instance1 "alive"

The -e flag makes this node ephemeral. You should see Created /services/app1/instance1 in the output.

Open a second terminal window on your machine.
Connect this second terminal to zoo2 by running:

docker exec -it zoo2 zkCli.sh -server zoo2:2181

You now have two CLI sessions open: Terminal 1 on zoo1, Terminal 2 on zoo2.

From Terminal 2, verify the ephemeral node exists by running:

ls /services/app1

You should see [instance1] in the output. The node is visible from any connected client because the data is replicated across the ensemble.

Why connect to a different server?

Connecting Terminal 2 to zoo2 (instead of zoo1) proves that znodes are replicated across the entire ensemble. Every server in the cluster sees the same data, regardless of which server created it.

Switch back to Terminal 1 and kill its session by typing:

quit

This closes the CLI session that created the ephemeral node.

Switch to Terminal 2 and check the children of /services/app1 again:

ls /services/app1

The output now shows []. The instance1 node has vanished because the session that created it ended.

Why do ephemeral nodes matter?

In a real system, each application instance creates an ephemeral node when it starts. If the instance crashes or loses network connectivity, its session times out and the node disappears automatically.

Other services watching that path get notified instantly that the instance is gone. No manual cleanup or health-check polling required.

Reconnect Terminal 1 to zoo1 so both terminals are ready for upcoming steps:

docker exec -it zoo1 zkCli.sh

You should see the CONNECTED state in the welcome message.

Terminal 2 still shows instance1?

Ephemeral node deletion depends on session timeout, not just quit. Wait a few seconds and run ls /services/app1 again. The ZooKeeper session timeout is typically a few seconds. The node will disappear once the server detects the session is gone.

help me understand why my ephemeral node hasn't disappeared yet

Create sequential nodes and observe ordering

Sequential nodes get an automatically incrementing number appended to their name. ZooKeeper guarantees these numbers increase monotonically, which is the foundation for ordering-based recipes like leader election.

In Terminal 2, create three sequential nodes by running each command one at a time:

create -s /services/app1/worker- "w1"

create -s /services/app1/worker- "w2"

create -s /services/app1/worker- "w3"

What does the -s flag do?

The -s flag tells ZooKeeper to append a 10-digit, zero-padded sequence number to the node name. Each create -s response shows you the full path including the assigned number.

The sequence is monotonically increasing per parent node. This ordering guarantee is what makes leader election and distributed locking possible.

List the children to see the assigned sequence numbers:

ls /services/app1

You should see nodes named worker-0000000001, worker-0000000002, and worker-0000000003.

Why the 10-digit format?

The zero-padded format (%010d) means simple lexicographic sorting equals numeric sorting. Any client can determine ordering just by comparing node names as strings, without parsing numbers.

Seeing different sequence numbers?

The counter is per parent node and tracks all child changes. If you created extra nodes or retried commands, your numbers may differ. The important thing is that they are strictly increasing.

help me understand why my sequential node numbers are different than expected

Verify the data model

Before you move on, take a moment to predict what you'll see. If you run get -s /services/app1 right now, what data value and ephemeralOwner value do you expect?

From Terminal 2, run the following to verify:

get -s /services/app1

You should see host2:9090 as the data. The stat metadata shows dataVersion greater than 0 (confirming your earlier update), and ephemeralOwner = 0x0 confirming this is a persistent node.

You've now explored all three fundamental znode types that power ZooKeeper's coordination recipes. Next up, you'll see how watches let distributed processes react to changes in real time.

See Watches in Action

You've explored ZooKeeper's data model by creating persistent, ephemeral, and sequential nodes. Now you know how to store and retrieve coordination data.

There's a problem though. If a distributed process needs to react when data changes, it would have to keep asking ZooKeeper "has anything changed yet?" over and over. This constant polling wastes resources and defeats the purpose of a coordination service. Watches solve this by letting ZooKeeper push notifications to clients the instant something changes.

In this step, get ready to:

Set a data watch on a znode and trigger it from a separate terminal.
Set a child watch and observe it fire when a new node is created.

Set a data watch and trigger it

A data watch tells ZooKeeper: "notify me the next time this node's value changes." You set it by adding the -w flag to a get command.

Switch to Terminal 1 (connected to zoo1).
Set a data watch on /services/app1 by running this command:

get -w /services/app1

You should see the current data for the node printed back to you. The watch is now silently armed in the background.

What does the -w flag do?

The -w flag registers a one-time watch with ZooKeeper. The next time the node's data changes, ZooKeeper sends a notification to this client. After that single notification fires, the watch is gone. You would need to set it again to get another notification.

Before you trigger the watch, predict what you think will happen in Terminal 1 when you change the node's data from Terminal 2.

Switch to Terminal 2 (connected to zoo2).
Update the node's data by running this command:

set /services/app1 "host3:7070"

Switch back to Terminal 1 and look at the output.

Terminal 1 immediately prints a watch notification. It looks like this:

WATCHER:: WatchedEvent state:SyncConnected type:NodeDataChanged path:/services/app1

That notification arrived without Terminal 1 asking for it. ZooKeeper pushed it the instant the data changed.

Why do watches fire only once?

Watches are one-time triggers. After a watch fires, it is consumed. If you run set again in Terminal 2, Terminal 1 will NOT receive another notification unless you re-register the watch with another get -w command.

This design keeps ZooKeeper lightweight. Clients re-register only the watches they still care about.

Don't see the watch notification?

Make sure you ran get -w /services/app1 BEFORE running the set command. The watch must be armed first. If you already triggered the set, re-run get -w /services/app1 in Terminal 1 and try the set again from Terminal 2.

Help me debug why my ZooKeeper watch isn't firing

Set a child watch and trigger it

Data watches fire when a node's value changes. Child watches fire when children are added or removed from a node. This is how services detect when new instances register or existing ones disappear.

Switch to Terminal 1 (connected to zoo1).
Set a child watch on /services/app1 by running:

ls -w /services/app1

You should see the current list of children (your worker nodes) printed back. The child watch is now armed.

Switch to Terminal 2 (connected to zoo2).
Create a new ephemeral child node by running:

create -e /services/app1/new-instance "alive"

Switch back to Terminal 1 and look at the output.

Terminal 1 prints a child watch notification:

WATCHER:: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/services/app1

ZooKeeper detected that a new child was added and immediately notified Terminal 1. In a real system, this is how a load balancer discovers new service instances the moment they come online.

How do real systems use child watches?

Imagine a service registry where each running instance creates an ephemeral child node under /services/app1. A load balancer sets a child watch on that path. When a new instance starts, the watch fires and the load balancer re-reads the children list to update its routing table.

When an instance crashes, its ephemeral node vanishes and the same watch fires. The load balancer removes it from the pool. No polling required.

Child watch didn't fire?

Confirm you ran ls -w /services/app1 (not get -w) in Terminal 1. The ls -w command watches for child changes. The get -w command watches for data changes on the node itself.

Help me understand why my child watch isn't firing in ZooKeeper

You now have hands-on proof that watches work. ZooKeeper pushes notifications in real time. Watches fire exactly once. Next up, you'll use these primitives to build the most common ZooKeeper coordination pattern: leader election.

Implement Leader Election

Your configuration data works and watches fire in real time across terminals. But imagine you have three application servers all running at the same time. How do they agree on which one is "in charge" without a human picking one manually?

Leader election is a coordination pattern where competing processes agree on exactly one leader at any given time. In this step, you'll implement leader election using ephemeral sequential nodes and observe what happens when the leader crashes.

In this step, get ready to:

Set up an election namespace with candidates from three separate terminals.
Identify which candidate wins the election based on sequence numbers.
Simulate a leader crash and observe automatic failover with zero configuration.

Set up the election namespace

The leader election recipe starts with a parent znode that all candidates register under. Each candidate creates an ephemeral sequential child node. Because the nodes are ephemeral, they vanish when the candidate disconnects.

In terminal 1 (your zoo1 CLI session from earlier), create the election parent node by running:

create /election ""

Now register terminal 1 as a candidate by creating an ephemeral sequential node:

create -s -e /election/candidate- "node1"

ZooKeeper responds with the full path of the created node, something like:

Created /election/candidate-0000000000

What does this command do?

The -s flag makes it sequential (ZooKeeper appends a 10-digit counter). The -e flag makes it ephemeral (it disappears when the session ends).

The data "node1" is a label identifying which process this candidate represents. In a real system, this would be a hostname or process ID.

Switch to terminal 2 (your zoo2 CLI session) and register a second candidate:

create -s -e /election/candidate- "node2"

You should see a response like Created /election/candidate-0000000001.

Open a third terminal and connect to zoo3 by running:

docker exec -it zoo3 zkCli.sh

In terminal 3, register the third candidate:

create -s -e /election/candidate- "node3"

You should see Created /election/candidate-0000000002.

Verify all three candidates registered by running this from any terminal:

ls /election

You should see three nodes listed: candidate-0000000000, candidate-0000000001, and candidate-0000000002.

Don't see three candidates?

Make sure you used both -s and -e flags in every create command. If a terminal disconnected before you ran ls, that candidate's ephemeral node will have already vanished. Reconnect and create it again.

Help me figure out why my election candidates aren't showing up.

Determine the leader

The election rule is simple: the candidate with the lowest sequence number wins. No voting, no consensus rounds. ZooKeeper's sequential counter guarantees a total ordering.

From any terminal, list the election candidates:

ls /election

The node candidate-0000000000 has the lowest sequence number. That candidate is the current leader.

Confirm which process the leader represents by running:

get /election/candidate-0000000000

You should see node1 in the output. This tells you that the process registered from terminal 1 is the leader.

Why does the lowest number win?

This is the leader election recipe from ZooKeeper's official documentation. Each process calls create with the sequence and ephemeral flags, then calls getChildren() on the election node. The process whose child has the lowest sequence number holds the leadership.

Because sequence numbers are monotonically increasing, the first process to register always gets the lowest number. Ties are impossible.

Simulate leader failure and observe automatic re-election

In a real system, the leader might crash at any moment. When that happens, its ephemeral node vanishes and the remaining candidates need a new leader. You are about to see this happen live.

Before you do this, think about what you expect to happen. If you close terminal 1's session, what will happen to candidate-0000000000? Which node will become the new leader?

In terminal 1 (the zoo1 CLI session), type the following to kill the session:

quit

This ends terminal 1's ZooKeeper session. The ephemeral node it created is now gone.

Switch to terminal 2 or terminal 3 and check the election state:

ls /election

You should see only two candidates now: candidate-0000000001 and candidate-0000000002. The previous leader (candidate-0000000000) has disappeared because its session ended.

The new leader is candidate-0000000001 because it now has the lowest sequence number. No reconfiguration was needed. No process had to "vote" or reach consensus. Failover happened automatically the moment the leader's session closed.

Why is this called automatic failover?

The remaining candidates did not need to re-register or run any election logic. The rule stays the same: the lowest sequence number wins. When a leader disappears, the next candidate in line inherits leadership with zero extra work.

This is why ephemeral nodes are critical. If the nodes were persistent, a crashed leader's node would stay forever and no failover could happen.

Still seeing three candidates?

ZooKeeper takes a few seconds to detect a session timeout and remove the ephemeral node. Wait 5-10 seconds and run ls /election again. If the node persists, make sure you typed quit in the correct terminal (the one connected to zoo1 that created the first candidate).

Help me understand why my ephemeral node didn't disappear after I quit.

You just implemented the most common ZooKeeper coordination pattern from scratch. Next up, you'll test what happens when a ZooKeeper server itself goes down and prove that the cluster survives.

Test Fault Tolerance of the Ensemble

Your leader election works. When a client disconnects, a new leader takes over automatically. But what happens when a ZooKeeper server itself crashes?

In production, hardware fails constantly. ZooKeeper handles this with a quorum rule: as long as a majority of servers are alive, the cluster stays operational. You're about to test this by deliberately killing servers and finding the breaking point.

In this step, get ready to:

Kill one ZooKeeper node and confirm the cluster keeps serving data.
Observe what happens when quorum is lost.
Restore the ensemble and verify data integrity.

Kill one ZooKeeper node and verify the cluster survives

You currently have 3 nodes running. A 3-node ensemble only needs 2 nodes alive (a majority) to keep serving requests. Time to prove it.

Open a new terminal window and navigate to your zk-lab folder on your Desktop by running:

cd ~/Desktop/zk-lab

Stop zoo3 to simulate a server crash by running this command:

docker compose stop zoo3

What does this simulate?

Stopping a container mimics a server going offline unexpectedly. The remaining nodes (zoo1 and zoo2) must decide whether they can still serve requests without zoo3.

Before you connect to the cluster, predict: will zoo1 still respond to your commands with only 2 out of 3 nodes running?

Connect to zoo1's CLI by running:

docker exec -it zoo1 zkCli.sh

Check your election data is still accessible by running:

ls /election

Verify your service data is intact by running:

get /services/app1

The cluster still responds and your data is intact. Two out of three nodes form a majority, so the ensemble keeps working.

Type quit to exit the CLI.
Check which node is the current ensemble leader by running:

docker exec zoo1 zkServer.sh status

You'll see zoo1 reporting as either leader or follower. With zoo3 gone, one of the two surviving nodes has taken over leadership duties.

CLI won't connect to zoo1?

Make sure you ran docker compose stop zoo3 (not docker compose stop zoo1). Zoo1 must still be running for you to connect.

Run docker compose ps to confirm zoo1 and zoo2 show as running. Help me troubleshoot my ZooKeeper connection after stopping a node.

Observe what happens when quorum is lost

The cluster survived one node going down. What happens if you kill a second node, leaving only zoo1 alive?

Stop zoo2 as well by running this command in your zk-lab terminal:

docker compose stop zoo2

Now only zoo1 is left. Before you try connecting, predict: can a single node out of three form a majority?

Try connecting to zoo1 by running:

docker exec -it zoo1 zkCli.sh

The CLI attempts to connect but fails. You'll see connection loss messages or the session will hang without responding to commands. ZooKeeper refuses to serve requests because it cannot form a quorum.

What is the quorum rule?

With N servers in an ensemble, ZooKeeper requires N/2 + 1 nodes alive to serve requests. For a 3-node cluster, that means 2 nodes minimum.

With only 1 of 3 nodes running, there is no majority. ZooKeeper shuts down operations rather than risk serving stale or inconsistent data. This is why production ensembles always use an odd number of nodes: 3, 5, or 7.

Press Ctrl+C to exit the stuck CLI session.
Restore the cluster by starting both stopped nodes:

docker compose start zoo2 zoo3

Wait a few seconds for the nodes to rejoin the ensemble.
Reconnect to zoo1 to confirm the cluster is healthy again by running:

docker exec -it zoo1 zkCli.sh

Before running your verification, predict: will your data from earlier steps still be there, or did the quorum loss destroy it?

Verify everything is back by running:

ls /

You should see [election, services, zookeeper] in the output. All your persistent data survived the outage. ZooKeeper replicates data across all nodes, so no information was lost during the quorum disruption.

Why did the data survive?

ZooKeeper replicates every write to a majority of nodes before acknowledging it. When the cluster comes back, nodes sync their transaction logs and converge on the same state.

Ephemeral nodes tied to sessions that expired during the outage will be gone. But persistent nodes (like /services and /election) are durable by design.

Cluster still not responding after restart?

Run docker compose ps to confirm all three containers show a running state. If a container shows as restarting, wait 10 seconds and check again.

If the CLI still hangs, exit with Ctrl+C and retry the docker exec -it zoo1 zkCli.sh command. Help me get my ZooKeeper cluster responding after a quorum loss recovery.

Secret mission

Implement a Distributed Lock

Leader election picks a coordinator. But what about exclusive access to a shared resource? Build a distributed lock using ephemeral sequential nodes and discover why the 'herd effect' matters.

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally using Docker, so there are no ongoing costs.

Resources you used:

Docker containers: zoo1, zoo2, zoo3 (running locally).
Project folder: zk-lab on your Desktop containing compose.yaml.
Docker image: zookeeper:3.9.5 downloaded to your machine.

Keep everything running

No action needed. Choose this if you are still actively experimenting with ZooKeeper. The ensemble uses minimal system resources while running.

Your three containers continue running in the background.
You can reconnect at any time with docker exec -it zoo1 zkCli.sh.
The zookeeper:3.9.5 image remains cached on your machine.

Pause - I'll come back to this later

Stop the containers to free up memory without removing them. Your data and configuration stay intact so you can resume later.

Open a terminal in your zk-lab folder and run:

docker compose stop

This stops all three containers without deleting them.
When you are ready to resume, run this from the same folder:

docker compose start

All three nodes come back up with your znodes intact.

Delete - I don't want to use this again

Remove all project resources and start fresh. This removes the containers, network, project folder, and downloaded image.

Open a terminal in your zk-lab folder and remove the containers and network:

docker compose down

Remove the Docker image to free disk space:

docker rmi zookeeper:3.9.5

Delete the project folder. On macOS or Linux:

cd ~/Desktop
rm -rf zk-lab

On Windows (PowerShell):

cd ~/Desktop
Remove-Item -Recurse -Force zk-lab

Confirm removal by running docker ps -a. You should see no zoo1, zoo2, or zoo3 containers listed.

Nice Work!

You just built and operated a 3-node ZooKeeper ensemble from scratch, explored its coordination primitives hands-on, and simulated real distributed systems scenarios like leader election and fault tolerance.

You've learned how to:

Deploy a 3-node ZooKeeper ensemble using Docker Compose and verify cluster health with a single command.
Implement the leader election recipe using ephemeral sequential nodes and observe automatic failover when the leader disappears.
Test ensemble fault tolerance by killing servers and proving the cluster survives as long as a quorum majority remains alive.
Secret Mission: Built a distributed lock and understood why watching only the predecessor node avoids the herd effect.

Ready to quiz yourself?