Docker Swarm Full-Stack Cluster Lab

Introduction

30 Second Summary

When you sit down for a distributed systems exam, the last thing you want is to encounter a technology for the first time. Having a battle-tested practice environment means you can focus on solving problems instead of fighting unfamiliar tooling.

In this project, you will build a complete Docker Swarm exam preparation lab with a 3-node cluster, reverse proxy, shared storage, message broker, and application patterns deployed declaratively across VMs.

What You'll Build

A full-stack distributed systems environment running on 3 VirtualBox VMs, covering every assessed topic from container orchestration to OAuth.

By the end of this project, you'll have:

• A 3-node Docker Swarm cluster with Traefik reverse proxy and MicroCeph shared storage fully configured.
• RabbitMQ messaging patterns implemented in Node.js, covering pub/sub, work queues, routing, topics, and RPC.
• Declarative stack files for every component, ready to deploy and adapt under exam conditions.
• Secret Mission: Configure a clustered 3-node RabbitMQ deployment with quorum queues for fault tolerance.

Are there any prerequisites?

You should have basic Docker knowledge (building images, running containers), familiarity with Linux CLI commands, and an understanding of networking concepts like ports and DNS.

Before We Start

Every topic in your exam runs on a 3-node VirtualBox cluster. Before you can practice any of them, you need working VMs with Docker installed and a reliable way to push files from your Windows host to each node.

In this step you will provision your 3 Ubuntu VMs, verify Docker Engine is ready on all of them, and set up the edit-locally-then-SCP workflow you will use for the rest of the project.

In this step, get ready to:

• Provision and SSH into 3 Ubuntu VMs (node1, node2, node3).
• Verify Docker Engine and the Docker Compose plugin are working on every node.
• Establish a local project directory and confirm SCP connectivity from Windows to each VM.

Run the VirtualBox VM provisioning script

The provisioning script creates 3 Ubuntu VMs inside VirtualBox and configures their networking so each VM is reachable from your Windows host via SSH.

• Run the provided automatic setup script to create the 3 VMs.
• Wait for all 3 VMs to finish booting. VirtualBox Manager should show them in a Running state.

Once the VMs are running, note each VM's IP address. You can find these inside each VM by running:

hostname -I

What does this command do?

`hostname -I` prints the IP addresses assigned to the VM's network interfaces. Note the address that your Windows host can reach (typically a host-only or bridged adapter IP).

• Record the IP for each VM. You will use these throughout the project. For example:
• node1: 192.168.56.101 (your actual IP will differ).
• node2: 192.168.56.102 (your actual IP will differ).
• node3: 192.168.56.103 (your actual IP will differ).

• Verify SSH access from your Windows terminal to each VM by running:

ssh [[VM_USER="your VM username"]]@[[NODE1_IP="node1 IP address"]]

Repeat for node2 and node3 using their respective IP addresses. You should land in a bash shell on each VM.

Can't SSH into a VM?

• Make sure the VM is fully booted. Give it 30-60 seconds after VirtualBox shows it running.
• Confirm the VM's network adapter is set to Host-Only or Bridged in VirtualBox settings so your Windows host can reach it.
• Check that the OpenSSH server is running inside the VM.

Help me troubleshoot SSH connectivity to my VirtualBox VM from Windows.

Verify Docker is installed and working

Docker Engine must be installed on every node before you can deploy Swarm services. You will check each VM individually.

• SSH into node1 and check the Docker Engine version by running:

docker --version

• Check the Docker Compose plugin by running:

docker compose version

• Verify your user is in the docker group by running:

groups

You should see docker listed among the groups. This allows you to run Docker commands without sudo.

✔️ I see version numbers and docker in my groups

Docker is ready on this node. Repeat the three checks on node2 and node3.

ⓧ Command not found or docker not in groups

Docker is not installed or your user is not in the docker group. Follow the steps below to install Docker Engine from the official apt repository and configure your user.

• Set up Docker's apt repository by running:

# Add Docker's official GPG key:
sudo apt update
sudo apt install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
sudo tee /etc/apt/sources.list.d/docker.sources <<EOF
Types: deb
URIs: https://download.docker.com/linux/ubuntu
Suites: $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}")
Components: stable
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/docker.asc
EOF

sudo apt update

• Install Docker Engine and all required packages by running:

sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

• Add your user to the docker group so you can run Docker without sudo:

sudo usermod -aG docker $USER
newgrp docker

• Verify the installation by running:

docker --version
docker compose version
groups

You should now see version numbers for both Docker and the Compose plugin, and docker listed in your groups output.

Repeat this entire installation process on each remaining VM.

Why does every node need Docker?

Docker Swarm runs containers across multiple machines. Every node in the cluster must have Docker Engine installed so it can receive and execute tasks scheduled by the swarm manager.

Set up the local development workflow

Throughout this project you will edit stack files on Windows and push them to the VMs via SCP. This avoids running a heavy editor inside the limited-RAM VMs.

• On your Windows machine, create a project directory for all your stack files. Open a terminal (PowerShell or Command Prompt) and run:

mkdir C:\swarm-lab
cd C:\swarm-lab

• Verify the directory was created by running:

dir

You should see an empty directory listing confirming C:\swarm-lab exists.

• Create a small test file and SCP it to node1 to confirm file transfer works:

echo "hello" > test.txt
scp test.txt [[VM_USER="your VM username"]]@[[NODE1_IP="node1 IP address"]]:~/

What does this SCP command do?

SCP (Secure Copy) transfers files over SSH. The command copies test.txt from your Windows machine to the home directory (~/) on node1. Windows 10 and later include an SCP client by default.

• Repeat the SCP test for node2 and node3 using their IP addresses.
• SSH into each node and confirm test.txt arrived by running:

ls ~/test.txt

You should see the file listed. This confirms your workflow: edit on Windows, SCP to VM, deploy on VM.

SCP connection refused?

• Ensure you can SSH to the same VM first. SCP uses SSH under the hood, so if SSH works, SCP should too.
• Double-check the IP address and username. A typo in either will cause a connection failure.
• If you are prompted to accept a host key fingerprint, type yes and press Enter.

Help me troubleshoot SCP from Windows to my Ubuntu VM.

Your 3-node foundation is in place. Next up, you will link these nodes together into a Docker Swarm cluster.

Initialize Docker Swarm Cluster

You have 3 Ubuntu VMs with Docker installed and ready to go. Right now, each VM runs Docker independently. To orchestrate containers across all 3 nodes, you need to connect them into a single Docker Swarm cluster.

A Swarm cluster has manager nodes (which handle scheduling and orchestration) and worker nodes (which run containers). You will designate node1 as the manager, then join node2 and node3 to the cluster.

In this step, get ready to:

• Initialize the Swarm on node1 as the manager node.
• Join node2 and node3 to the cluster.
• Verify all 3 nodes are active and overlay networking works.

Initialize Swarm on node1

The docker swarm init command converts a standalone Docker host into a Swarm manager. It generates two join tokens (one for workers, one for managers) that other nodes use to join the cluster.

• SSH into node1 from your Windows host.
• Initialize the Swarm by running this command (replace the IP with node1's actual IP address):

docker swarm init --advertise-addr [[NODE1_IP="IP address of node1"]]

What does this command do?

The --advertise-addr flag tells the Swarm which IP address to publish to other nodes. This is required when your VM has multiple network interfaces (which VirtualBox VMs typically do).

After running this, Docker prints a docker swarm join command containing the worker join token. Copy this entire command and save it somewhere. You will need it for node2 and node3.

• Copy the docker swarm join --token ... command from the output and save it in a text file on your Windows host.
• Retrieve the manager join token by running:

docker swarm join-token manager

• Copy and save the manager join command as well. You now have both tokens available for future use.

Why save both tokens?

The worker token lets nodes join as workers (they run containers but cannot manage the cluster). The manager token lets nodes join as managers (they participate in orchestration and Raft consensus).

For your exam, you may need either topology. Having both tokens saved means you can quickly switch between a 1-manager + 2-workers setup and a 3-manager setup.

Seeing an error about multiple IPs?

If you get an error mentioning multiple addresses, double-check that you included the correct --advertise-addr flag with node1's IP. VirtualBox VMs often have both a NAT adapter and a host-only adapter.

Use the host-only adapter IP (typically in the 192.168.x.x range) so other VMs can reach the manager.

Help me troubleshoot my docker swarm init error with multiple IP addresses

Join node2 and node3 to the swarm

With the manager running on node1, you can now add the other two nodes. Which token you use depends on the topology you want for your exam scenario.

Workers vs. managers: which topology?

For a 1-manager + 2-workers setup, use the worker join token on node2 and node3. This is the simpler topology and keeps management centralized on node1.

For a 3-manager setup, use the manager join token on both nodes. This gives you fault tolerance (the cluster survives if one manager goes down) and satisfies exam requirements that need Traefik on multiple managers. Docker recommends an odd number of managers for Raft quorum.

• Open a new terminal window and SSH into node2.
• Run the join command you saved earlier. For a worker join, it looks like this:

docker swarm join --token [[WORKER_TOKEN="your worker join token from the swarm init output"]] [[NODE1_IP="IP address of node1"]]:2377

You should see the message: This node joined a swarm as a worker. (or "as a manager" if you used the manager token).

• Open another terminal window and SSH into node3.
• Run the same join command on node3:

docker swarm join --token [[WORKER_TOKEN="your worker join token from the swarm init output"]] [[NODE1_IP="IP address of node1"]]:2377

You should see the same confirmation message on node3.

Node failed to join?

The Swarm uses three ports for communication: TCP 2377 (cluster management), TCP/UDP 7946 (node communication), and UDP 4789 (overlay network traffic). Make sure these ports are not blocked by a firewall on any of the VMs.

Also verify that node2 and node3 can reach node1's IP address. Try ping from node2 to node1's IP to confirm network connectivity.

Help me troubleshoot why my node cannot join the Docker Swarm cluster

Verify the cluster is healthy

All 3 nodes should now be part of the Swarm. You can only run cluster management commands on manager nodes, so make sure you run the next commands on node1.

• Switch back to your node1 SSH session.
• List all nodes in the cluster by running:

docker node ls

• Confirm all 3 nodes show STATUS: Ready and AVAILABILITY: Active in the output.

What should I see?

The output shows a table with columns for ID, HOSTNAME, STATUS, AVAILABILITY, and MANAGER STATUS. The node marked with * is the one you are currently connected to. Manager nodes show Leader or Reachable in the MANAGER STATUS column.

• Test that overlay networking works by creating an attachable overlay network:

docker network create --driver overlay --attachable test-overlay

• Verify the network was created by running:

docker network ls

You should see test-overlay listed with the overlay driver and swarm scope. You will also see the default ingress overlay network that Swarm created automatically during initialization.

Network creation failed?

Overlay networks can only be created on manager nodes. If you see a permissions error, confirm you are running the command on node1 (the manager), not on a worker node.

Help me fix my Docker overlay network creation error

Your 3-node Swarm cluster is live and communicating over overlay networks. Next up, you will deploy Traefik as a reverse proxy so services on the cluster are accessible via HTTP routing.

Deploy Traefik as Reverse Proxy

Your Swarm cluster is ready. Now you need a way to route incoming HTTP traffic to the correct service without manually exposing ports on every container.

Traefik is a reverse proxy that integrates directly with Docker Swarm. It reads labels from your services and automatically configures routing rules. In this step, you will deploy Traefik in global mode across all manager nodes and verify routing to backend services.

In this step, get ready to:

• Create and deploy a Traefik stack file with the Swarm provider configured.
• Verify host-based routing with a whoami test service.
• Add path-based routing for a second service and confirm load balancing.

Create the Traefik stack file

Traefik needs access to the Docker Swarm API so it can discover services and their labels. The Swarm API is only accessible on manager nodes, so Traefik must be constrained to run there.

• On your Windows machine, create a file called traefik-stack.yml in your project directory.
• Add the Traefik service definition by pasting the following into traefik-stack.yml:

version: "3.8"

services:
  # Traefik reverse proxy - runs on every manager node
  traefik:
    image: traefik:v3.7
    command:
      # Enable the Docker Swarm provider
      - "--providers.swarm.endpoint=unix:///var/run/docker.sock"
      # Watch for service changes in real time
      - "--providers.swarm.watch=true"
      # Only route services with traefik.enable=true
      - "--providers.swarm.exposedbydefault=false"
      # HTTP-only entrypoint on port 80
      - "--entrypoints.web.address=:80"
    ports:
      # Publish port 80 in host mode for direct access on each node
      - target: 80
        published: 80
        protocol: tcp
        mode: host
    volumes:
      # Mount the Docker socket so Traefik can read Swarm state
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      - traefik_proxy
    deploy:
      # Global mode runs one instance on every matching node
      mode: global
      placement:
        constraints:
          - node.role == manager

networks:
  traefik_proxy:
    external: true

What does this configuration do?

The --providers.swarm.endpoint flag tells Traefik to read service metadata from the Docker socket. The --providers.swarm.exposedbydefault=false setting means Traefik ignores services unless they explicitly opt in with a traefik.enable=true label.

Global mode with node.role == manager places one Traefik container on every manager node. Publishing port 80 in host mode binds directly to the node's IP, making Traefik reachable at any manager's address.

The traefik_proxy network is marked external: true because you will create it manually before deploying. This keeps the network name predictable across stack deployments.

Why global mode instead of replicated?

The exam requires Traefik to be active on multiple nodes. Global mode guarantees exactly one instance per matching node. Combined with host-mode port publishing, each manager's IP becomes a direct entry point to the cluster.

Replicated mode with a single replica would leave a single point of failure. Global mode on managers gives you high availability without manual scaling.

Deploy and verify Traefik

With the stack file ready, you will SCP it to node1, create the overlay network, and deploy the stack. Then you will add a whoami test service to confirm routing works end to end.

• SCP the stack file from your Windows project directory to node1 by running this command in your local terminal:

scp traefik-stack.yml [[VMUSER="your-vm-username"]]@[[NODE1_IP="node1 IP address"]]:~/

• SSH into node1 (if not already connected).
• Create the overlay network by running this command on node1:

docker network create --driver overlay --attachable traefik_proxy

• Deploy the Traefik stack by running:

docker stack deploy -c traefik-stack.yml traefik

• Verify Traefik is running on all manager nodes by running:

docker service ls

You should see a service called traefik_traefik with the MODE column showing global.

• Check which nodes Traefik is running on by running:

docker service ps traefik_traefik

You should see one task per manager node, each with STATE: Running.

Traefik service showing 0/1 replicas?

Check the service logs for errors:

docker service logs traefik_traefik

• If you see a network error, make sure you created the traefik_proxy network before deploying the stack.
• If you see a socket permission error, verify the Docker socket is accessible at /var/run/docker.sock on the node.

Help me troubleshoot why my Traefik service isn't starting in Docker Swarm.

Now add a whoami test service to the stack file to verify Traefik routes traffic correctly.

• Open traefik-stack.yml on your Windows machine and add the following whoami service below the traefik service (inside the services: block, before the networks: section):

  # Test service for verifying Traefik routing
  whoami:
    image: traefik/whoami
    networks:
      - traefik_proxy
    deploy:
      replicas: 2
      labels:
        # Opt in to Traefik service discovery
        - "traefik.enable=true"
        # Route requests with Host header matching whoami.local
        - "traefik.http.routers.whoami.rule=Host(`whoami.local`)"
        # Use the HTTP entrypoint
        - "traefik.http.routers.whoami.entrypoints=web"
        # Tell Traefik which port the service listens on (required in Swarm)
        - "traefik.http.services.whoami.loadbalancer.server.port=80"

Why are labels inside the deploy section?

In Swarm mode, Traefik reads labels from services, not containers. Labels placed under deploy: are service-level labels visible to Traefik's Swarm provider.

The traefik.http.services.whoami.loadbalancer.server.port=80 label is mandatory in Swarm because Docker Swarm does not provide port detection to Traefik.

• SCP the updated file to node1 and redeploy the stack:

docker stack deploy -c traefik-stack.yml traefik

• Test that Traefik routes to the whoami service by running:

curl -H "Host: whoami.local" http://localhost/

You should see output from the whoami container showing the hostname, IP addresses, and HTTP headers. The Hostname: line confirms which container handled the request.

Getting a 404 or empty response?

• Verify the whoami service is running: docker service ls should show traefik_whoami with 2/2 replicas.
• Check that the Host header in your curl command exactly matches the rule: whoami.local.
• Ensure the whoami service is on the same traefik_proxy network as Traefik.

Why is Traefik returning 404 for my whoami service?

Add a second service with path-based routing

Host-based routing matches the entire domain. Path-based routing lets you direct different URL paths to different backend services on the same host.

• Open traefik-stack.yml and add the whoami-api service below the whoami service (still inside services:):

  # Second test service with path-based routing
  whoami-api:
    image: traefik/whoami
    networks:
      - traefik_proxy
    environment:
      - WHOAMI_NAME=API Service
    deploy:
      replicas: 2
      labels:
        - "traefik.enable=true"
        # Route requests where the path starts with /api
        - "traefik.http.routers.whoami-api.rule=PathPrefix(`/api`)"
        - "traefik.http.routers.whoami-api.entrypoints=web"
        - "traefik.http.services.whoami-api.loadbalancer.server.port=80"

What does PathPrefix do?

The PathPrefix(`/api`) rule matches any request whose URL path starts with /api. This is useful for separating API backends from frontend services behind the same reverse proxy.

The WHOAMI_NAME=API Service environment variable makes it easy to identify which service responded when you test with curl.

• SCP the updated file to node1 and redeploy:

docker stack deploy -c traefik-stack.yml traefik

• Verify that the root path routes to the whoami service:

curl -H "Host: whoami.local" http://localhost/

• Verify that /api routes to the whoami-api service:

curl http://localhost/api

The response should include Name: API Service confirming Traefik directed the request to the whoami-api backend.

• Confirm Traefik is load-balancing across replicas by running the curl command multiple times:

curl http://localhost/api
curl http://localhost/api
curl http://localhost/api

Each response should show a different Hostname: value as Traefik distributes requests across the 2 replicas.

Both paths returning the same service?

• Check that the router names are unique. whoami and whoami-api must be distinct in the label names.
• Verify the stack redeployed successfully: docker service ls should show both traefik_whoami and traefik_whoami-api services.
• Confirm labels have the correct backtick syntax around the path value.

Help me debug path-based routing in Traefik on Swarm.

✔️ Awesome, I've got everything!

Great. Make sure you have saved your final traefik-stack.yml with all three services (traefik, whoami, whoami-api).

ⓧ I'd like to double check the full code

Here is the complete traefik-stack.yml with all three services:

version: "3.8"

services:
  # Traefik reverse proxy - runs on every manager node
  traefik:
    image: traefik:v3.7
    command:
      # Enable the Docker Swarm provider
      - "--providers.swarm.endpoint=unix:///var/run/docker.sock"
      # Watch for service changes in real time
      - "--providers.swarm.watch=true"
      # Only route services with traefik.enable=true
      - "--providers.swarm.exposedbydefault=false"
      # HTTP-only entrypoint on port 80
      - "--entrypoints.web.address=:80"
    ports:
      # Publish port 80 in host mode for direct access on each node
      - target: 80
        published: 80
        protocol: tcp
        mode: host
    volumes:
      # Mount the Docker socket so Traefik can read Swarm state
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      - traefik_proxy
    deploy:
      # Global mode runs one instance on every matching node
      mode: global
      placement:
        constraints:
          - node.role == manager

  # Test service for verifying Traefik routing
  whoami:
    image: traefik/whoami
    networks:
      - traefik_proxy
    deploy:
      replicas: 2
      labels:
        # Opt in to Traefik service discovery
        - "traefik.enable=true"
        # Route requests with Host header matching whoami.local
        - "traefik.http.routers.whoami.rule=Host(`whoami.local`)"
        # Use the HTTP entrypoint
        - "traefik.http.routers.whoami.entrypoints=web"
        # Tell Traefik which port the service listens on (required in Swarm)
        - "traefik.http.services.whoami.loadbalancer.server.port=80"

  # Second test service with path-based routing
  whoami-api:
    image: traefik/whoami
    networks:
      - traefik_proxy
    environment:
      - WHOAMI_NAME=API Service
    deploy:
      replicas: 2
      labels:
        - "traefik.enable=true"
        # Route requests where the path starts with /api
        - "traefik.http.routers.whoami-api.rule=PathPrefix(`/api`)"
        - "traefik.http.routers.whoami-api.entrypoints=web"
        - "traefik.http.services.whoami-api.loadbalancer.server.port=80"

networks:
  traefik_proxy:
    external: true

Your reverse proxy is live. Traefik is routing HTTP traffic across your Swarm cluster using both host-based and path-based rules. Next up, you will set up MicroCeph to give your cluster shared storage that any node can read from and write to.

Set Up MicroCeph Shared Storage

Your Swarm cluster can schedule services on any node, but containers on different nodes have no way to access the same files.

MicroCeph solves this by creating a distributed storage cluster across all 3 VMs. Once mounted, every node sees the same /mnt/shared directory. Any file written on one node instantly appears on the others.

In this step, get ready to:

• Install MicroCeph on all 3 nodes and hold auto-updates.
• Bootstrap a 3-node Ceph cluster with join tokens.
• Create and mount a CephFS shared filesystem on every node.

Install MicroCeph on all 3 nodes

MicroCeph is distributed as a snap package. You need to install it on every node that will participate in the storage cluster.

• On node1 (where you already are), install MicroCeph by running:

sudo snap install microceph

• Prevent auto-updates from breaking your cluster by running:

sudo snap refresh --hold microceph

Why hold auto-updates?

Snap packages auto-update by default. In a storage cluster, an unexpected version change on one node can break replication or cause data inconsistency. Holding the snap keeps all 3 nodes on the same version until you explicitly upgrade.

• Open a new terminal and SSH into node2.
• Run the same two commands on node2:

sudo snap install microceph
sudo snap refresh --hold microceph

• Open another terminal and SSH into node3.
• Run the same two commands on node3:

sudo snap install microceph
sudo snap refresh --hold microceph

• Verify the install succeeded on any node by running:

snap list microceph

You should see a line showing the microceph snap with its version and revision.

Snap install failed?

Make sure the VM has internet access. Try ping snapcraft.io to check connectivity.

If you see a permission error, confirm you are using sudo before the command.

Help me troubleshoot my MicroCeph snap installation failure.

Bootstrap the cluster and add nodes

With MicroCeph installed on all 3 VMs, you can form them into a single storage cluster. One node bootstraps (creates) the cluster, then generates unique join tokens for the other nodes.

• Switch back to your node1 terminal.
• Bootstrap the MicroCeph cluster by running:

sudo microceph cluster bootstrap

What does this command do?

This initializes node1 as the first member of a new Ceph cluster. It starts the monitor, manager, and metadata services on this node.

• Generate a join token for node2 by running:

sudo microceph cluster add node-2

• Copy the token that appears in the output. You will need it shortly.
• Generate a join token for node3 by running:

sudo microceph cluster add node-3

• Copy this token as well.

Each token is unique

The tokens are long base64 strings. Each one is unique and can only be used once. Paste them carefully when joining nodes.

• Switch to your node2 terminal.
• Join node2 to the cluster by running this command with the token you generated for node-2:

sudo microceph cluster join [[NODE2_TOKEN="paste the token generated for node-2"]]

• Switch to your node3 terminal.
• Join node3 to the cluster by running this command with the token you generated for node-3:

sudo microceph cluster join [[NODE3_TOKEN="paste the token generated for node-3"]]

• Switch back to your node1 terminal.
• Verify all 3 nodes are in the cluster by running:

sudo microceph.ceph status

You should see mon: 3 daemons in the services section, confirming all 3 nodes are clustered.

Join command failed?

Make sure you are pasting the correct token for the correct node. Each token is tied to a specific node name.

If the token has expired, generate a new one on node1 with sudo microceph cluster add <node-name> and try again.

Help me troubleshoot my MicroCeph cluster join failure.

Add storage and mount CephFS

The cluster is formed but has no storage yet. You need to give each node disk space for Ceph to manage. For VMs without physical disks, MicroCeph can create file-backed loop devices instead.

• On node1, add loop-backed storage by running:

sudo microceph disk add loop,4G,3

What does this command do?

This creates 3 file-backed 4GB OSDs (Object Storage Daemons) on this node. Each OSD acts as a virtual disk that Ceph uses to store and replicate data.

• Switch to your node2 terminal and run the same command:

sudo microceph disk add loop,4G,3

• Switch to your node3 terminal and run the same command:

sudo microceph disk add loop,4G,3

Physical disks work too

If your VMs have dedicated physical disks available, you can use those instead with sudo microceph disk add /dev/vdb --wipe. The loop approach is simpler for lab environments.

Now create the CephFS filesystem. This only needs to run once, on any node in the cluster.

• Switch back to your node1 terminal.
• Create the metadata pool by running:

sudo ceph osd pool create cephfs_meta

• Create the data pool by running:

sudo ceph osd pool create cephfs_data

• Create the CephFS filesystem using both pools by running:

sudo ceph fs new sharedfs cephfs_meta cephfs_data

What are these pools?

CephFS requires two pools. The metadata pool (cephfs_meta) stores file names, directory structures, and permissions. The data pool (cephfs_data) stores the actual file contents.

MicroCeph configures 3-way replication by default. Every file you write to this filesystem is automatically copied across all 3 nodes.

Before you can mount the filesystem, each node needs the ceph-common package and symlinks to the cluster configuration.

• On node1, install the Ceph client tools and set up symlinks by running these commands:

sudo apt install ceph-common -y
sudo ln -s /var/snap/microceph/current/conf/ceph.keyring /etc/ceph/ceph.keyring
sudo ln -s /var/snap/microceph/current/conf/ceph.conf /etc/ceph/ceph.conf

Why create symlinks?

The kernel CephFS driver looks for cluster credentials in /etc/ceph/ by default. MicroCeph stores its config under the snap directory. The symlinks connect the two so the mount command can find the authentication keys.

• Still on node1, create the mount point and mount the filesystem by running:

sudo mkdir -p /mnt/shared && sudo mount -t ceph :/ /mnt/shared/ -o name=admin,fs=sharedfs

• Switch to your node2 terminal and run the full setup:

sudo apt install ceph-common -y
sudo ln -s /var/snap/microceph/current/conf/ceph.keyring /etc/ceph/ceph.keyring
sudo ln -s /var/snap/microceph/current/conf/ceph.conf /etc/ceph/ceph.conf
sudo mkdir -p /mnt/shared && sudo mount -t ceph :/ /mnt/shared/ -o name=admin,fs=sharedfs

• Switch to your node3 terminal and run the same commands:

sudo apt install ceph-common -y
sudo ln -s /var/snap/microceph/current/conf/ceph.keyring /etc/ceph/ceph.keyring
sudo ln -s /var/snap/microceph/current/conf/ceph.conf /etc/ceph/ceph.conf
sudo mkdir -p /mnt/shared && sudo mount -t ceph :/ /mnt/shared/ -o name=admin,fs=sharedfs

Now verify that the shared filesystem works across all nodes.

• On node1, create a test file by running:

sudo touch /mnt/shared/testfile.txt

• On node2 or node3, confirm the file is visible by running:

ls /mnt/shared/

You should see testfile.txt listed. This confirms that all 3 nodes share the same filesystem.

• Confirm the overall cluster health from node1 by running:

sudo microceph.ceph status

You should see health: HEALTH_OK with 3 monitors, 3 managers, and 9 OSDs (3 per node) listed under services.

Mount command failed?

If you see an error about missing mount.ceph, make sure you installed ceph-common on that node.

If you see a permission or key error, verify the symlinks exist by running ls -la /etc/ceph/. You should see both ceph.conf and ceph.keyring pointing to the MicroCeph snap directory.

Help me troubleshoot my CephFS mount failure.

Your shared storage layer is live. Every node in the Swarm can now read and write to the same /mnt/shared directory. Next up, you will deploy RabbitMQ on the cluster and practice all 6 messaging patterns.

Deploy RabbitMQ and Practice Messaging Patterns

Your cluster now has shared storage across all three nodes. The next piece of infrastructure is a message broker that lets services communicate asynchronously.

In this step, you will deploy RabbitMQ as a Swarm service and implement all six official messaging patterns using Node.js. This gives you a working reference for every pattern the exam may test.

In this step, get ready to:

• Deploy RabbitMQ with a declarative stack file.
• Create Node.js scripts for all 6 RabbitMQ messaging patterns.
• Make the entire deployment reproducible with a single command.

Deploy RabbitMQ as a Swarm service

RabbitMQ needs an explicit hostname so its data directory stays stable across container restarts. You will create a stack file that uses the official rabbitmq:4-management image, which bundles the management UI on port 15672.

• On your Windows machine, create a file called rabbitmq-stack.yml in your project directory (alongside traefik-stack.yml) and paste the following:

version: '3.8'

services:
  rabbitmq:
    image: rabbitmq:4-management
    hostname: rabbitmq
    ports:
      - "5672:5672"    # AMQP protocol port
      - "15672:15672"  # Management UI port
    configs:
      - source: rabbitmq_conf
        target: /etc/rabbitmq/rabbitmq.conf
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure

configs:
  rabbitmq_conf:
    file: ./rabbitmq.conf

What does this stack file do?

The hostname: rabbitmq line ensures the container's data directory is always named consistently. Without it, RabbitMQ creates a new database every time the container restarts.

The configs section mounts an external configuration file into the container at /etc/rabbitmq/rabbitmq.conf. This makes configuration declarative and version-controlled.

• Create a rabbitmq.conf file in the same project directory with the following content:

# RabbitMQ configuration file
# Enable the default guest user for local access
loopback_users.guest = false

Why disable loopback for guest?

By default, the guest user can only connect from localhost. Since you are connecting from outside the container (from your VM or Windows host), setting loopback_users.guest = false allows guest access from any IP. This is acceptable in a lab environment.

• SCP both files to node1 by running these commands from your Windows terminal:

scp rabbitmq-stack.yml [[VMUSER="your-vm-username"]]@[[NODE1_IP="node1 IP address"]]:~/
scp rabbitmq.conf [[VMUSER="your-vm-username"]]@[[NODE1_IP="node1 IP address"]]:~/

• Switch back to your SSH session on node1 and deploy the stack by running:

docker stack deploy -c rabbitmq-stack.yml messaging

• Verify RabbitMQ is running by checking the service status:

docker service ls

You should see a service called messaging_rabbitmq with 1/1 replicas.

• Test the management UI by opening your browser and navigating to http://<NODE-IP>:15672. Log in with username guest and password guest.

Management UI not loading?

RabbitMQ can take 15-30 seconds to start. Run docker service ps messaging_rabbitmq to check the task state. If it shows Running but the UI still does not load, check that port 15672 is not blocked by a firewall on the VM.

Help me debug my RabbitMQ management UI not loading

Create Node.js messaging scripts

RabbitMQ supports six core messaging patterns. You will implement each one using the amqplib library so you have a working reference for the exam.

• On your Windows machine, create a new folder called messaging inside your project directory.
• Open a terminal in the messaging folder and initialize the Node.js project by running:

npm init -y

• Install the amqplib library by running:

npm install amqplib

What is amqplib?

amqplib is the official AMQP 0-9-1 client library for Node.js. It provides both a callback-based and a Promise/async API for connecting to RabbitMQ. All six official RabbitMQ tutorials use this library.

Now create each of the six pattern scripts. In the code below, replace amqp://localhost with amqp://<NODE-IP> when running from Windows, or leave as-is if running directly on the VM.

Pattern 1: Hello World (send.js and receive.js)

• Create send.js with the following content:

#!/usr/bin/env node
const amqp = require('amqplib');

async function main() {
  // Connect to RabbitMQ
  const connection = await amqp.connect('amqp://localhost');
  const channel = await connection.createChannel();

  const queue = 'hello';
  const msg = 'Hello World!';

  // Declare a durable quorum queue
  await channel.assertQueue(queue, {
    durable: true,
    arguments: { 'x-queue-type': 'quorum' }
  });

  // Send the message
  channel.sendToQueue(queue, Buffer.from(msg));
  console.log(" [x] Sent %s", msg);

  // Close connection after a short delay
  setTimeout(function() {
    connection.close();
    process.exit(0);
  }, 500);
}

main();

• Create receive.js with the following content:

#!/usr/bin/env node
const amqp = require('amqplib');

async function main() {
  // Connect to RabbitMQ
  const connection = await amqp.connect('amqp://localhost');
  const channel = await connection.createChannel();

  const queue = 'hello';

  // Declare the same queue the sender uses
  await channel.assertQueue(queue, {
    durable: true,
    arguments: { 'x-queue-type': 'quorum' }
  });

  console.log(" [*] Waiting for messages in %s. To exit press CTRL+C", queue);

  // Consume messages as they arrive
  channel.consume(queue, function(msg) {
    console.log(" [x] Received %s", msg.content.toString());
  }, { noAck: true });
}

main();

What does this pattern demonstrate?

Hello World is the simplest pattern. One producer sends a message to a named queue and one consumer reads from it. The queue is declared with x-queue-type: quorum for durability across RabbitMQ node failures.

Pattern 2: Work Queues (new_task.js and worker.js)

• Create new_task.js with the following content:

#!/usr/bin/env node
const amqp = require('amqplib');

async function main() {
  const connection = await amqp.connect('amqp://localhost');
  const channel = await connection.createChannel();

  const queue = 'task_queue';
  // Use command line args as the message, dots simulate work duration
  const msg = process.argv.slice(2).join(' ') || "Hello World!";

  // Durable quorum queue survives broker restarts
  await channel.assertQueue(queue, {
    durable: true,
    arguments: { 'x-queue-type': 'quorum' }
  });

  // Persistent messages are written to disk
  channel.sendToQueue(queue, Buffer.from(msg), { persistent: true });
  console.log(" [x] Sent '%s'", msg);

  setTimeout(function() {
    connection.close();
    process.exit(0);
  }, 500);
}

main();

• Create worker.js with the following content:

#!/usr/bin/env node
const amqp = require('amqplib');

async function main() {
  const connection = await amqp.connect('amqp://localhost');
  const channel = await connection.createChannel();

  const queue = 'task_queue';

  await channel.assertQueue(queue, {
    durable: true,
    arguments: { 'x-queue-type': 'quorum' }
  });

  // Only process one message at a time for fair dispatch
  channel.prefetch(1);
  console.log(" [*] Waiting for messages in %s. To exit press CTRL+C", queue);

  channel.consume(queue, function(msg) {
    // Each dot in the message simulates 1 second of work
    const secs = msg.content.toString().split('.').length - 1;
    console.log(" [x] Received %s", msg.content.toString());

    setTimeout(function() {
      console.log(" [x] Done");
      channel.ack(msg);
    }, secs * 1000);
  }, { noAck: false });
}

main();

What does this pattern demonstrate?

Work Queues distribute tasks among multiple workers using round-robin dispatching. The prefetch(1) setting ensures fair dispatch. Manual acknowledgements (noAck: false) prevent message loss if a worker crashes mid-task.

Pattern 3: Publish/Subscribe (emit_log.js and receive_logs.js)

• Create emit_log.js with the following content:

#!/usr/bin/env node
const amqp = require('amqplib');

async function main() {
  const connection = await amqp.connect('amqp://localhost');
  const channel = await connection.createChannel();

  const exchange = 'logs';
  const msg = process.argv.slice(2).join(' ') || 'Hello World!';

  // Declare a fanout exchange that broadcasts to all bound queues
  await channel.assertExchange(exchange, 'fanout', { durable: false });

  // Publish to the exchange with empty routing key (fanout ignores it)
  channel.publish(exchange, '', Buffer.from(msg));
  console.log(" [x] Sent %s", msg);

  setTimeout(function() {
    connection.close();
    process.exit(0);
  }, 500);
}

main();

• Create receive_logs.js with the following content:

#!/usr/bin/env node
const amqp = require('amqplib');

async function main() {
  const connection = await amqp.connect('amqp://localhost');
  const channel = await connection.createChannel();

  const exchange = 'logs';

  await channel.assertExchange(exchange, 'fanout', { durable: false });

  // Create a temporary exclusive queue with a server-generated name
  const q = await channel.assertQueue('', { exclusive: true });
  console.log(" [*] Waiting for messages in %s. To exit press CTRL+C", q.queue);

  // Bind our queue to the fanout exchange
  channel.bindQueue(q.queue, exchange, '');

  channel.consume(q.queue, function(msg) {
    if (msg.content) {
      console.log(" [x] %s", msg.content.toString());
    }
  }, { noAck: true });
}

main();

What does this pattern demonstrate?

Publish/Subscribe uses a fanout exchange to broadcast messages to all bound queues. Every running consumer gets a copy of every message. This is useful for logging systems or notifications.

Pattern 4: Routing (emit_log_direct.js and receive_logs_direct.js)

• Create emit_log_direct.js with the following content:

#!/usr/bin/env node
const amqp = require('amqplib');

async function main() {
  const connection = await amqp.connect('amqp://localhost');
  const channel = await connection.createChannel();

  const exchange = 'direct_logs';
  const args = process.argv.slice(2);
  const msg = args.slice(1).join(' ') || 'Hello World!';
  // First argument is the severity (routing key)
  const severity = (args.length > 0) ? args[0] : 'info';

  // Declare a direct exchange for severity-based routing
  await channel.assertExchange(exchange, 'direct', { durable: false });

  // Publish with the severity as the routing key
  channel.publish(exchange, severity, Buffer.from(msg));
  console.log(" [x] Sent %s: '%s'", severity, msg);

  setTimeout(function() {
    connection.close();
    process.exit(0);
  }, 500);
}

main();

• Create receive_logs_direct.js with the following content:

#!/usr/bin/env node
const amqp = require('amqplib');

const args = process.argv.slice(2);

if (args.length === 0) {
  console.log("Usage: receive_logs_direct.js [info] [warning] [error]");
  process.exit(1);
}

async function main() {
  const connection = await amqp.connect('amqp://localhost');
  const channel = await connection.createChannel();

  const exchange = 'direct_logs';

  await channel.assertExchange(exchange, 'direct', { durable: false });

  // Create a temporary exclusive queue
  const q = await channel.assertQueue('', { exclusive: true });
  console.log(' [*] Waiting for logs. To exit press CTRL+C');

  // Bind the queue for each severity level passed as argument
  for (const severity of args) {
    await channel.bindQueue(q.queue, exchange, severity);
  }

  channel.consume(q.queue, function(msg) {
    console.log(" [x] %s: '%s'", msg.fields.routingKey, msg.content.toString());
  }, { noAck: true });
}

main();

What does this pattern demonstrate?

Routing uses a direct exchange where messages are routed to queues based on an exact match between the routing key (severity) and the binding key. Consumers can subscribe to specific severities only.

Pattern 5: Topics (emit_log_topic.js and receive_logs_topic.js)

• Create emit_log_topic.js with the following content:

#!/usr/bin/env node
const amqp = require('amqplib');

async function main() {
  const connection = await amqp.connect('amqp://localhost');
  const channel = await connection.createChannel();

  const exchange = 'topic_logs';
  const args = process.argv.slice(2);
  // First argument is the routing key (e.g. kern.critical)
  const key = (args.length > 0) ? args[0] : 'anonymous.info';
  const msg = args.slice(1).join(' ') || 'Hello World!';

  // Declare a topic exchange for wildcard-based routing
  await channel.assertExchange(exchange, 'topic', { durable: false });

  channel.publish(exchange, key, Buffer.from(msg));
  console.log(" [x] Sent %s: '%s'", key, msg);

  setTimeout(function() {
    connection.close();
    process.exit(0);
  }, 500);
}

main();

• Create receive_logs_topic.js with the following content:

#!/usr/bin/env node
const amqp = require('amqplib');

const args = process.argv.slice(2);

if (args.length === 0) {
  console.log("Usage: receive_logs_topic.js <facility>.<severity>");
  process.exit(1);
}

async function main() {
  const connection = await amqp.connect('amqp://localhost');
  const channel = await connection.createChannel();

  const exchange = 'topic_logs';

  await channel.assertExchange(exchange, 'topic', { durable: false });

  // Create a temporary exclusive queue
  const q = await channel.assertQueue('', { exclusive: true });
  console.log(' [*] Waiting for logs. To exit press CTRL+C');

  // Bind with each pattern (supports * and # wildcards)
  for (const key of args) {
    await channel.bindQueue(q.queue, exchange, key);
  }

  channel.consume(q.queue, function(msg) {
    console.log(" [x] %s:'%s'", msg.fields.routingKey, msg.content.toString());
  }, { noAck: true });
}

main();

What does this pattern demonstrate?

Topics extend routing with wildcard patterns. The * character matches exactly one word and # matches zero or more words. This allows consumers to subscribe to subsets like kern.* or *.critical.

Pattern 6: RPC (rpc_server.js and rpc_client.js)

• Create rpc_server.js with the following content:

#!/usr/bin/env node
const amqp = require('amqplib');

async function main() {
  const connection = await amqp.connect('amqp://localhost');
  const channel = await connection.createChannel();

  const queue = 'rpc_queue';

  await channel.assertQueue(queue, {
    durable: true,
    arguments: { 'x-queue-type': 'quorum' }
  });

  // Process one request at a time
  channel.prefetch(1);
  console.log(' [x] Awaiting RPC requests');

  channel.consume(queue, function reply(msg) {
    const n = parseInt(msg.content.toString());
    console.log(" [.] fib(%d)", n);

    const r = fibonacci(n);

    // Send the result back to the reply-to queue with matching correlationId
    channel.sendToQueue(msg.properties.replyTo,
      Buffer.from(r.toString()), {
        correlationId: msg.properties.correlationId
      });

    channel.ack(msg);
  });
}

// Simple recursive Fibonacci (for demonstration only)
function fibonacci(n) {
  if (n === 0 || n === 1)
    return n;
  else
    return fibonacci(n - 1) + fibonacci(n - 2);
}

main();

• Create rpc_client.js with the following content:

#!/usr/bin/env node
const amqp = require('amqplib');

const args = process.argv.slice(2);

if (args.length === 0) {
  console.log("Usage: rpc_client.js num");
  process.exit(1);
}

async function main() {
  const connection = await amqp.connect('amqp://localhost');
  const channel = await connection.createChannel();

  const correlationId = generateUuid();
  const num = parseInt(args[0]);

  console.log(' [x] Requesting fib(%d)', num);

  const result = await new Promise((resolve) => {
    // Consume from Direct Reply-to pseudo-queue (noAck required)
    channel.consume('amq.rabbitmq.reply-to', (msg) => {
      if (msg.properties.correlationId === correlationId) {
        resolve(msg.content.toString());
      }
    }, { noAck: true });

    // Send the RPC request
    channel.sendToQueue('rpc_queue',
      Buffer.from(num.toString()), {
        correlationId: correlationId,
        replyTo: 'amq.rabbitmq.reply-to'
      });
  });

  console.log(' [.] Got %s', result);
  await connection.close();
}

function generateUuid() {
  return Math.random().toString() +
         Math.random().toString() +
         Math.random().toString();
}

main();

What does this pattern demonstrate?

RPC uses a request-reply pattern. The client sends a request with a correlationId and a replyTo address. The server processes the request and sends the result back. The client uses amq.rabbitmq.reply-to (Direct Reply-to) for efficient response routing without creating a temporary queue.

• SCP all the scripts to node1 by running this from your Windows terminal:

scp send.js receive.js new_task.js worker.js emit_log.js receive_logs.js emit_log_direct.js receive_logs_direct.js emit_log_topic.js receive_logs_topic.js rpc_server.js rpc_client.js package.json [[VMUSER="your-vm-username"]]@[[NODE1_IP="node1 IP address"]]:~/messaging/

• SSH into node1 and install the dependencies on the VM by running:

cd ~/messaging && npm install

• Test the Hello World pattern by running send.js followed by receive.js:

node send.js

You should see [x] Sent Hello World! in the output.

node receive.js

You should see [x] Received Hello World! in the output. Press Ctrl+C to stop the consumer.

• Test the RPC pattern by running the server in one terminal and the client in another:

node rpc_server.js

node rpc_client.js 30

The client should output the 30th Fibonacci number (832040).

Connection refused when running scripts?

If you see an ECONNREFUSED error, the scripts are trying to connect to localhost:5672 but RabbitMQ may be running on a different node in the Swarm. Check which node the messaging_rabbitmq task is scheduled on with docker service ps messaging_rabbitmq. If it is running on the same node, wait a few seconds for RabbitMQ to finish starting.

Help me fix ECONNREFUSED when connecting to RabbitMQ

Make the deployment declarative

The exam requires that you can tear down and recreate your entire RabbitMQ setup with a single command. Your stack file already handles this. Let's verify by removing and redeploying.

• Remove the existing messaging stack by running:

docker stack rm messaging

• Wait a few seconds for the services to shut down, then redeploy by running:

docker stack deploy -c rabbitmq-stack.yml messaging

• Verify the service is back up by running:

docker service ls

You should see messaging_rabbitmq with 1/1 replicas again. Wait 15-20 seconds for RabbitMQ to initialize, then test again:

node ~/messaging/send.js

If you see [x] Sent Hello World! without errors, your deployment is fully declarative and reproducible.

Why is declarative deployment important for the exam?

Under exam conditions you may need to tear down and rebuild services quickly. A declarative stack file means you only need to run one docker stack deploy command to bring everything back. No manual configuration steps means fewer mistakes under time pressure.

Stack redeploy not working?

If docker stack deploy shows an error about the config already existing, wait 10 more seconds after docker stack rm for Docker to fully remove the old resources. Then try deploying again.

Help me fix my messaging stack redeploy error

✔️ Awesome, I've got everything!

Great. Make sure you have saved all 12 script files and that both rabbitmq-stack.yml and rabbitmq.conf are in your project directory.

ⓧ I'd like to double check the full code

Here is the full rabbitmq-stack.yml for reference:

version: '3.8'

services:
  rabbitmq:
    image: rabbitmq:4-management
    hostname: rabbitmq
    ports:
      - "5672:5672"    # AMQP protocol port
      - "15672:15672"  # Management UI port
    configs:
      - source: rabbitmq_conf
        target: /etc/rabbitmq/rabbitmq.conf
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure

configs:
  rabbitmq_conf:
    file: ./rabbitmq.conf

And rabbitmq.conf:

# RabbitMQ configuration file
# Enable the default guest user for local access
loopback_users.guest = false

Your message broker is deployed and all six patterns are tested. Next up, you will build a microkernel plugin system to cover the software architecture exam topic.

Build a Microkernel Plugin System

Your cluster now handles messaging with RabbitMQ. The next exam topic is the microkernel architecture pattern, where a minimal core exposes hooks that plugins can register against to extend functionality.

In this step, you will build a calculator application that demonstrates the pattern. A tiny core manages a registry of operations, and each plugin installs itself by registering a handler. You will containerize the app and deploy it behind Traefik on your Swarm cluster.

In this step, get ready to:

• Understand the microkernel/plugin architecture pattern.
• Build a calculator core with arithmetic plugins.
• Containerize and deploy the plugin system behind Traefik.

Understand the microkernel plugin pattern

The exam may hand you a base application and ask you to add plugin support. Understanding the core pattern beforehand saves you from designing it under pressure.

What is a microkernel architecture?

A microkernel architecture separates a system into a small, stable core and a set of interchangeable plugins. The core provides a registry and lifecycle hooks. Plugins register themselves through a well-known interface without modifying the core.

The exam example might be a spreadsheet app where formula plugins (SUM, AVERAGE) plug in, or a calculator where arithmetic operations are plugins. The pattern is the same regardless of domain.

• The core exposes a registerOperation(name, handler) method.
• Each plugin exports an install(core) function that calls registerOperation to add its handler.
• The core never knows about specific operations. New capabilities come from loading new plugins.

Build the plugin system skeleton

You will create four files: the core module, two plugin modules, and an HTTP server that ties everything together. Then you will containerize the app and deploy it as a Swarm service.

• Create the project directory on node1 by running these commands:

mkdir -p ~/plugin-system/plugins
cd ~/plugin-system

What does this command do?

This creates a plugin-system directory with a plugins subdirectory inside it. The plugins folder holds each plugin as its own module file.

• Create core.js by running:

cat > ~/plugin-system/core.js << 'EOF'
// Microkernel core: plugin registry and operation dispatch
class Core {
  constructor() {
    this.plugins = {};
    this.operations = {};
  }

  // Register a named operation handler
  registerOperation(name, handler) {
    this.operations[name] = handler;
    console.log(`Operation registered: ${name}`);
  }

  // Execute a registered operation by name
  execute(name, ...args) {
    if (!this.operations[name]) {
      throw new Error(`Unknown operation: ${name}`);
    }
    return this.operations[name](...args);
  }

  // Load a plugin by calling its install function
  loadPlugin(plugin) {
    plugin.install(this);
    this.plugins[plugin.name] = plugin;
    console.log(`Plugin loaded: ${plugin.name}`);
  }

  // List all registered operation names
  listOperations() {
    return Object.keys(this.operations);
  }
}

module.exports = Core;
EOF

What does this code do?

• The Core class maintains two registries: plugins (loaded plugin metadata) and operations (name-to-handler map).
• loadPlugin(plugin) is the key hook. It calls the plugin's install method, passing itself so the plugin can register operations.
• execute(name, ...args) dispatches to whatever handler was registered under that name. The core never hard-codes any operation.

• Create the add plugin by running:

cat > ~/plugin-system/plugins/add.js << 'EOF'
// Plugin: addition operation
module.exports = {
  name: 'add',
  install(core) {
    core.registerOperation('add', (a, b) => a + b);
  }
};
EOF

• Create the multiply plugin by running:

cat > ~/plugin-system/plugins/multiply.js << 'EOF'
// Plugin: multiplication operation
module.exports = {
  name: 'multiply',
  install(core) {
    core.registerOperation('multiply', (a, b) => a * b);
  }
};
EOF

What do the plugins do?

Each plugin exports a name string and an install(core) function. When the core loads the plugin, it calls install and the plugin registers its operation handler. This is the entire plugin contract. Any new plugin that follows this shape can be dropped in without changing the core.

• Create index.js to wire the core, plugins, and an HTTP server together by running:

cat > ~/plugin-system/index.js << 'EOF'
const http = require('http');
const Core = require('./core');
const addPlugin = require('./plugins/add');
const multiplyPlugin = require('./plugins/multiply');

// Initialize the microkernel core
const core = new Core();

// Load plugins into the core
core.loadPlugin(addPlugin);
core.loadPlugin(multiplyPlugin);

// Create HTTP server to expose plugin operations
const server = http.createServer((req, res) => {
  const url = new URL(req.url, `http://${req.headers.host}`);

  if (url.pathname === '/') {
    // List available operations
    res.writeHead(200, { 'Content-Type': 'application/json' });
    res.end(JSON.stringify({
      service: 'microkernel-calculator',
      operations: core.listOperations()
    }));
  } else if (url.pathname === '/execute') {
    // Execute an operation from query parameters
    const op = url.searchParams.get('op');
    const a = parseFloat(url.searchParams.get('a'));
    const b = parseFloat(url.searchParams.get('b'));

    try {
      const result = core.execute(op, a, b);
      res.writeHead(200, { 'Content-Type': 'application/json' });
      res.end(JSON.stringify({ operation: op, a, b, result }));
    } catch (err) {
      res.writeHead(400, { 'Content-Type': 'application/json' });
      res.end(JSON.stringify({ error: err.message }));
    }
  } else {
    res.writeHead(404, { 'Content-Type': 'application/json' });
    res.end(JSON.stringify({ error: 'Not found' }));
  }
});

const PORT = 3000;
server.listen(PORT, () => {
  console.log(`Microkernel calculator running on port ${PORT}`);
});
EOF

What does this code do?

• It creates a Core instance and loads both plugins at startup.
• The / endpoint lists all registered operations so you can confirm which plugins loaded.
• The /execute?op=add&a=5&b=3 endpoint dispatches to the named operation. If the operation is unknown, it returns a 400 error.
• The HTTP server uses only the Node.js standard library. No npm install is needed.

• Verify the application runs locally by running:

node ~/plugin-system/index.js &

• Test the calculator by running:

curl http://localhost:3000/
curl http://localhost:3000/execute?op=add&a=5&b=3
curl http://localhost:3000/execute?op=multiply&a=4&b=7

• You should see JSON responses listing the operations, then {"operation":"add","a":5,"b":3,"result":8}, then {"operation":"multiply","a":4,"b":7,"result":28}.
• Stop the background process by running:

kill %1

Not seeing the expected output?

• If you see a syntax error, double-check that the heredoc terminated correctly. Re-run the cat command for the file that failed.
• If curl returns connection refused, ensure the node process started without errors by checking the terminal output above.

Help me debug why my Node.js plugin system is not responding to curl requests.

• Create the Dockerfile by running:

cat > ~/plugin-system/Dockerfile << 'EOF'
FROM node:20-alpine
WORKDIR /app
COPY . .
EXPOSE 3000
CMD ["node", "index.js"]
EOF

What does this Dockerfile do?

It uses node:20-alpine as a lightweight base image, copies the application files into /app, and starts the server with node index.js. Since the app uses only the Node.js standard library, there is no npm install step needed.

• Build the Docker image by running:

cd ~/plugin-system
docker build -t plugin-calculator:latest .

• You should see each build step complete and a final line showing the image was tagged as plugin-calculator:latest.

Build failing?

• If Docker reports that it cannot find the Dockerfile, confirm you are inside ~/plugin-system before running the build command.
• If the node:20-alpine image pull fails, verify your VM has internet access.

Help me fix my Docker build for the plugin-calculator image.

• Create the stack file plugin-stack.yml by running:

cat > ~/plugin-system/plugin-stack.yml << 'EOF'
version: "3.8"

services:
  calculator:
    image: plugin-calculator:latest
    deploy:
      replicas: 1
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.calculator.rule=Host(`calculator.local`)"
        - "traefik.http.routers.calculator.entrypoints=web"
        - "traefik.http.services.calculator.loadbalancer.server.port=3000"
    networks:
      - traefik_proxy

networks:
  traefik_proxy:
    external: true
EOF

What does this stack file do?

• The traefik.enable=true label tells Traefik to discover this service.
• The router rule Host(`calculator.local`) routes requests with that Host header to this service.
• The loadbalancer.server.port=3000 label is mandatory in Swarm mode because Docker Swarm does not provide port detection to Traefik.
• The service connects to the existing traefik_proxy overlay network so Traefik can reach it.

• Deploy the plugin system stack by running:

docker stack deploy -c ~/plugin-system/plugin-stack.yml plugins

• Verify the service is running:

docker service ls

• You should see a plugins_calculator service with 1/1 replicas.
• Test routing through Traefik by running:

curl -H "Host: calculator.local" http://localhost/
curl -H "Host: calculator.local" http://localhost/execute?op=add&a=10&b=20
curl -H "Host: calculator.local" http://localhost/execute?op=multiply&a=6&b=9

• The first request should return the list of operations. The second should return {"operation":"add","a":10,"b":20,"result":30}. The third should return {"operation":"multiply","a":6,"b":9,"result":54}.

Curl returning 404 or connection refused?

• Run docker service ps plugins_calculator to check if the task is running. If it shows a failed state, check logs with docker service logs plugins_calculator.
• Verify the service joined the correct network with docker network inspect traefik_proxy. The plugins_calculator service should appear in the list.
• Ensure you are passing the Host header exactly as calculator.local. Traefik matches headers case-sensitively.

Help me debug why Traefik is not routing to my plugin calculator service.

• Copy the plugin system files to your Windows project directory for safekeeping. From your Windows terminal, run:

scp -r user@[[NODE1_IP="node1 IP address"]]:~/plugin-system .

✔️ Awesome, I've got everything!

Great. Make sure you have all five files: core.js, plugins/add.js, plugins/multiply.js, index.js, Dockerfile, and plugin-stack.yml.

ⓧ I'd like to double check the full code

// Microkernel core: plugin registry and operation dispatch
class Core {
  constructor() {
    this.plugins = {};
    this.operations = {};
  }

  // Register a named operation handler
  registerOperation(name, handler) {
    this.operations[name] = handler;
    console.log(`Operation registered: ${name}`);
  }

  // Execute a registered operation by name
  execute(name, ...args) {
    if (!this.operations[name]) {
      throw new Error(`Unknown operation: ${name}`);
    }
    return this.operations[name](...args);
  }

  // Load a plugin by calling its install function
  loadPlugin(plugin) {
    plugin.install(this);
    this.plugins[plugin.name] = plugin;
    console.log(`Plugin loaded: ${plugin.name}`);
  }

  // List all registered operation names
  listOperations() {
    return Object.keys(this.operations);
  }
}

module.exports = Core;

// Plugin: addition operation
module.exports = {
  name: 'add',
  install(core) {
    core.registerOperation('add', (a, b) => a + b);
  }
};

// Plugin: multiplication operation
module.exports = {
  name: 'multiply',
  install(core) {
    core.registerOperation('multiply', (a, b) => a * b);
  }
};

const http = require('http');
const Core = require('./core');
const addPlugin = require('./plugins/add');
const multiplyPlugin = require('./plugins/multiply');

// Initialize the microkernel core
const core = new Core();

// Load plugins into the core
core.loadPlugin(addPlugin);
core.loadPlugin(multiplyPlugin);

// Create HTTP server to expose plugin operations
const server = http.createServer((req, res) => {
  const url = new URL(req.url, `http://${req.headers.host}`);

  if (url.pathname === '/') {
    // List available operations
    res.writeHead(200, { 'Content-Type': 'application/json' });
    res.end(JSON.stringify({
      service: 'microkernel-calculator',
      operations: core.listOperations()
    }));
  } else if (url.pathname === '/execute') {
    // Execute an operation from query parameters
    const op = url.searchParams.get('op');
    const a = parseFloat(url.searchParams.get('a'));
    const b = parseFloat(url.searchParams.get('b'));

    try {
      const result = core.execute(op, a, b);
      res.writeHead(200, { 'Content-Type': 'application/json' });
      res.end(JSON.stringify({ operation: op, a, b, result }));
    } catch (err) {
      res.writeHead(400, { 'Content-Type': 'application/json' });
      res.end(JSON.stringify({ error: err.message }));
    }
  } else {
    res.writeHead(404, { 'Content-Type': 'application/json' });
    res.end(JSON.stringify({ error: 'Not found' }));
  }
});

const PORT = 3000;
server.listen(PORT, () => {
  console.log(`Microkernel calculator running on port ${PORT}`);
});

FROM node:20-alpine
WORKDIR /app
COPY . .
EXPOSE 3000
CMD ["node", "index.js"]

version: "3.8"

services:
  calculator:
    image: plugin-calculator:latest
    deploy:
      replicas: 1
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.calculator.rule=Host(`calculator.local`)"
        - "traefik.http.routers.calculator.entrypoints=web"
        - "traefik.http.services.calculator.loadbalancer.server.port=3000"
    networks:
      - traefik_proxy

networks:
  traefik_proxy:
    external: true

Your microkernel plugin system is live on the cluster. In the exam, you can adapt this skeleton by swapping the calculator operations for whatever domain the question requires. Next up, you will prepare microservices pattern templates.

Implement Microservices Pattern Templates

Your plugin system is deployed and accessible. Now it is time to prepare the microservices pattern templates that cover the remaining exam topic.

The exam may present a scenario requiring any of 5 common patterns: API Gateway, Service Discovery, Circuit Breaker, Event-Driven messaging, or Sidecar/Ambassador. In this step, you will build minimal template services for each pattern and deploy them on your Swarm cluster with Traefik routing. The goal is speed. Each template is small enough to copy and adapt under exam conditions.

In this step, get ready to:

• Review the 5 microservices patterns and understand what each demonstrates.
• Implement minimal Node.js services for each pattern.
• Deploy and verify all patterns on the Swarm cluster with Traefik routing.

Review the 5 patterns from the labs

Each pattern solves a different distributed systems problem. Your templates will be minimal implementations that prove the pattern works on your cluster.

• API Gateway receives all external requests and routes them to the correct backend service based on URL path.
• Service Discovery allows services to find and communicate with each other by name, without hardcoding IP addresses. Docker Swarm provides this natively through its embedded DNS.
• Circuit Breaker wraps calls to an unreliable service and stops sending requests when failures exceed a threshold, preventing cascade failures.
• Event-Driven decouples services through a message broker. Producers publish events without knowing who consumes them.
• Sidecar/Ambassador attaches a helper service alongside the main application to handle cross-cutting concerns like proxying external calls or logging.

How does Service Discovery work in Docker Swarm?

When services are attached to the same overlay network, Swarm's embedded DNS resolves service names to a Virtual IP (VIP). Calling http://users-service:3000 from another container automatically routes to a healthy replica of that service. No manual registry is needed.

Implement pattern templates

You will create all 5 pattern services in a single microservices/ directory with a shared Dockerfile. Each service is a small Node.js file deployed via one combined stack file.

• Create the microservices project directory on your Windows machine and initialize it by running:

mkdir microservices
cd microservices
npm init -y
npm install amqplib

Why amqplib?

The Event-Driven pattern template connects to RabbitMQ, which is already running on your cluster. The amqplib package provides the AMQP client needed for that connection. The other patterns use only Node.js built-in modules.

• Create the shared Dockerfile that all pattern services will use by saving the following content:

FROM node:20-alpine
WORKDIR /app
COPY package*.json ./
RUN npm install --production
COPY *.js ./
CMD ["node", "server.js"]

What does this Dockerfile do?

It copies all JavaScript files into the image and defaults to running server.js. The stack file overrides the CMD for each service to point at the correct script.

• Create gateway.js for the API Gateway pattern:

const http = require('http');

// Route table: path prefix -> backend service name and port
const routes = {
  '/users': { host: 'users-service', port: 3000 },
  '/orders': { host: 'orders-service', port: 3000 }
};

const server = http.createServer((req, res) => {
  // Find matching route by path prefix
  const match = Object.entries(routes).find(([prefix]) => req.url.startsWith(prefix));

  if (!match) {
    res.writeHead(404, { 'Content-Type': 'application/json' });
    return res.end(JSON.stringify({ error: 'Route not found' }));
  }

  const [, target] = match;

  // Proxy the request to the backend service
  const proxy = http.request(
    { hostname: target.host, port: target.port, path: req.url, method: req.method },
    (proxyRes) => {
      res.writeHead(proxyRes.statusCode, proxyRes.headers);
      proxyRes.pipe(res);
    }
  );

  proxy.on('error', () => {
    res.writeHead(503, { 'Content-Type': 'application/json' });
    res.end(JSON.stringify({ error: 'Service unavailable' }));
  });

  req.pipe(proxy);
});

server.listen(3000, () => console.log('API Gateway listening on port 3000'));

What does this code do?

The gateway listens on port 3000 and proxies incoming requests to backend services based on URL path. Requests starting with /users go to users-service and requests starting with /orders go to orders-service. Swarm DNS resolves those service names automatically because they share the same overlay network.

• Create backend.js which serves as the template backend for users-service and orders-service:

const http = require('http');
const os = require('os');

// SERVICE_NAME is set via environment variable in the stack file
const SERVICE_NAME = process.env.SERVICE_NAME || 'unknown';

const server = http.createServer((req, res) => {
  res.writeHead(200, { 'Content-Type': 'application/json' });
  // Return service name and container hostname to prove discovery + load balancing
  res.end(JSON.stringify({
    service: SERVICE_NAME,
    hostname: os.hostname(),
    path: req.url,
    timestamp: new Date().toISOString()
  }));
});

server.listen(3000, () => console.log(`${SERVICE_NAME} running on port 3000`));

How does this demonstrate Service Discovery?

When the gateway calls users-service:3000, Docker Swarm's embedded DNS resolves that name to the service's VIP. The response includes the container hostname, proving which replica answered. Calling the same URL repeatedly shows different hostnames as Swarm load-balances across replicas.

• Create unreliable.js which simulates an unreliable backend for the Circuit Breaker pattern:

const http = require('http');

// Simulate an unreliable service that fails 50% of the time
const server = http.createServer((req, res) => {
  if (Math.random() < 0.5) {
    res.writeHead(500, { 'Content-Type': 'application/json' });
    return res.end(JSON.stringify({ error: 'Internal server error' }));
  }
  res.writeHead(200, { 'Content-Type': 'application/json' });
  res.end(JSON.stringify({ status: 'ok', timestamp: new Date().toISOString() }));
});

server.listen(3000, () => console.log('Unreliable service on port 3000'));

• Create circuit-breaker.js which wraps calls to the unreliable service with circuit breaker logic:

const http = require('http');

// Circuit Breaker state machine
let state = 'CLOSED';
let failures = 0;
const THRESHOLD = 3;
const RESET_TIMEOUT = 10000;
let nextAttempt = Date.now();

function callBackend() {
  return new Promise((resolve, reject) => {
    const req = http.get('http://unreliable-service:3000/', (res) => {
      let data = '';
      res.on('data', (chunk) => { data += chunk; });
      res.on('end', () => {
        if (res.statusCode >= 500) reject(new Error('Backend error'));
        else resolve(data);
      });
    });
    req.on('error', reject);
    req.setTimeout(3000, () => { req.destroy(); reject(new Error('Timeout')); });
  });
}

const server = http.createServer(async (req, res) => {
  // If circuit is OPEN, check if reset timeout has passed
  if (state === 'OPEN') {
    if (Date.now() > nextAttempt) {
      state = 'HALF-OPEN';
    } else {
      res.writeHead(503, { 'Content-Type': 'application/json' });
      return res.end(JSON.stringify({ error: 'Circuit OPEN', state, nextAttempt: new Date(nextAttempt).toISOString() }));
    }
  }

  try {
    const result = await callBackend();
    // Success resets the breaker
    failures = 0;
    state = 'CLOSED';
    res.writeHead(200, { 'Content-Type': 'application/json' });
    res.end(JSON.stringify({ state, backend: JSON.parse(result) }));
  } catch (err) {
    failures++;
    if (failures >= THRESHOLD) {
      state = 'OPEN';
      nextAttempt = Date.now() + RESET_TIMEOUT;
    }
    res.writeHead(502, { 'Content-Type': 'application/json' });
    res.end(JSON.stringify({ state, failures, error: err.message }));
  }
});

server.listen(3000, () => console.log('Circuit Breaker client on port 3000'));

What does the circuit breaker do?

The breaker tracks consecutive failures. After 3 failures (the threshold), the circuit opens and immediately returns 503 without calling the backend. After 10 seconds, it enters HALF-OPEN state and allows one test request through. A successful test closes the circuit again.

This prevents a failing backend from overwhelming itself with retries and protects upstream callers from slow timeouts.

• Create event-producer.js for the Event-Driven pattern:

const http = require('http');
const amqp = require('amqplib');

const QUEUE = 'events';
let channel = null;

// Connect to RabbitMQ on startup
async function connect() {
  const conn = await amqp.connect('amqp://guest:guest@messaging_rabbitmq:5672');
  channel = await conn.createChannel();
  await channel.assertQueue(QUEUE, { durable: true });
  console.log('Connected to RabbitMQ');
}

const server = http.createServer((req, res) => {
  if (!channel) {
    res.writeHead(503, { 'Content-Type': 'application/json' });
    return res.end(JSON.stringify({ error: 'Not connected to broker' }));
  }
  // Publish an event with the current timestamp
  const event = JSON.stringify({ type: 'order.created', timestamp: new Date().toISOString() });
  channel.sendToQueue(QUEUE, Buffer.from(event), { persistent: true });
  res.writeHead(200, { 'Content-Type': 'application/json' });
  res.end(JSON.stringify({ published: event }));
});

connect().catch(console.error);
server.listen(3000, () => console.log('Event Producer on port 3000'));

• Create event-consumer.js which consumes messages from the same queue:

const amqp = require('amqplib');

const QUEUE = 'events';

async function consume() {
  // Connect to RabbitMQ using the stack service name
  const conn = await amqp.connect('amqp://guest:guest@messaging_rabbitmq:5672');
  const channel = await conn.createChannel();
  await channel.assertQueue(QUEUE, { durable: true });
  // Process one message at a time
  channel.prefetch(1);
  console.log('Consumer waiting for events...');
  channel.consume(QUEUE, (msg) => {
    const event = JSON.parse(msg.content.toString());
    console.log('Received:', event);
    channel.ack(msg);
  });
}

consume().catch(console.error);

How does Event-Driven messaging decouple services?

The producer publishes events to RabbitMQ without knowing which services consume them. The consumer pulls events independently. You can add more consumers or change the processing logic without modifying the producer. This decoupling is the core value of event-driven architecture.

• Create ambassador.js for the Sidecar/Ambassador pattern:

const http = require('http');

// Ambassador proxies external API calls and adds caching/retry logic
const cache = {};

const server = http.createServer((req, res) => {
  const target = req.url.slice(1); // Strip leading slash to get target URL path

  // Return cached response if available
  if (cache[target]) {
    res.writeHead(200, { 'Content-Type': 'application/json' });
    return res.end(JSON.stringify({ source: 'cache', data: cache[target] }));
  }

  // Simulate an external API call with a timestamp
  const result = { external_api: target, fetched_at: new Date().toISOString() };
  cache[target] = result;

  res.writeHead(200, { 'Content-Type': 'application/json' });
  res.end(JSON.stringify({ source: 'live', data: result }));
});

server.listen(3001, () => console.log('Ambassador sidecar on port 3001'));

• Create app-with-sidecar.js which uses the ambassador for external calls:

const http = require('http');

// Main app delegates external calls to the ambassador sidecar
function callAmbassador(path) {
  return new Promise((resolve, reject) => {
    http.get(`http://ambassador-service:3001/${path}`, (res) => {
      let data = '';
      res.on('data', (chunk) => { data += chunk; });
      res.on('end', () => resolve(JSON.parse(data)));
    }).on('error', reject);
  });
}

const server = http.createServer(async (req, res) => {
  try {
    // Route through ambassador for external data
    const result = await callAmbassador('weather-api');
    res.writeHead(200, { 'Content-Type': 'application/json' });
    res.end(JSON.stringify({ app: 'main', ambassador_response: result }));
  } catch (err) {
    res.writeHead(502, { 'Content-Type': 'application/json' });
    res.end(JSON.stringify({ error: 'Ambassador unavailable' }));
  }
});

server.listen(3000, () => console.log('App with sidecar on port 3000'));

What is the Sidecar/Ambassador pattern?

The main application never calls external APIs directly. Instead, it routes all external requests through the ambassador service running alongside it on the same network. The ambassador handles caching, retries, and connection pooling. This separates application logic from infrastructure concerns.

In Docker Swarm, the sidecar runs as a separate service on the same overlay network. The app reaches it by Swarm DNS name.

• Create the combined stack file microservices-stack.yml that deploys all pattern services:

services:
  # --- API Gateway Pattern ---
  api-gateway:
    image: microservices:latest
    command: ["node", "gateway.js"]
    networks:
      - traefik_proxy
      - microservices_internal
    deploy:
      replicas: 1
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.api-gateway.rule=Host(`gateway.local`)"
        - "traefik.http.routers.api-gateway.entrypoints=web"
        - "traefik.http.services.api-gateway.loadbalancer.server.port=3000"

  # --- Service Discovery Pattern (backend services) ---
  users-service:
    image: microservices:latest
    command: ["node", "backend.js"]
    environment:
      - SERVICE_NAME=users-service
    networks:
      - microservices_internal
    deploy:
      replicas: 2

  orders-service:
    image: microservices:latest
    command: ["node", "backend.js"]
    environment:
      - SERVICE_NAME=orders-service
    networks:
      - microservices_internal
    deploy:
      replicas: 2

  # --- Circuit Breaker Pattern ---
  circuit-breaker:
    image: microservices:latest
    command: ["node", "circuit-breaker.js"]
    networks:
      - traefik_proxy
      - microservices_internal
    deploy:
      replicas: 1
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.circuit-breaker.rule=Host(`breaker.local`)"
        - "traefik.http.routers.circuit-breaker.entrypoints=web"
        - "traefik.http.services.circuit-breaker.loadbalancer.server.port=3000"

  unreliable-service:
    image: microservices:latest
    command: ["node", "unreliable.js"]
    networks:
      - microservices_internal
    deploy:
      replicas: 1

  # --- Event-Driven Pattern ---
  event-producer:
    image: microservices:latest
    command: ["node", "event-producer.js"]
    networks:
      - traefik_proxy
      - microservices_internal
    deploy:
      replicas: 1
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.event-producer.rule=Host(`events.local`)"
        - "traefik.http.routers.event-producer.entrypoints=web"
        - "traefik.http.services.event-producer.loadbalancer.server.port=3000"

  event-consumer:
    image: microservices:latest
    command: ["node", "event-consumer.js"]
    networks:
      - microservices_internal
    deploy:
      replicas: 1

  # --- Sidecar/Ambassador Pattern ---
  app-with-sidecar:
    image: microservices:latest
    command: ["node", "app-with-sidecar.js"]
    networks:
      - traefik_proxy
      - microservices_internal
    deploy:
      replicas: 1
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.app-sidecar.rule=Host(`sidecar.local`)"
        - "traefik.http.routers.app-sidecar.entrypoints=web"
        - "traefik.http.services.app-sidecar.loadbalancer.server.port=3000"

  ambassador-service:
    image: microservices:latest
    command: ["node", "ambassador.js"]
    networks:
      - microservices_internal
    deploy:
      replicas: 1

networks:
  traefik_proxy:
    external: true
  microservices_internal:
    driver: overlay

What does this stack file do?

Each service uses the same Docker image but overrides the command to run a different script. Services that need external access (via Traefik) are attached to both traefik_proxy and the internal network. Backend-only services sit on microservices_internal only, making them unreachable from outside the cluster.

The traefik_proxy network is marked external: true because it already exists from your Traefik deployment. The internal overlay network is created automatically when the stack deploys.

• Save all files.
• SCP the entire microservices directory to node1 by running:

scp -r microservices/ user@<NODE1-IP>:~/

• SSH into node1 and build the Docker image by running:

cd ~/microservices
docker build -t microservices:latest .

• Deploy the stack by running:

docker stack deploy -c microservices-stack.yml patterns

• Verify all services are running by running:

docker service ls

• You should see all 8 services from the patterns stack listed with their desired replicas matching the current replicas.

Services stuck at 0/N replicas?

If some services show 0 replicas after 30 seconds, check the service logs:

docker service ps patterns_api-gateway --no-trunc

Common causes: the image microservices:latest was only built on node1 but the service got scheduled to node2 or node3. Since you are not using a registry, build the image on all nodes or constrain all services to node1 for testing.

help me debug why my Swarm services show 0 replicas

• Test the API Gateway pattern and Service Discovery by running:

curl -H "Host: gateway.local" http://localhost/users

• You should see a JSON response with "service": "users-service" and a container hostname. Run the command multiple times to see different hostnames as Swarm load-balances across replicas.
• Test the Circuit Breaker pattern by running:

for i in $(seq 1 6); do curl -s -H "Host: breaker.local" http://localhost/ | python3 -m json.tool; done

• You should see a mix of successful and failed responses. After 3 consecutive failures, the circuit opens and subsequent requests return "state": "OPEN" immediately without contacting the backend.
• Test the Event-Driven pattern by running:

curl -H "Host: events.local" http://localhost/

• You should see a JSON response confirming an event was published. Check the consumer logs to see it was received:

docker service logs patterns_event-consumer --tail 5

• You should see a log line showing Received: { type: 'order.created', ... }.
• Test the Sidecar/Ambassador pattern by running:

curl -H "Host: sidecar.local" http://localhost/

• The first request shows "source": "live". Run it again to see "source": "cache", proving the ambassador caches external calls.

Getting connection refused on curl?

Verify Traefik can reach the services by checking they are on the traefik_proxy network:

docker network inspect traefik_proxy --format '{{range .Services}}{{.Name}} {{end}}'

If pattern services are missing, ensure the stack file references traefik_proxy as external: true and that the network was created with --attachable.

help me debug Traefik not routing to my Swarm services

✔️ Awesome, I've got everything!

All 5 patterns are deployed and verified. Make sure you have saved microservices-stack.yml in your Windows project directory alongside the JavaScript files.

ⓧ I'd like to double check the full code

Here is the complete microservices-stack.yml for reference:

services:
  # --- API Gateway Pattern ---
  api-gateway:
    image: microservices:latest
    command: ["node", "gateway.js"]
    networks:
      - traefik_proxy
      - microservices_internal
    deploy:
      replicas: 1
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.api-gateway.rule=Host(`gateway.local`)"
        - "traefik.http.routers.api-gateway.entrypoints=web"
        - "traefik.http.services.api-gateway.loadbalancer.server.port=3000"

  # --- Service Discovery Pattern (backend services) ---
  users-service:
    image: microservices:latest
    command: ["node", "backend.js"]
    environment:
      - SERVICE_NAME=users-service
    networks:
      - microservices_internal
    deploy:
      replicas: 2

  orders-service:
    image: microservices:latest
    command: ["node", "backend.js"]
    environment:
      - SERVICE_NAME=orders-service
    networks:
      - microservices_internal
    deploy:
      replicas: 2

  # --- Circuit Breaker Pattern ---
  circuit-breaker:
    image: microservices:latest
    command: ["node", "circuit-breaker.js"]
    networks:
      - traefik_proxy
      - microservices_internal
    deploy:
      replicas: 1
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.circuit-breaker.rule=Host(`breaker.local`)"
        - "traefik.http.routers.circuit-breaker.entrypoints=web"
        - "traefik.http.services.circuit-breaker.loadbalancer.server.port=3000"

  unreliable-service:
    image: microservices:latest
    command: ["node", "unreliable.js"]
    networks:
      - microservices_internal
    deploy:
      replicas: 1

  # --- Event-Driven Pattern ---
  event-producer:
    image: microservices:latest
    command: ["node", "event-producer.js"]
    networks:
      - traefik_proxy
      - microservices_internal
    deploy:
      replicas: 1
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.event-producer.rule=Host(`events.local`)"
        - "traefik.http.routers.event-producer.entrypoints=web"
        - "traefik.http.services.event-producer.loadbalancer.server.port=3000"

  event-consumer:
    image: microservices:latest
    command: ["node", "event-consumer.js"]
    networks:
      - microservices_internal
    deploy:
      replicas: 1

  # --- Sidecar/Ambassador Pattern ---
  app-with-sidecar:
    image: microservices:latest
    command: ["node", "app-with-sidecar.js"]
    networks:
      - traefik_proxy
      - microservices_internal
    deploy:
      replicas: 1
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.app-sidecar.rule=Host(`sidecar.local`)"
        - "traefik.http.routers.app-sidecar.entrypoints=web"
        - "traefik.http.services.app-sidecar.loadbalancer.server.port=3000"

  ambassador-service:
    image: microservices:latest
    command: ["node", "ambassador.js"]
    networks:
      - microservices_internal
    deploy:
      replicas: 1

networks:
  traefik_proxy:
    external: true
  microservices_internal:
    driver: overlay

All 5 microservices patterns are now deployed as reference templates on your cluster. Next up, you will implement the OAuth/OIDC flow with GitHub to complete the final exam topic.

Implement OAuth/OIDC with GitHub

Your cluster now has message brokering, shared storage, and pattern templates deployed. The final exam topic is authentication.

In this step, you will implement the OAuth 2.0 authorization code flow using GitHub as the identity provider. Your application will redirect users to GitHub, receive an authorization code, exchange it for an access token, and fetch the user's profile. This demonstrates the OIDC-like pattern the exam assesses.

In this step, get ready to:

• Register a GitHub OAuth App with a callback URL pointing to Traefik.
• Build a Node.js web app that completes the full OAuth flow.
• Containerize and deploy the app as a Docker stack behind Traefik.

Register a GitHub OAuth App

Before your application can authenticate users via GitHub, you need to register it as an OAuth App on GitHub. This gives you a Client ID and Client Secret that your app uses to identify itself during the authorization flow.

• Open your browser and navigate to GitHub.
• Click your profile picture in the upper-right corner, then click Settings.
• In the left sidebar, click Developer settings.
• In the left sidebar, click OAuth Apps.
• Click New OAuth App.
• In Application name, type swarm-oauth-demo.
• In Homepage URL, enter http://your Traefik manager node IP.
• In Authorization callback URL, enter http://your Traefik manager node IP/callback.
• Click Register application.

Why does the callback URL matter?

The callback URL is the address GitHub redirects users to after they authorize your app. GitHub sends a temporary authorization code to this URL, which your app then exchanges for an access token.

Since your app sits behind Traefik, the callback URL must point at a Traefik manager node IP so the request hits Traefik first, then routes to your service.

• On the app's settings page, note the Client ID displayed at the top.
• Click Generate a new client secret.
• Copy the Client Secret immediately and store it somewhere safe. You will not be able to see it again.

Keep your Client Secret safe

The Client Secret is a credential. Do not commit it to version control or share it publicly. You will pass it to your container as an environment variable.

Implement the OAuth flow application

Now you will create a Node.js web application that handles all four stages of the OAuth authorization code flow: redirect to GitHub, receive the callback, exchange the code for a token, and fetch user info.

• On your Windows machine, create a new directory called oauth-app inside your project folder.
• Create a file called app.js inside the oauth-app directory with the following content:

const http = require('http');
const https = require('https');
const url = require('url');

// OAuth credentials from environment variables
const CLIENT_ID = process.env.GITHUB_CLIENT_ID;
const CLIENT_SECRET = process.env.GITHUB_CLIENT_SECRET;
const CALLBACK_URL = process.env.CALLBACK_URL || 'http://localhost/callback';

const PORT = 3000;

const server = http.createServer((req, res) => {
  const parsedUrl = url.parse(req.url, true);

  // Home route - shows login link
  if (parsedUrl.pathname === '/') {
    const authUrl = `https://github.com/login/oauth/authorize?client_id=${CLIENT_ID}&redirect_uri=${CALLBACK_URL}&scope=read:user`;
    res.writeHead(200, { 'Content-Type': 'text/html' });
    res.end(`
      <h1>OAuth/OIDC Demo</h1>
      <p><a href="${authUrl}">Login with GitHub</a></p>
    `);
    return;
  }

  // Callback route - exchanges code for token, fetches user info
  if (parsedUrl.pathname === '/callback') {
    const code = parsedUrl.query.code;
    if (!code) {
      res.writeHead(400, { 'Content-Type': 'text/plain' });
      res.end('Missing authorization code');
      return;
    }

    // Exchange the authorization code for an access token
    const tokenData = JSON.stringify({
      client_id: CLIENT_ID,
      client_secret: CLIENT_SECRET,
      code: code
    });

    const tokenOptions = {
      hostname: 'github.com',
      path: '/login/oauth/access_token',
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'Accept': 'application/json',
        'Content-Length': Buffer.byteLength(tokenData)
      }
    };

    const tokenReq = https.request(tokenOptions, (tokenRes) => {
      let body = '';
      tokenRes.on('data', (chunk) => body += chunk);
      tokenRes.on('end', () => {
        const tokenResponse = JSON.parse(body);
        const accessToken = tokenResponse.access_token;

        // Fetch user info from GitHub API
        const userOptions = {
          hostname: 'api.github.com',
          path: '/user',
          method: 'GET',
          headers: {
            'Authorization': `Bearer ${accessToken}`,
            'User-Agent': 'swarm-oauth-demo',
            'Accept': 'application/json'
          }
        };

        const userReq = https.request(userOptions, (userRes) => {
          let userData = '';
          userRes.on('data', (chunk) => userData += chunk);
          userRes.on('end', () => {
            const user = JSON.parse(userData);
            res.writeHead(200, { 'Content-Type': 'text/html' });
            res.end(`
              <h1>Authentication Successful</h1>
              <p><strong>Username:</strong> ${user.login}</p>
              <p><strong>Name:</strong> ${user.name || 'N/A'}</p>
              <p><strong>ID:</strong> ${user.id}</p>
              <p><strong>Avatar:</strong></p>
              <img src="${user.avatar_url}" width="100" />
              <p><a href="/">Back to home</a></p>
            `);
          });
        });
        userReq.end();
      });
    });

    tokenReq.write(tokenData);
    tokenReq.end();
    return;
  }

  // 404 for unknown routes
  res.writeHead(404, { 'Content-Type': 'text/plain' });
  res.end('Not found');
});

server.listen(PORT, () => {
  console.log(`OAuth app listening on port ${PORT}`);
});

What does this code do?

The application has two routes. The home route / generates a link that redirects the user to GitHub's authorization endpoint at https://github.com/login/oauth/authorize with your Client ID and requested scope.

The /callback route receives the temporary authorization code from GitHub, POSTs it to https://github.com/login/oauth/access_token along with the Client ID and Secret to exchange it for an access token. It then uses that token to call https://api.github.com/user and display the authenticated user's profile.

• Create a Dockerfile inside the oauth-app directory:

FROM node:20-alpine

# Set working directory
WORKDIR /app

# Copy application code
COPY app.js .

# Expose the app port
EXPOSE 3000

# Start the application
CMD ["node", "app.js"]

What does this Dockerfile do?

This uses the lightweight node:20-alpine base image. The app uses only Node.js built-in modules (http, https, url), so no npm install step is needed.

• Create the stack file oauth-stack.yml inside the oauth-app directory:

version: '3.8'

services:
  oauth-app:
    image: oauth-app:latest
    networks:
      - traefik_proxy
    environment:
      - GITHUB_CLIENT_ID=your-client-id-here
      - GITHUB_CLIENT_SECRET=your-client-secret-here
      - CALLBACK_URL=http://your-traefik-ip/callback
    deploy:
      replicas: 1
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.oauth.rule=Host(`oauth.local`)"
        - "traefik.http.routers.oauth.entrypoints=web"
        - "traefik.http.services.oauth.loadbalancer.server.port=3000"

networks:
  traefik_proxy:
    external: true

What does this stack file do?

The stack deploys your OAuth app image and connects it to the existing traefik_proxy overlay network. The deploy labels tell Traefik to route requests with Host: oauth.local to port 3000 on the container.

Replace the environment variable placeholders with your actual GitHub Client ID, Client Secret, and Traefik IP before deploying.

• Save all three files (app.js, Dockerfile, oauth-stack.yml).

Unsure about the callback URL value?

The CALLBACK_URL environment variable must exactly match the Authorization callback URL you entered in the GitHub OAuth App settings. If they differ, GitHub will reject the callback with an error.

help me figure out why my OAuth callback URL is being rejected

Build, deploy, and test the OAuth flow

With the application code ready, you will now transfer it to node1, build the Docker image, deploy the stack, and verify the full authentication round-trip.

• SCP the entire oauth-app directory to node1 by running this command from your Windows machine:

scp -r oauth-app/ [[VM_USER="your VM username"]]@[[NODE1_IP="node1 IP address"]]:~/oauth-app/

• Switch back to your SSH session on node1.
• Build the Docker image by running:

cd ~/oauth-app && docker build -t oauth-app:latest .

• Before deploying, edit oauth-stack.yml on node1 to replace the placeholder values with your actual GitHub Client ID, Client Secret, and Traefik manager IP.
• Deploy the stack by running:

docker stack deploy -c oauth-stack.yml oauth-app

• Verify the service is running:

docker service ls

You should see oauth-app_oauth-app with 1/1 replicas.

• Test the full OAuth flow by running:

curl -H "Host: oauth.local" http://localhost/

You should see HTML containing a "Login with GitHub" link pointing to https://github.com/login/oauth/authorize.

• To complete the full end-to-end test, open a browser on your Windows machine and navigate to http://your Traefik manager node IP/ with the Host header set to oauth.local.
• Click the Login with GitHub link.
• GitHub presents an authorization page. Click Authorize.
• GitHub redirects you back to your app with a code. The app exchanges it for a token and displays your GitHub username, name, and avatar.

Testing without browser Host header manipulation

If you cannot set the Host header in your browser, add an entry to your Windows hosts file (C:\Windows\System32\drivers\etc\hosts) mapping oauth.local to your Traefik manager IP. Then you can access http://oauth.local/ directly in the browser.

Not seeing your user info after authorization?

Make sure the GITHUB_CLIENT_ID and GITHUB_CLIENT_SECRET values in oauth-stack.yml match exactly what GitHub shows on your app's settings page.

Check that the CALLBACK_URL environment variable matches the Authorization callback URL registered in GitHub.

Inspect logs with docker service logs oauth-app_oauth-app to see if there are errors during the token exchange.

help me debug my OAuth token exchange with GitHub

✔️ Awesome, I've got everything!

Great. Make sure oauth-stack.yml is saved in your Windows project directory under oauth-app/ alongside app.js and Dockerfile.

ⓧ I'd like to double check the full code

Here is the full app.js:

const http = require('http');
const https = require('https');
const url = require('url');

// OAuth credentials from environment variables
const CLIENT_ID = process.env.GITHUB_CLIENT_ID;
const CLIENT_SECRET = process.env.GITHUB_CLIENT_SECRET;
const CALLBACK_URL = process.env.CALLBACK_URL || 'http://localhost/callback';

const PORT = 3000;

const server = http.createServer((req, res) => {
  const parsedUrl = url.parse(req.url, true);

  // Home route - shows login link
  if (parsedUrl.pathname === '/') {
    const authUrl = `https://github.com/login/oauth/authorize?client_id=${CLIENT_ID}&redirect_uri=${CALLBACK_URL}&scope=read:user`;
    res.writeHead(200, { 'Content-Type': 'text/html' });
    res.end(`
      <h1>OAuth/OIDC Demo</h1>
      <p><a href="${authUrl}">Login with GitHub</a></p>
    `);
    return;
  }

  // Callback route - exchanges code for token, fetches user info
  if (parsedUrl.pathname === '/callback') {
    const code = parsedUrl.query.code;
    if (!code) {
      res.writeHead(400, { 'Content-Type': 'text/plain' });
      res.end('Missing authorization code');
      return;
    }

    // Exchange the authorization code for an access token
    const tokenData = JSON.stringify({
      client_id: CLIENT_ID,
      client_secret: CLIENT_SECRET,
      code: code
    });

    const tokenOptions = {
      hostname: 'github.com',
      path: '/login/oauth/access_token',
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'Accept': 'application/json',
        'Content-Length': Buffer.byteLength(tokenData)
      }
    };

    const tokenReq = https.request(tokenOptions, (tokenRes) => {
      let body = '';
      tokenRes.on('data', (chunk) => body += chunk);
      tokenRes.on('end', () => {
        const tokenResponse = JSON.parse(body);
        const accessToken = tokenResponse.access_token;

        // Fetch user info from GitHub API
        const userOptions = {
          hostname: 'api.github.com',
          path: '/user',
          method: 'GET',
          headers: {
            'Authorization': `Bearer ${accessToken}`,
            'User-Agent': 'swarm-oauth-demo',
            'Accept': 'application/json'
          }
        };

        const userReq = https.request(userOptions, (userRes) => {
          let userData = '';
          userRes.on('data', (chunk) => userData += chunk);
          userRes.on('end', () => {
            const user = JSON.parse(userData);
            res.writeHead(200, { 'Content-Type': 'text/html' });
            res.end(`
              <h1>Authentication Successful</h1>
              <p><strong>Username:</strong> ${user.login}</p>
              <p><strong>Name:</strong> ${user.name || 'N/A'}</p>
              <p><strong>ID:</strong> ${user.id}</p>
              <p><strong>Avatar:</strong></p>
              <img src="${user.avatar_url}" width="100" />
              <p><a href="/">Back to home</a></p>
            `);
          });
        });
        userReq.end();
      });
    });

    tokenReq.write(tokenData);
    tokenReq.end();
    return;
  }

  // 404 for unknown routes
  res.writeHead(404, { 'Content-Type': 'text/plain' });
  res.end('Not found');
});

server.listen(PORT, () => {
  console.log(`OAuth app listening on port ${PORT}`);
});

Here is the full Dockerfile:

FROM node:20-alpine

# Set working directory
WORKDIR /app

# Copy application code
COPY app.js .

# Expose the app port
EXPOSE 3000

# Start the application
CMD ["node", "app.js"]

Here is the full oauth-stack.yml:

version: '3.8'

services:
  oauth-app:
    image: oauth-app:latest
    networks:
      - traefik_proxy
    environment:
      - GITHUB_CLIENT_ID=your-client-id-here
      - GITHUB_CLIENT_SECRET=your-client-secret-here
      - CALLBACK_URL=http://your-traefik-ip/callback
    deploy:
      replicas: 1
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.oauth.rule=Host(`oauth.local`)"
        - "traefik.http.routers.oauth.entrypoints=web"
        - "traefik.http.services.oauth.loadbalancer.server.port=3000"

networks:
  traefik_proxy:
    external: true

Your OAuth/OIDC implementation is deployed and tested. You now have a working reference for the full authorization code grant flow that you can adapt during the exam. Next, you will tackle the Secret Mission and configure a clustered RabbitMQ deployment for fault tolerance.

Secret mission

Clustered RabbitMQ with Quorum Queues

Your single-node RabbitMQ setup works, but it has a single point of failure. In this secret mission, configure a 3-node RabbitMQ cluster running in Docker Swarm with automatic DNS peer discovery and quorum queues for fault tolerance. Prove your cluster can survive a node going down without losing messages.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely on local VirtualBox VMs and a free GitHub OAuth App, so there are no ongoing costs.

Resources you used:

• 3 VirtualBox VMs (node1, node2, node3) running Ubuntu with Docker Engine.
• Docker Swarm cluster with stacks: traefik, messaging, oauth-app, plugins, and microservices patterns.
• MicroCeph 3-node cluster with CephFS mounted at /mnt/shared on all VMs.
• Overlay networks: traefik_proxy and rabbitmq_internal.
• GitHub OAuth App registration.
• Windows project directory with all stack files and scripts.

Keep everything running

No action needed. Choose this if you are still actively practicing for your exam and want all services accessible immediately.

• Your VMs continue running in VirtualBox. The Docker Swarm cluster, all stacks, and MicroCeph remain operational.
• Your GitHub OAuth App stays registered and functional.
• Your Windows project directory stays as-is for quick SCP transfers.

Pause - I'll come back to this later

Shut down VMs to free up RAM and CPU on your host machine. All state is preserved and you can resume later.

• In VirtualBox, right-click each VM (node1, node2, node3) and select Close > Save State. This preserves the running state of all services.
• When you return, right-click each VM and select Start. Docker Swarm and all stacks will resume automatically.
• Your GitHub OAuth App and Windows project directory require no action.

Delete - I don't want to use this again

Remove all project resources completely. Do this only if you have finished your exam or no longer need the practice environment.

• Remove all Docker stacks from node1 (the manager) by running:

docker stack rm traefik messaging oauth-app plugins
docker stack rm $(docker stack ls --format '{{.Name}}')

• Remove overlay networks by running:

docker network rm traefik_proxy rabbitmq_internal

• Remove the Docker secret by running:

docker secret rm rabbitmq-erlang-cookie

• Leave the Swarm on each node. SSH into each VM and run:

docker swarm leave --force

• Unmount CephFS on each VM by running:

sudo umount /mnt/shared

• Remove MicroCeph on each VM by running:

sudo snap remove microceph

• Delete the VirtualBox VMs. In VirtualBox, right-click each VM (node1, node2, node3), select Remove, then choose Delete all files.
• Delete the GitHub OAuth App. Go to GitHub Settings > Developer settings > OAuth Apps. Select your app, scroll down, and click Delete application.
• Delete your Windows project directory by opening File Explorer, navigating to the folder containing your stack files, and deleting it.

That's a Wrap!

That's a Wrap!

Nice work! You have built a complete distributed systems exam preparation environment from scratch, covering every assessed topic end-to-end on a real 3-node cluster.

Here's what you accomplished:

• Deployed a 3-node Docker Swarm cluster with Traefik v3 as a global-mode reverse proxy, MicroCeph shared storage with CephFS, and RabbitMQ messaging, all deployed declaratively via docker stack files.
• Implemented all 6 RabbitMQ messaging patterns in Node.js (Hello World, Work Queues, Pub/Sub, Routing, Topics, and RPC) and tested them against a live Swarm-deployed broker.
• Built exam-ready templates for a microkernel plugin system, microservices patterns, and a full OAuth/OIDC flow with GitHub as identity provider.
• Configured a clustered 3-node RabbitMQ deployment with DNS peer discovery and quorum queues for fault tolerance.

Ready to quiz yourself?