Build a Slurm HPC Cluster on Your Homelab

Introduction

30 Second Summary

When you have a rack of servers sitting idle, running a job on one machine is easy. Running dozens of jobs across all of them at once, deciding which machine gets what work and when, is the hard part that every supercomputer and research cluster solves with a job scheduler.

In this project, you will build a real 4-node Slurm cluster across your homelab, configuring one node as the controller and three as compute workers that accept and run parallel jobs on demand.

What You'll Build

Picture submitting a job from your head node and watching three different machines report back, then flooding the queue with work and seeing Slurm's backfill scheduler intelligently slot short jobs into gaps behind long-running ones.

By the end of this project, you'll have:

• A 4-node Slurm cluster where you submit srun --nodes=3 hostname and see three different hostnames come back from your physical compute nodes.
• Batch job workflows where you write scripts, submit them with sbatch, and watch them queue, dispatch, and complete across different machines.
• Partition-based scheduling that separates debug and batch workloads, letting Slurm's backfill algorithm fill scheduling gaps so short jobs no longer wait behind long ones.
• Secret Mission: Configure GPU scheduling with GRES so jobs can explicitly request your Quadro M620 GPUs.

Are there any prerequisites?

This project assumes you have 4 Ubuntu 24.04 nodes networked via Tailscale, with SSH access between all nodes and sudo privileges on each one.

You should be comfortable with basic Linux administration (editing files, running services, using the terminal). No prior Slurm or HPC experience is needed.

Before We Start

Before we dive into installing packages and configuring services, let's lock in what you're about to build across your homelab and why a job scheduler matters.

Install Munge and Slurm on All Nodes

Before your Slurm cluster can schedule a single job, every node needs two things: a way to prove its identity and the Slurm daemons themselves.

Munge handles authentication by encrypting credentials with a shared secret key. If a node does not have the same key, the cluster will refuse to trust it. In this step, you will install Munge on all 4 nodes, distribute the shared key, then install the Slurm packages.

In this step, get ready to:

• Install Munge and Slurm client tools on all 4 nodes.
• Generate and distribute the Munge key so all nodes trust each other.
• Install the Slurm controller and worker daemons on their respective nodes.

Install Munge and Slurm client tools on all nodes

Every node in a Slurm cluster needs the Munge authentication library and the Slurm client commands (sinfo, squeue, sbatch). You will run the same install command on all 4 nodes.

• On the head node (your current SSH session), install the packages by running:

sudo apt update && sudo apt install -y munge slurm-client

• SSH into each of your 3 compute nodes and run the same command. For example:

ssh compute1 'sudo apt update && sudo apt install -y munge slurm-client'

• Repeat for compute2 and compute3.

What does this install?

The munge package provides the authentication daemon and the mungekey utility for key management. The slurm-client package provides the user-facing commands like sinfo, squeue, srun, and sbatch.

On Ubuntu 24.04, these come from the universe repository at version 23.11.4.

• Verify the install on the head node by running:

munge --version

You should see a version string confirming Munge is installed.

Package not found?

Make sure the universe repository is enabled. Run sudo add-apt-repository universe && sudo apt update then try the install again.

Help me troubleshoot installing Munge on Ubuntu 24.04

Generate the Munge key and distribute it to compute nodes

Munge uses a single shared secret key to encrypt credentials. You generate this key once on the head node, then copy it to every other node so they all share the same trust boundary.

Why does every node need the same key?

When a node sends a job request, Munge encrypts the sender's identity into a credential using this key. The receiving node decrypts it with the same key to verify who sent the request. Without a matching key, nodes reject each other.

• On the head node, generate a new Munge key by running:

sudo mungekey --verbose --force

What does this command do?

The mungekey utility creates a random cryptographic key at /etc/munge/munge.key. The --force flag overwrites any existing key (useful if the package created a placeholder during install). The --verbose flag confirms the key was written successfully.

• Verify the key file exists and has the correct ownership by running:

ls -la /etc/munge/munge.key

You should see the file owned by munge:munge with restricted permissions.

• Copy the key to each compute node using scp. Run these commands from the head node:

sudo scp /etc/munge/munge.key compute1:/etc/munge/munge.key
sudo scp /etc/munge/munge.key compute2:/etc/munge/munge.key
sudo scp /etc/munge/munge.key compute3:/etc/munge/munge.key

• On each compute node, fix the ownership and permissions so Munge can read the key. SSH in and run:

sudo chown munge:munge /etc/munge/munge.key
sudo chmod 400 /etc/munge/munge.key

• Repeat the ownership fix on all 3 compute nodes.
• Now start and enable the Munge service on all 4 nodes (head and all compute nodes). Run on each:

sudo systemctl enable munge
sudo systemctl start munge

• Verify Munge is running on the head node by running:

systemctl status munge

You should see active (running) in the output.

Munge service failed to start?

Check that /etc/munge/munge.key exists, is owned by munge:munge, and has permissions 400 or 600. Run sudo journalctl -u munge to see the exact error.

Help me fix Munge not starting on my node

Install Slurm daemons and verify cross-node authentication

With Munge running on all nodes, you can now install the role-specific Slurm daemons. The head node runs slurmctld (the controller). The compute nodes run slurmd (the worker daemon).

• On the head node, install the controller daemon by running:

sudo apt install -y slurmctld

• On each of the 3 compute nodes, install the worker daemon. SSH in and run:

sudo apt install -y slurmd

What is the difference between slurmctld and slurmd?

slurmctld is the central brain of the cluster. It tracks which nodes are available, decides where to run jobs, and manages the queue.

slurmd runs on each worker node. It waits for instructions from the controller, launches jobs locally, and reports back when they finish.

Why do UIDs and GIDs matter?

Slurm requires consistent user IDs across the cluster. Because you are installing fresh on all 4 nodes from the same Ubuntu 24.04 base, the slurm and munge users created by the packages will get the same UIDs. If you ever add a node that was set up differently, you will need to create these users with matching UIDs first.

Before you run the verification command, what do you think will happen when the head node tries to send a Munge credential to a compute node over SSH?

• Test cross-node Munge authentication from the head node by running:

munge -n | ssh compute1 unmunge

You should see STATUS: Success (0) in the output. This confirms that compute1 decrypted the credential using the same shared key.

• Run the same test against compute2 and compute3 to confirm all nodes can authenticate.

Seeing STATUS: Error or connection refused?

If you see a decode error, the keys do not match. Re-copy /etc/munge/munge.key from the head node to the failing compute node, fix ownership with sudo chown munge:munge /etc/munge/munge.key, then restart Munge with sudo systemctl restart munge.

If SSH itself fails, make sure the Tailscale hostname resolves. Run ssh compute1 hostname to test basic connectivity.

Help me troubleshoot cross-node Munge authentication

Authentication is working across all your nodes. Next up, you will write the Slurm configuration file that tells the cluster what hardware it has and bring the daemons online.

Bring the Cluster Online

Munge is running and every node has its Slurm packages installed. But right now, Slurm has no idea what your cluster looks like.

Without a configuration file telling it which node is the controller, which are the workers, and what hardware they have, the daemons cannot start. In this step you will discover your hardware, write the central slurm.conf, distribute it to every node, and bring the cluster online.

In this step, get ready to:

• Discover your compute node hardware specs with slurmd -C.
• Write and distribute slurm.conf and cgroup.conf to all nodes.
• Start the cluster daemons and verify all nodes report idle.

Discover your compute node hardware

Slurm needs exact hardware details for each compute node: CPU count, socket layout, cores, threads, and memory. The slurmd daemon can report what it detects on the running machine.

• SSH into one of your compute nodes by running:

ssh compute1

• Discover the node's hardware configuration by running:

slurmd -C

You should see output similar to this:

NodeName=compute1 CPUs=8 Boards=1 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=23000

What do these values mean?

• CPUs=8 is the total logical processors (4 cores x 2 threads on your Xeon E3-1245 v5).
• CoresPerSocket=4 and ThreadsPerCore=2 describe the physical CPU layout.
• RealMemory=23000 is usable RAM in megabytes (slightly less than your 24GB due to kernel reservations).

These are the exact values you will plug into slurm.conf. Since all three compute nodes have identical hardware, you only need to run this once.

• Return to your head node by running:

exit

Write and distribute the cluster configuration

Slurm's entire cluster topology lives in a single file: /etc/slurm/slurm.conf. This file must be identical on every node in the cluster.

• Open the config file for editing on the head node by running:

sudo nano /etc/slurm/slurm.conf

• Paste the following content as the first section of the file. This sets the cluster identity, authentication, ports, paths, and scheduling:

#
# slurm.conf - Slurm configuration for homelab cluster
#
ClusterName=homelab
SlurmctldHost=head([[HEAD_TAILSCALE_IP="your head node Tailscale IP"]])

# Authentication
AuthType=auth/munge

# Ports
SlurmctldPort=6817
SlurmdPort=6818

# Paths
StateSaveLocation=/var/spool/slurmctld
SlurmdSpoolDir=/var/spool/slurmd
SlurmctldLogFile=/var/log/slurm/slurmctld.log
SlurmdLogFile=/var/log/slurm/slurmd.log
SlurmctldPidFile=/run/slurmctld.pid
SlurmdPidFile=/run/slurmd.pid

# Process tracking and task management
ProctrackType=proctrack/cgroup
TaskPlugin=task/cgroup,task/affinity

# Scheduling
SchedulerType=sched/backfill
SelectType=select/cons_tres
SelectTypeParameters=CR_Core_Memory

What does this section configure?

• SlurmctldHost=head(IP) tells every node where to find the controller. The IP in parentheses lets Slurm resolve the host over your Tailscale mesh without DNS.
• SchedulerType=sched/backfill enables the backfill scheduler which can slot short jobs into gaps behind longer ones.
• SelectType=select/cons_tres tracks consumable resources (cores and memory) per node, so Slurm knows exactly what is available.
• ProctrackType=proctrack/cgroup uses Linux cgroups to track and contain job processes.

• Continue pasting the rest of the file directly below. This section defines timers, accounting, and your node and partition layout:

# Timers
SlurmctldTimeout=120
SlurmdTimeout=300
InactiveLimit=0
MinJobAge=300
KillWait=30
WaitTime=0

# Accounting (minimal, no database)
AccountingStorageType=accounting_storage/none
JobCompType=jobcomp/filetxt
JobCompLoc=/var/log/slurm/job_completions.log
JobAcctGatherType=jobacct_gather/cgroup
JobAcctGatherFrequency=30

# User
SlurmUser=slurm

#
# Node Definitions
#
NodeName=compute[1-3] CPUs=8 Sockets=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=23000 State=UNKNOWN NodeAddr=[[COMPUTE1_IP="compute1 Tailscale IP"]],[[COMPUTE2_IP="compute2 Tailscale IP"]],[[COMPUTE3_IP="compute3 Tailscale IP"]]

#
# Partition Definitions
#
PartitionName=all Nodes=compute[1-3] MaxTime=INFINITE State=UP Default=YES

What does this section configure?

• The NodeName line declares all three compute nodes with the hardware values you discovered via slurmd -C. The NodeAddr field points to each node's Tailscale IP so Slurm can reach them across your mesh network.
• PartitionName=all creates a single partition containing all three compute nodes. MaxTime=INFINITE means jobs have no time limit. This is intentional for now. You will see why this causes scheduling problems in a later step.
• AccountingStorageType=accounting_storage/none keeps the setup simple by skipping the database-backed accounting daemon.

Find your Tailscale IPs

Run tailscale ip -4 on each node to get its stable Tailscale IPv4 address. These addresses persist unless you remove the node from your tailnet.

• Save and close the file (Ctrl+O, Enter, Ctrl+X in nano).
• Verify the file was saved by running:

head -5 /etc/slurm/slurm.conf

You should see the first five lines of your config, starting with the comment header and ClusterName=homelab.

Now create the cgroup configuration file. This controls how Slurm uses Linux cgroups to enforce resource limits on jobs.

• Create the cgroup config file by running:

sudo nano /etc/slurm/cgroup.conf

• Paste the following content:

#
# cgroup.conf - Slurm cgroup configuration
#
CgroupPlugin=autodetect
ConstrainCores=yes
ConstrainRAMSpace=yes
ConstrainSwapSpace=no
AllowedRAMSpace=100

What does cgroup.conf do?

ConstrainCores=yes ensures a job can only use the CPU cores it was allocated. ConstrainRAMSpace=yes kills jobs that exceed their memory allocation instead of letting them slow down the entire node with swap thrashing.

• Save and close the file.

Both config files must be identical on every node. Distribute them to all three compute nodes now.

• Copy both config files to all compute nodes by running:

for node in compute1 compute2 compute3; do
  sudo scp /etc/slurm/slurm.conf /etc/slurm/cgroup.conf $node:/etc/slurm/
done

You should see the scp transfer output for each node, confirming both files were copied successfully.

Permission denied when copying?

• Make sure your SSH key allows root or sudo access to the compute nodes. If scp fails, try copying as your user first, then moving the files into place with sudo mv on the remote node.
• Verify /etc/slurm/ exists on the compute nodes. If not, create it with sudo mkdir -p /etc/slurm on each compute node first.

Help me fix my scp permission errors

✔️ Awesome, I've got everything!

Great. Double check you have saved both files and distributed them to all nodes.

ⓧ I'd like to double check the full code

The full slurm.conf exceeds 30 lines. Scroll back up and confirm your file contains both sections in order: part 1 (cluster identity through scheduling) and part 2 (timers through partition definitions). Your cgroup.conf should match this:

#
# cgroup.conf - Slurm cgroup configuration
#
CgroupPlugin=autodetect
ConstrainCores=yes
ConstrainRAMSpace=yes
ConstrainSwapSpace=no
AllowedRAMSpace=100

Start the cluster and verify it works

The configuration is distributed. Time to bring the daemons online and confirm the cluster sees all its nodes.

• Create the required spool and log directories on the head node by running:

sudo mkdir -p /var/spool/slurmctld /var/log/slurm
sudo chown slurm:slurm /var/spool/slurmctld /var/log/slurm

• Start and enable the controller daemon on the head node by running:

sudo systemctl start slurmctld
sudo systemctl enable slurmctld

• Verify the controller is running by running:

sudo systemctl status slurmctld --no-pager

You should see active (running) in the output.

Seeing 'failed' or 'inactive'?

• Check the logs with sudo journalctl -u slurmctld --no-pager -n 30. Common causes include typos in slurm.conf or missing spool directories.
• Verify the slurm user owns the spool directory with ls -la /var/spool/slurmctld.

Help me troubleshoot slurmctld startup

• Now start slurmd on each compute node. Run this from the head node:

for node in compute1 compute2 compute3; do
  ssh $node "sudo mkdir -p /var/spool/slurmd /var/log/slurm && sudo chown slurm:slurm /var/spool/slurmd /var/log/slurm && sudo systemctl start slurmd && sudo systemctl enable slurmd"
done

Each command should complete silently with no error output.

Before you run the final check, think about what you expect. You configured 3 compute nodes in a partition called all. What state should those nodes be in if no jobs are running?

• Query the cluster status by running:

sinfo

You should see output like this:

PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
all*      up    infinite      3   idle  compute[1-3]

All three compute nodes show idle which means they are online, registered with the controller, and ready to accept jobs. The * after the partition name means it is the default partition.

Nodes showing as down or unknown?

• Check slurmd logs on the compute node: sudo journalctl -u slurmd --no-pager -n 20. Look for config mismatches or connectivity errors.
• Verify the slurm user has the same UID on all nodes by running id slurm on each. Mismatched UIDs cause authentication failures.
• If nodes are in down state, resume them with sudo scontrol update nodename=compute[1-3] state=idle after fixing the underlying issue.
• Confirm Tailscale connectivity by pinging each compute node's Tailscale IP from the head node.

Help me troubleshoot my node status

Your cluster is alive. Three compute nodes are registered, healthy, and waiting for work. Next up, you will submit your first parallel job and watch Slurm dispatch it across all three machines.

Run Your First Parallel Job

Your cluster is online and all three compute nodes are sitting idle. That green status is promising, but it does not prove that jobs actually execute across multiple physical machines.

Time to put the cluster to real use. You will submit interactive and batch jobs, then flood the queue to see how Slurm distributes work across your three compute nodes.

In this step, get ready to:

• Run an interactive parallel command across all 3 nodes.
• Write and submit a batch script with sbatch.
• Flood the queue and observe job distribution across the cluster.

Run an interactive job across all nodes

The srun command lets you run a command interactively across allocated nodes. The output comes straight back to your terminal so you can see which machines responded.

Before you run this, what do you expect to see? Think about how many nodes you are requesting and what the hostname command returns.

• Run your first parallel job across all 3 compute nodes:

srun --nodes=3 hostname

You should see three different hostnames printed, one from each compute node. Each line confirms a different physical machine executed the command.

What does this prove?

Each hostname in the output corresponds to a different compute node in your cluster. Seeing three distinct names confirms that Slurm successfully allocated resources on all three machines and ran your command in parallel across them.

Not seeing three hostnames?

If you see fewer than 3 hostnames, check that all nodes show as idle with sinfo. A node in down or drain state cannot accept jobs.

If you see a timeout error, verify that slurmd is running on all compute nodes with sudo systemctl status slurmd.

Still stuck? Help me troubleshoot why srun is not reaching all my compute nodes.

Write and submit a batch script

Interactive jobs with srun tie up your terminal while they run. Batch scripts let you submit work and walk away. Slurm queues the job, runs it when resources are available, and writes output to a file.

• Create your first batch script by running this command on your head node:

cat << 'EOF' > ~/hello.sh
#!/bin/bash
#SBATCH --job-name=hello-slurm
#SBATCH --output=hello_%j.out
#SBATCH --ntasks=1
#SBATCH --time=00:01:00

echo "Hello from $(hostname) at $(date)"
echo "Job ID: $SLURM_JOB_ID"
echo "Node: $SLURM_NODELIST"
sleep 5
echo "Done!"
EOF

What does this script do?

• The #SBATCH lines are directives that Slurm reads before running the script. They set the job name, output file pattern, task count, and time limit.
• The %j in the output filename gets replaced with the job ID, so each run creates a unique file.
• The sleep 5 gives you time to observe the job in the queue before it completes.

• Verify the file was created by running:

ls ~/hello.sh

You should see hello.sh listed in the output.

✔️ Awesome, I've got everything!

Great. Your batch script is ready to submit.

ⓧ I'd like to double check the full code

#!/bin/bash
#SBATCH --job-name=hello-slurm
#SBATCH --output=hello_%j.out
#SBATCH --ntasks=1
#SBATCH --time=00:01:00

echo "Hello from $(hostname) at $(date)"
echo "Job ID: $SLURM_JOB_ID"
echo "Node: $SLURM_NODELIST"
sleep 5
echo "Done!"

• Submit the batch script to Slurm by running:

sbatch ~/hello.sh

You should see a confirmation like Submitted batch job 1 (your job number may differ).

• Immediately check the queue to watch the job transition through states:

squeue

You should see your job listed with state R (RUNNING) or PD (PENDING). Wait a few seconds and run squeue again. After the 5-second sleep finishes, the job disappears from the queue because it completed.

• Read the output file to confirm the job ran successfully:

cat ~/hello_*.out

You should see the hostname of the compute node that ran your job, the job ID, and the node name. This confirms Slurm dispatched your script to a compute node and captured the output.

No output file appearing?

If you do not see a hello_*.out file, check that squeue shows the job completed (no longer listed). The file is only written after the job finishes.

If the job failed immediately, run scontrol show job <jobid> to see the reason. Common causes include a missing home directory on the compute node or permission issues with the output path.

Need more help? Help me figure out why my sbatch job is not producing an output file.

Flood the queue and observe distribution

A single job proves one node works. Submitting many jobs at once reveals how Slurm distributes work across the cluster. With 3 compute nodes and 6 jobs, you should see Slurm spread the load.

• Submit hello.sh six times in rapid succession by running:

for i in {1..6}; do sbatch ~/hello.sh; done

You should see six "Submitted batch job" confirmations, each with a different job ID.

• Immediately check the queue to see how Slurm distributed the jobs:

squeue

You should see multiple jobs running simultaneously across different nodes in the NODELIST column. Some jobs may show compute1, others compute2 or compute3.

• Check node utilization to see the cluster under load:

sinfo

Nodes actively running jobs now show state alloc or mix instead of idle. This confirms Slurm is actively using your hardware.

• Wait about 15 seconds for all jobs to complete, then verify job history. Before you run the next command, do you expect to see all 6 jobs listed as completed on different nodes?

sacct

You should see your completed jobs listed with their assigned node names, confirming that work ran on different machines across the cluster.

What does sacct show you?

The sacct command displays job accounting data. It shows job IDs, names, states, and which nodes ran each job. Seeing different node names across your jobs proves the scheduler distributed work rather than piling everything onto one machine.

If sacct returns empty output, your cluster does not have full accounting storage configured. You can still verify job distribution by checking the output files directly:

grep "Hello from" ~/hello_*.out

This shows which hostname ran each job. You should see a mix of your compute node names across the output files.

All jobs ran on the same node?

If all 6 jobs landed on one node, that is expected with single-task jobs. Slurm's default scheduling may pack jobs onto the first available node. Each job only requests 1 task and 1 CPU, so one 8-CPU node can handle all of them simultaneously.

To force distribution, you can resubmit with --exclusive or request more CPUs per job. The important thing is that Slurm processed all 6 jobs and they completed successfully.

Want help understanding Slurm's scheduling decisions? Explain how Slurm decides which node to assign a job to when multiple nodes have available resources.

Your cluster is running real parallel workloads. Next up, you will push the scheduler to its limits by overloading the queue and discovering why time limits matter.

Overload the Scheduler

Your cluster proved it can distribute work across three physical machines. The next question is how it handles contention.

Real HPC clusters serve dozens of users submitting jobs simultaneously. When demand exceeds capacity, the scheduler decides who waits and who runs. Time to flood your queue and see how your current configuration handles the pressure.

In this step, get ready to:

• Submit an exclusive job that locks all three compute nodes.
• Flood the queue with short jobs and observe what the scheduler does with them.
• Identify why the backfill scheduler cannot help in the current configuration.

Lock down the cluster with an exclusive job

The --exclusive flag tells Slurm to give a job sole access to every resource on its allocated nodes. Combined with --nodes=3, this locks out your entire cluster for one job.

• Submit a long-running exclusive job that claims all three compute nodes by running this command:

sbatch --nodes=3 --exclusive --job-name=blocker --wrap="sleep 300"

You should see a confirmation like Submitted batch job 12345 with your own job ID.

• Confirm the blocker job is running by checking the queue:

squeue

You should see your blocker job with state R (Running) allocated across all three compute nodes. It will sleep for 5 minutes, holding every CPU on every node.

What does --exclusive do?

When a job runs with --exclusive, Slurm reserves all CPUs and memory on the allocated nodes for that job alone. No other job can share those nodes, even if cores are idle. This simulates a real scenario where a large parallel computation monopolises the cluster.

Flood the queue with short jobs

While the blocker holds every node, you will submit a batch of tiny jobs that each need only 1 CPU and 10 seconds. These should be trivial to run, if only there were somewhere to put them.

• Create a short batch script called flood.sh by running this command:

cat << 'EOF' > ~/flood.sh
#!/bin/bash
#SBATCH --job-name=flood
#SBATCH --output=flood_%j.out
#SBATCH --ntasks=1
#SBATCH --time=00:01:00

echo "Flood job on $(hostname) - Job $SLURM_JOB_ID"
sleep 10
EOF
chmod +x ~/flood.sh

What does this script do?

Each flood job requests a single task, sleeps for just 10 seconds, and prints which node ran it. The --time=00:01:00 directive sets a 1-minute time limit on the job itself. These are intentionally tiny so you can clearly see them stuck waiting.

• Submit 5 copies of the flood script by running this loop:

for i in {1..5}; do sbatch ~/flood.sh; done

You should see 5 submission confirmations, one per line.

Before you check the queue, predict what you think will happen. Will the short flood jobs start running, or will they wait? Why?

• Check the queue to see what the scheduler decided:

squeue

You should see all 5 flood jobs stuck in PD (PENDING) state with the reason (Resources) in the NODELIST(REASON) column. The blocker job still shows R (Running).

Why can't the short jobs run?

The blocker has exclusive access to all 3 nodes. Even though the flood jobs only need 1 CPU each, there is literally nowhere to put them. They wait.

Here is the deeper problem. Your partition is configured with MaxTime=INFINITE. The backfill scheduler works by predicting when running jobs will end, then slotting smaller jobs into those gaps. With no time limit, Slurm cannot predict when the blocker will finish. It has no gap to fill, so backfill is impossible.

Even though you know the blocker sleeps for only 300 seconds, the scheduler does not know that. It sees INFINITE and gives up.

• Run squeue a few more times over the next 30 seconds to confirm nothing changes. The flood jobs remain stuck.

Flood jobs not showing as PENDING?

If you see the flood jobs already running, your blocker job may have finished or failed before you submitted the flood. Check that the blocker is still listed with state R in squeue output. If it completed, cancel everything with scancel --user=$USER and resubmit the blocker first, then the flood jobs.

Why aren't my flood jobs showing as PENDING in squeue?

Clear the queue

You have seen the scheduling wall firsthand. The short jobs could finish in 10 seconds, but the scheduler has no information about when the long job ends. Without that information, backfill cannot do its job.

• Cancel all your jobs at once by running:

scancel --user=$USER

• Verify the queue is empty:

squeue

You should see no jobs listed. The queue is clean.

What did you just prove?

A single partition with MaxTime=INFINITE gives the backfill scheduler zero information to work with. It cannot predict when resources free up, so it cannot fill gaps with shorter jobs. The fix is to give the scheduler the information it needs: time limits and separate partitions.

✔️ Awesome, I've got everything!

Great. You have experienced the scheduling wall and your queue is clear.

ⓧ I'd like to double check the full code

Here is the flood.sh script you created. Confirm it matches:

#!/bin/bash
#SBATCH --job-name=flood
#SBATCH --output=flood_%j.out
#SBATCH --ntasks=1
#SBATCH --time=00:01:00

echo "Flood job on $(hostname) - Job $SLURM_JOB_ID"
sleep 10

You just felt the exact problem that real HPC administrators solve every day. In the next step, you will split your cluster into partitions with enforced time limits, giving the backfill scheduler the information it needs to fill gaps intelligently.

Tame the Queue with Partitions and Time Limits

In the last step, you watched short jobs sit blocked behind a long-running job with no way forward. The backfill scheduler could not help because MaxTime=INFINITE gives it zero information about when resources will free up.

The fix is giving the scheduler what it needs: time limits. By splitting your nodes into partitions with enforced MaxTime values, Slurm can predict when resources become available and slot shorter jobs into the gaps.

In this step, get ready to:

• Edit slurm.conf to define debug and batch partitions with time limits.
• Apply the new configuration live with scontrol reconfigure.
• Prove the fix works by running long and short jobs on separate partitions simultaneously.

Edit slurm.conf to define two partitions

You need two partitions: a small "debug" partition for quick interactive work, and a larger "batch" partition for longer compute jobs. The debug partition gets a strict 5-minute time limit and one dedicated node, while batch spans all three nodes with a 60-minute cap.

• Open /etc/slurm/slurm.conf on your head node in your text editor:

sudo nano /etc/slurm/slurm.conf

• Scroll to the # Node Definitions comment section.
• Find this comment block:

#
# Node Definitions
#

• Add a helpful reminder line so the section now looks like this:

#
# Node Definitions
# Replace Tailscale IPs with your actual values from `tailscale ip -4`
#

• Now scroll down to the # Partition Definitions section.
• Find this line:

PartitionName=all Nodes=compute[1-3] MaxTime=INFINITE State=UP Default=YES

• Delete that line and replace it with these two partition definitions:

PartitionName=debug Nodes=compute1 MaxTime=00:05:00 State=UP Default=YES
PartitionName=batch Nodes=compute[1-3] MaxTime=01:00:00 State=UP

What do these partitions do?

The debug partition isolates compute1 for quick jobs with a 5-minute maximum. It is the default partition, so jobs without an explicit --partition flag land here.

The batch partition spans all three compute nodes and allows jobs up to 60 minutes. Because each partition has a finite MaxTime, the backfill scheduler can now predict exactly when resources will free up and fill gaps with shorter jobs.

• Save and exit the editor (in nano: Ctrl+O, Enter, Ctrl+X).

Now distribute the updated config to all compute nodes so every node has an identical copy.

• Copy the updated slurm.conf to each compute node by running these commands:

sudo scp /etc/slurm/slurm.conf compute1:/etc/slurm/slurm.conf
sudo scp /etc/slurm/slurm.conf compute2:/etc/slurm/slurm.conf
sudo scp /etc/slurm/slurm.conf compute3:/etc/slurm/slurm.conf

Why must the config be identical everywhere?

Slurm requires slurm.conf to have consistent contents on every node in the cluster. If the head node and a compute node disagree about partition definitions, the compute node will refuse to register properly.

Permission denied during scp?

If your SSH user cannot write directly to /etc/slurm/ on the remote nodes, copy to a temp path first, then move it with ssh:

scp /etc/slurm/slurm.conf compute1:/tmp/slurm.conf
ssh compute1 sudo cp /tmp/slurm.conf /etc/slurm/slurm.conf

Repeat for each compute node. Help me fix scp permission errors to my compute nodes.

Apply the new configuration live

The config files are in place, but the running daemons are still using the old single-partition layout. You can reload the configuration without restarting any daemons or interrupting running jobs.

• Tell Slurm to re-read its configuration by running this command on the head node:

sudo scontrol reconfigure

What does scontrol reconfigure do?

This command instructs all Slurm daemons (slurmctld and every slurmd) to re-read slurm.conf without restarting. In Slurm 23.11, it spawns new daemon processes and hands control to them once they start successfully. If the new config has errors, the old daemons keep running.

The command produces no output on success. Verify the new partitions are live by checking cluster status.

• Confirm both partitions appear by running:

sinfo

You should see two partitions listed: debug (with compute1 in idle state) and batch (with compute[1-3] in idle state). The debug partition will have an asterisk (*) indicating it is the default.

Nodes showing as down or drained?

If sinfo shows nodes in a down or drain state, the config file on that node may not match the head node. SSH into the problem node and compare its /etc/slurm/slurm.conf with the head node's copy.

If the configs match but a node is still down, bring it back with:

sudo scontrol update NodeName=compute1 State=IDLE

Help me troubleshoot why my nodes show as down after reconfigure.

Prove backfill scheduling works across partitions

Time to prove the fix. You will submit a long-running job to the batch partition that ties up two nodes, then submit short jobs to the debug partition and watch them start immediately on the dedicated debug node.

• Submit a long job to the batch partition by running:

sbatch --partition=batch --nodes=2 --time=10:00 --wrap="sleep 300"

What does this job do?

This submits a 300-second sleep across 2 of the 3 batch nodes with a 10-minute time limit. The --time=10:00 flag tells the scheduler exactly when those resources will be available again. That is the information backfill needs.

• Confirm the long job is running by checking the queue:

squeue

You should see your job in the R (running) state on the batch partition, consuming 2 nodes.

• Now submit two short jobs to the debug partition by running:

sbatch --partition=debug --wrap="echo 'Quick job on' $(hostname); sleep 5"
sbatch --partition=debug --wrap="echo 'Quick job on' $(hostname); sleep 5"

Before you check the queue, think about what you expect to see. In the previous step, short jobs got stuck behind the long one. What should happen now that debug has its own dedicated node?

• Check the queue again:

squeue

The short jobs are running (or already completed) on compute1 in the debug partition. They did not wait for the batch job to finish. That is partition isolation in action.

• Inspect the scheduling details of the batch job to see how Slurm made its decisions. Grab the job ID from the squeue output and run:

scontrol show job <jobid>

What does scontrol show job reveal?

This output shows the full scheduling record for a job: which partition it ran on, the exact nodes allocated, its time limit, start time, and end time. Look for the TimeLimit and EndTime fields. These are the values the backfill scheduler uses to predict when resources free up.

• Clean up when you are done observing by running:

scancel --user=$USER

Short jobs stuck in PENDING?

If your debug jobs are pending instead of running, check that compute1 is idle and not already claimed by the batch job. Run sinfo -p debug to verify the debug node is available.

Also confirm you used --partition=debug when submitting the short jobs. Without it, they go to the default partition which is also debug, but double check.

Help me figure out why my debug partition jobs are stuck pending.

✔️ Awesome, I've got everything!

Great. Double check you have saved your /etc/slurm/slurm.conf and distributed it identically to all nodes.

ⓧ I'd like to double check the full code

Your /etc/slurm/slurm.conf is a long file. Scroll through it and confirm these sections are present in order:

• ClusterName, SlurmctldHost, AuthType at the top.
• Ports section (6817, 6818).
• Paths section (StateSaveLocation, SlurmdSpoolDir, log files, pid files).
• Process tracking (proctrack/cgroup) and TaskPlugin.
• Scheduling section (sched/backfill, select/cons_tres, CR_Core_Memory).
• Timers section.
• Accounting section (accounting_storage/none, jobcomp/filetxt).
• SlurmUser=slurm.
• Node Definitions with the comment about Tailscale IPs, then your NodeName line.
• Partition Definitions with exactly two lines: PartitionName=debug and PartitionName=batch.

The partition lines should read:

PartitionName=debug Nodes=compute1 MaxTime=00:05:00 State=UP Default=YES
PartitionName=batch Nodes=compute[1-3] MaxTime=01:00:00 State=UP

If anything is missing, edit the file, re-distribute to all nodes, and run sudo scontrol reconfigure again.

Secret mission

Configure GPU Scheduling With GRES

Your compute nodes have NVIDIA Quadro M620 GPUs sitting idle. Real HPC clusters let users explicitly request GPUs in their jobs. In this secret mission, you will configure GRES (Generic RESource) scheduling so Slurm becomes GPU-aware, letting jobs request hardware accelerators with --gres=gpu:1 and routing them to nodes with available GPUs.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally on your homelab nodes, so there are no ongoing costs.

Resources you used:

• Slurm controller service (slurmctld) running on the head node.
• Slurm compute daemon (slurmd) running on all 3 compute nodes.
• Munge authentication service running on all 4 nodes.
• Configuration files: /etc/slurm/slurm.conf, /etc/slurm/cgroup.conf, and /etc/slurm/gres.conf on all nodes.
• Batch scripts: ~/hello.sh, ~/flood.sh, and ~/gpu-test.sh on the head node.
• Spool directories: /var/spool/slurmctld and /var/spool/slurmd on respective nodes.

Keep everything running

No action needed. Choose this if you want to keep experimenting with your cluster. Your services are enabled and will persist across reboots.

• Your cluster is ready for further experimentation: MPI jobs, job arrays, or adding accounting with slurmdbd.
• Run sinfo anytime to confirm nodes are still idle and responsive.
• The Munge key on all nodes remains valid indefinitely.

Pause - I'll come back to this later

Stop the services to free up system resources. You can restart them anytime without reconfiguring.

• On the head node, stop the controller:
• On each compute node, stop the compute daemon:

# On the head node
sudo systemctl stop slurmctld

# On each compute node (run on compute1, compute2, compute3)
sudo systemctl stop slurmd

• When you want to resume, start the services in the same order (slurmctld first, then slurmd on each compute node).
• Your configuration files and spool state remain intact.

Delete - I don't want to use this again

Remove all Slurm and Munge packages, configuration files, and spool directories from every node.

• On all 4 nodes (head and each compute node), purge the packages:

sudo apt purge slurmctld slurmd slurm-client munge -y
sudo apt autoremove -y

• On all 4 nodes, remove the configuration directory and spool directories:

sudo rm -rf /etc/slurm/
sudo rm -rf /var/spool/slurmctld
sudo rm -rf /var/spool/slurmd
sudo rm -rf /var/log/slurm/

• On the head node, remove the batch scripts and their output files:

rm -f ~/hello.sh ~/flood.sh ~/gpu-test.sh
rm -f ~/hello_*.out ~/flood_*.out ~/gpu_*.out

• On the head node, remove the Munge key:

sudo rm -rf /etc/munge/

What about the slurm user?

The apt purge command removes the packages and their configuration. The slurm system user may remain on your system. This is harmless. If you want to remove it, run sudo userdel slurm on each node.

Nice Work!

Nice Work!

You just built a production-style Slurm cluster from scratch across your homelab. You have a real job scheduler distributing work across physical machines, with partitions, time limits, and backfill scheduling working exactly like they do in university and enterprise HPC environments.

You learned how to:

• Build a real 4-node Slurm cluster with Munge cross-node authentication, shared key distribution, and Tailscale addressing.
• Submit parallel jobs with srun and sbatch, watching work distribute across physical machines and querying cluster state with sinfo, squeue, and scontrol.
• Hit the scheduling wall firsthand, then tame the queue by creating debug and batch partitions with enforced time limits that enable intelligent backfill scheduling.
• Secret Mission: Configured GRES GPU scheduling so jobs can explicitly request your Quadro M620 GPUs via --gres=gpu:1.

Ready to quiz yourself?