Jenkins Disaster Recovery Replication

Replicate Jenkins to a warm standby server using rsync and cron.

Introduction

30 Second Summary

Your Jenkins server holds everything your team needs to build and ship software. If that server goes down tomorrow, every job, build record, and plugin you configured disappears with it.

In this project, you will set up automated disaster recovery replication for your Jenkins server using rsync and cron. Every 3 hours, the primary server pushes a full copy of its data to a replica so you always have a warm backup ready to take over.

What You'll Build

You visit your replica's Jenkins UI at 192.168.1.230 and find every job, workspace, and build record from production mirrored exactly, ready to take over if the primary ever fails.

By the end of this project, you'll have:

  • A fully synced replica at 192.168.1.230 with an exact copy of every job, workspace, build record, and plugin from your primary server.
  • An automated sync that runs every 3 hours using a cron job, logging each run so you can verify your replication is healthy.
  • A tested manual switchover procedure where you stop the primary, visit the replica, and confirm it works as a fully functional standalone Jenkins instance.
  • Secret Mission: Add HAProxy for automatic failover so traffic routes to the replica automatically when the primary goes down.

Are there any prerequisites?

This project assumes you have a primary Jenkins server running at 192.168.1.253 (accessible at http://jenkins.polus.com/) and a second on-prem Linux server at 192.168.1.230 available as a replica. Basic familiarity with the Linux command line is also helpful.

Before We Start

Before we start building, let's lock in what you're about to set up and why it matters.

Set Up SSH Access and Prepare the Replica

Your production Jenkins at 192.168.1.253 holds every job definition, build record, and plugin your team depends on. To replicate all of that to the replica server, you'll use rsync, a tool that transfers files over SSH.

For the automated sync to work, SSH must connect between servers without stopping for a password. In this step, you'll set up passwordless SSH and verify that the replica is ready to receive data.

In this step, get ready to:
  • Generate an SSH key pair and copy it to the replica for passwordless access.
  • Verify Jenkins is installed and running on the replica server.
  • Stop Jenkins on the replica to prepare for the first data sync.
Generate an SSH key and enable passwordless access

rsync sends files over SSH. If SSH prompts for a password, the automated cron job you set up later will hang and never complete. An SSH key pair lets the primary authenticate to the replica automatically.

  • Open a terminal on your primary server at 192.168.1.253.
  • Generate an ed25519 SSH key pair by running:
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N ""

What does this command do?

  • -t ed25519 creates a modern, high-security key type.
  • -f ~/.ssh/id_ed25519 saves the key pair to the default SSH directory.
  • -N "" sets an empty passphrase so the key works without manual entry during automated syncs.

Already have an SSH key?

If ~/.ssh/id_ed25519 already exists on the primary, you can skip the ssh-keygen command and move straight to copying the key.

  • Copy the public key to the replica by running:
ssh-copy-id [[SSH_USER="your SSH username on the replica"]]@192.168.1.230

Expect a password prompt

ssh-copy-id will ask for your password on the replica one final time. Enter it when prompted. After this, all future SSH connections from the primary use the key pair instead.

Before you run the next command, what do you expect to see if the key was copied successfully?

  • Test the passwordless connection by running:
ssh [[SSH_USER="your SSH username on the replica"]]@192.168.1.230 'hostname'

The replica's hostname prints immediately, with no password prompt. That confirms passwordless SSH is working.

Still seeing a password prompt?

  • Make sure you ran ssh-copy-id with the correct username for the replica. The username must match the account you log in with on 192.168.1.230.
  • Check that ~/.ssh/id_ed25519.pub exists on the primary. If ssh-keygen was skipped and no key file exists, go back and run it.
  • Verify that the SSH service is running on the replica. Run sudo systemctl status sshd on 192.168.1.230 to confirm.

Why isn't my passwordless SSH connection working between my Jenkins servers?

Verify Jenkins is installed on the replica

The replica needs its own independent Jenkins installation. During a switchover, it runs as a standalone server, so it cannot rely on the primary for anything.

  • Open a browser and visit http://192.168.1.230:8080.

✔️ I see the Jenkins login page

Jenkins is installed and running on the replica. You're all set to move on.

ⓧ The page doesn't load

Jenkins is either not installed or not running on the replica. Install it using the same package manager method used on your primary server at 192.168.1.253.

  • SSH into the replica and run the same installation commands you used for the primary (e.g. apt or yum package manager steps).
  • Once installed, start Jenkins by running:
sudo systemctl start jenkins
  • Wait a minute for Jenkins to fully start, then refresh http://192.168.1.230:8080 in your browser. You should now see the Jenkins login page.
  • Confirm Jenkins is responding programmatically by running this from the primary:
curl -s -o /dev/null -w "%{http_code}" http://192.168.1.230:8080/login

You should see 200 printed in your terminal. That confirms Jenkins on the replica is fully started and serving requests.

Seeing a different HTTP code?

  • A 403 means Jenkins is running but blocking the request. This is OK for now. The important thing is that Jenkins is reachable.
  • A 000 or connection error means Jenkins hasn't finished starting. Wait 30 seconds and try the curl command again.

Why can't I reach Jenkins on my replica server at port 8080?

Stop Jenkins on the replica

Jenkins continuously writes to its home directory as builds run and configurations change. If rsync copies files while Jenkins is running on the replica, you risk corrupted XML files and inconsistent state.

  • On the replica server (192.168.1.230), stop Jenkins by running:
sudo systemctl stop jenkins

Running commands from the primary?

If you're working from the primary server, SSH into the replica first using the passwordless connection you just set up. Then run the stop command directly.

What do you think will happen if you try to access http://192.168.1.230:8080 now?

  • Open http://192.168.1.230:8080 in your browser to verify.

The page should fail to load. That confirms Jenkins is stopped on the replica and safe for the first full data sync.

Page still loading?

  • Jenkins can take a few seconds to fully shut down. Wait 10 seconds and refresh the page.
  • If the page still loads, confirm the stop command ran on the correct server (192.168.1.230, not 192.168.1.253). Stopping Jenkins on the primary would take down your production instance.

Help me confirm Jenkins is stopped on my replica server.

Your servers can now talk to each other securely, and the replica is ready to receive data. Next up, you'll run the first full sync and see your production jobs appear on the replica for the first time.

Run the First Full Sync

In Step 1 you set up passwordless SSH between your primary and replica servers and confirmed Jenkins is installed on the replica.

Your primary Jenkins at 192.168.1.253 holds everything your team depends on: jobs, workspaces, build history, plugins, and system configuration. The replica has a fresh Jenkins installation with none of that data. In this step, you will copy the entire JENKINS_HOME directory from the primary to the replica so both servers have identical data.

In this step, get ready to:
  • Run the initial full rsync from the primary to the replica.
  • Start Jenkins on the replica and confirm it is serving your data.
  • Cross-check specific jobs and plugins to verify the sync is accurate.
Run the initial rsync from primary to replica

The rsync command copies files between servers incrementally. This first run transfers everything since the replica's JENKINS_HOME is empty.

  • On your primary server (192.168.1.253), transfer the entire JENKINS_HOME to the replica by running this command:
sudo rsync -avz --delete -e ssh /var/lib/jenkins/ [[REPLICA_USER="your-ssh-user-here"]]@192.168.1.230:/var/lib/jenkins/

What does this command do?

  • -a (archive) preserves file permissions, ownership, and timestamps so Jenkins can read everything correctly on the replica.
  • -v (verbose) shows you every file being transferred so you can watch the progress.
  • -z compresses the data during transfer, speeding things up over the network.
  • --delete removes any files on the replica that no longer exist on the primary. This prevents the replica from accumulating stale data.
  • The trailing / on the source path tells rsync to copy the contents of the directory, not the directory itself.

This first run transfers the entire JENKINS_HOME and may take several minutes depending on its size. You will see a stream of filenames as each file is copied.

When rsync finishes, you should see a summary at the bottom showing the total number of files sent and bytes transferred.

Seeing permission denied or SSH errors?

  • Make sure you are running the command with sudo. The /var/lib/jenkins/ directory is owned by the jenkins user, so root access is needed to read all files.
  • If SSH fails, confirm passwordless access still works by running ssh user@192.168.1.230 'hostname' from the primary.
  • Check that the replica has enough disk space to hold the entire JENKINS_HOME. Run df -h on 192.168.1.230 to verify available space.

Why is my rsync failing with permission or SSH errors?

Start Jenkins on the replica and verify it is serving

Now that the data is on the replica, you need to start Jenkins so it reads the synced JENKINS_HOME and serves your jobs.

  • On the replica server (192.168.1.230), start Jenkins by running:
sudo systemctl start jenkins

Jenkins takes a moment to fully start. Check whether it is ready by running this command on the replica:

  • Check that Jenkins is responding by running:
curl -s -o /dev/null -w "%{http_code}" http://192.168.1.230:8080/login

What does this command do?

This curl command sends a request to the Jenkins login page and prints only the HTTP status code. When Jenkins is fully started and serving, it returns 200.

If you see 000 or a connection error, wait 15-20 seconds and run the command again. Jenkins needs time to load all the synced data.

Once you see 200, Jenkins is ready.

Still not getting 200 after a minute?

  • Check that Jenkins is actually running with sudo systemctl status jenkins. If it shows "failed", check the Jenkins logs for errors.
  • Verify no firewall is blocking port 8080 on the replica.

Why isn't Jenkins starting on my replica server after rsync?

Cross-check jobs and plugins between servers

Getting an HTTP 200 confirms Jenkins is running, but you need to confirm the data actually matches. This is where you verify the sync worked end to end.

Before you open the replica's dashboard, what do you expect to see? Will it look like a fresh Jenkins install, or an exact mirror of your primary?

  • Open your browser and visit http://192.168.1.230:8080.
  • Confirm that the Jenkins dashboard shows the same jobs listed on your primary at http://jenkins.polus.com/.
  • Pick a job on the primary and note its latest build number and result (pass or fail).
  • Open the same job on the replica and verify the same build numbers and results appear in the build history.
  • On the primary, navigate to Manage Jenkins, then Plugins, and note a few installed plugins.
  • On the replica, navigate to Manage Jenkins, then Plugins, and confirm the same plugins are listed.
  • On the replica, navigate to Manage Jenkins, then System, and verify the system configuration matches the primary.

Your replica's dashboard is an exact mirror of the primary. Every job, build record, plugin, and system setting has been faithfully copied.

What about builds that were running during the sync?

Any build that was actively running on the primary at the moment you ran rsync may appear in a partial state on the replica. Completed builds and their full history are synced cleanly. The next sync will pick up anything that finished after this run.

Your replica now holds an exact copy of your production Jenkins. Next up, you will automate this sync so it runs every 3 hours without any manual intervention.

Automate Replication Every 3 Hours

Your replica at 192.168.1.230 now holds a full copy of your production Jenkins data. The problem is that copy is already stale. Every new build, every config change, every plugin update on the primary makes the replica fall further behind.

Running rsync manually every time defeats the purpose of disaster recovery. You need a script that runs unattended every 3 hours, handles stopping and starting Jenkins on the replica cleanly, excludes runtime files that waste bandwidth, and logs every run for audit.

In this step, get ready to:
  • Create a sync script that stops the replica, runs rsync with proper exclusions, restarts the replica, and logs every step.
  • Test the script manually and confirm it logs a successful sync.
  • Schedule the script with cron to run automatically every 3 hours.
Create the sync script directory and file

The sync script needs a dedicated home on the primary server. It also needs to handle three jobs cleanly: stop Jenkins on the replica so rsync does not corrupt data, transfer the files, and start Jenkins back up.

  • On the primary server (192.168.1.253), create the script directory by running this command:
sudo mkdir -p /opt/jenkins-sync
  • Create and open the script file by running:
sudo nano /opt/jenkins-sync/sync-jenkins.sh

This opens the nano text editor with a blank file. You will paste the script in two chunks.

  • Paste the first section of the script, which sets up variables, logs the start time, and stops Jenkins on the replica before syncing:
#!/bin/bash
# Jenkins JENKINS_HOME replication script
# Syncs from primary (192.168.1.253) to replica (192.168.1.230)
# Run via cron every 3 hours

REPLICA_USER="[[SSH_USER=\"your SSH username for 192.168.1.230\"]]"
REPLICA_IP="192.168.1.230"
JENKINS_HOME="/var/lib/jenkins"
LOG_FILE="/var/log/jenkins-sync.log"

echo "$(date): Starting Jenkins sync to replica" >> "$LOG_FILE"

# Stop Jenkins on replica before sync
ssh "$REPLICA_USER@$REPLICA_IP" 'sudo systemctl stop jenkins' 2>> "$LOG_FILE"

if [ $? -ne 0 ]; then
    echo "$(date): WARNING - Failed to stop Jenkins on replica, proceeding with sync" >> "$LOG_FILE"
fi

What does this code do?

  • REPLICA_USER, REPLICA_IP, JENKINS_HOME, and LOG_FILE store the connection details and paths so they are easy to update in one place.
  • The echo line writes a timestamped "Starting" message to the log file before any work begins.
  • The ssh command stops Jenkins on the replica via your passwordless SSH connection. Jenkins must be stopped during sync to prevent data corruption from two instances writing to the same data.
  • The if [ $? -ne 0 ] check logs a warning if stopping Jenkins fails, but continues anyway. A failed stop is worth noting but should not block the sync entirely.
  • Directly below the code you just pasted, add the second section. This runs the rsync sync and restarts Jenkins on the replica after it finishes:

# Sync JENKINS_HOME excluding runtime files
rsync -avz --delete \
    --exclude='*.log' \
    --exclude='*.tmp' \
    --exclude='*.pid' \
    --exclude='war/' \
    --exclude='.cache/' \
    -e ssh \
    "$JENKINS_HOME/" "$REPLICA_USER@$REPLICA_IP:$JENKINS_HOME/" 2>> "$LOG_FILE"

SYNC_STATUS=$?

# Start Jenkins on replica after sync
ssh "$REPLICA_USER@$REPLICA_IP" 'sudo systemctl start jenkins' 2>> "$LOG_FILE"

if [ $SYNC_STATUS -eq 0 ]; then
    echo "$(date): Sync completed successfully" >> "$LOG_FILE"
else
    echo "$(date): Sync FAILED with exit code $SYNC_STATUS" >> "$LOG_FILE"
fi

What does this code do?

  • The rsync command uses the same flags you ran manually in Step 2 (-avz --delete), plus --exclude patterns for runtime files that Jenkins regenerates on startup.
  • Excluding war/, .cache/, *.tmp, *.pid, and *.log saves bandwidth and prevents startup issues on the replica.
  • SYNC_STATUS=$? captures rsync's exit code so the script can log whether the sync succeeded or failed.
  • After the sync, the script restarts Jenkins on the replica and writes either a "Sync completed successfully" or a "Sync FAILED" message with the exit code.
  • Save the file by pressing Ctrl+O, then Enter to confirm the filename.
  • Exit nano by pressing Ctrl+X.
  • Make the script executable by running:
sudo chmod +x /opt/jenkins-sync/sync-jenkins.sh
  • Confirm the script exists and is executable by running:
ls -l /opt/jenkins-sync/sync-jenkins.sh

You should see -rwxr-xr-x (or similar) at the start of the line, confirming the execute permission is set.

✔️ Awesome, I've got everything!

Great. Make sure you have saved sync-jenkins.sh and confirmed it is executable.

ⓧ I'd like to double check the full code

The full file is made up of the two chunks you pasted above. Scroll up and confirm both are present in order:

  • Chunk 1 (script header, variables, opening log, stop Jenkins, and error check). Starts with #!/bin/bash and ends with fi.
  • Chunk 2 (rsync sync with excludes, start Jenkins, status logging). Starts with # Sync JENKINS_HOME excluding runtime files and ends with fi.

You can also reopen the file to verify by running:

cat /opt/jenkins-sync/sync-jenkins.sh

File not found or not executable?

  • Make sure you used sudo with both mkdir and nano. The /opt/ directory is owned by root.
  • If the permissions do not show an x, re-run sudo chmod +x /opt/jenkins-sync/sync-jenkins.sh.
  • help me troubleshoot my sync-jenkins.sh file permissions
Test the script manually

Before handing this script to cron, you need to confirm it works end to end. A manual test run will write to the log file and give you a clear success or failure message.

Before you run the script, what do you expect to happen on the replica server during the sync?

  • Run the sync script manually by running this command:
sudo /opt/jenkins-sync/sync-jenkins.sh

The script stops Jenkins on the replica, transfers files via rsync, then restarts Jenkins. This may take a few minutes depending on how much data has changed since your last sync.

  • After the command finishes, check the log file by running:
cat /var/log/jenkins-sync.log

You should see two timestamped lines: a "Starting Jenkins sync to replica" message and a "Sync completed successfully" message.

  • Confirm Jenkins is back up on the replica by running:
curl -s -o /dev/null -w "%{http_code}" http://192.168.1.230:8080/login

This should return 200, confirming Jenkins restarted successfully on the replica after the sync.

Seeing "Sync FAILED" in the log?

  • Check that the REPLICA_USER value in the script matches the username you used when setting up passwordless SSH in Step 1.
  • If Jenkins on the replica did not restart (the curl command does not return 200), SSH into the replica and check Jenkins status with sudo systemctl status jenkins.
  • Make sure the script is being run with sudo. The rsync command needs root access to read all of /var/lib/jenkins/.
  • help me troubleshoot my Jenkins sync script failing

Why exclude runtime files?

Files like war/ (the exploded WAR file) and .cache/ are rebuilt automatically when Jenkins starts. Syncing them wastes bandwidth and can cause startup issues on the replica.

Temporary files (*.tmp, *.pid) and logs (*.log) are tied to the primary's running process. Copying them to the replica would reference stale process IDs and fill the replica with irrelevant log data.

Schedule the cron job

The script works. Now you need it to run automatically without anyone remembering to trigger it. A cron job on the primary server handles this by running the script at a fixed interval.

  • Open the root crontab editor on the primary server by running:
sudo crontab -e

What is crontab?

Crontab is the configuration file where you define scheduled tasks (cron jobs) on Linux. Each line specifies when to run a command using five time fields: minute, hour, day of month, month, and day of week.

If this is your first time editing root's crontab, you may be prompted to choose an editor. Select nano if asked.

  • Scroll to the bottom of the file and paste the following cron entry:
# Jenkins JENKINS_HOME replication to replica (192.168.1.230) - every 3 hours
0 */3 * * * /opt/jenkins-sync/sync-jenkins.sh

What does this cron schedule mean?

The five fields 0 */3 * * * mean: at minute 0, every 3rd hour, every day. The script runs at 00:00, 03:00, 06:00, 09:00, and so on.

This gives your team a maximum 3-hour recovery window. If the primary goes down, you lose at most 3 hours of data since the last successful sync.

  • Save and exit the crontab editor (Ctrl+O, Enter, Ctrl+X in nano).

Before you verify, what do you expect to see when you list the root crontab?

  • Confirm the cron entry was saved by running:
sudo crontab -l

You should see your cron line 0 */3 * * * /opt/jenkins-sync/sync-jenkins.sh in the output. Your Jenkins replication is now fully automated.

Cron entry not showing up?

  • Make sure you saved the crontab before exiting. In nano, press Ctrl+O to write, then Ctrl+X to exit.
  • Verify you used sudo crontab -e (root crontab), not crontab -e (your user crontab). The script needs root privileges to read /var/lib/jenkins/.
  • help me troubleshoot my cron job for Jenkins replication

Your replication pipeline is live. Every 3 hours, the primary will push a fresh copy of its Jenkins data to the replica automatically. Next up, you will put this setup to the ultimate test by simulating a real outage.

Verify Replication and Test Manual Switchover

Your cron job is now set to replicate JENKINS_HOME every 3 hours without any manual intervention. But here is the real question: if your primary server went down right now, could you actually trust the replica to take over?

A disaster recovery setup you have never tested under real conditions is a setup you cannot trust. In this step, you will simulate a real outage by shutting down Jenkins on the primary and confirming the replica can serve as a fully functional instance on its own.

In this step, get ready to:
  • Run a fresh manual sync to bring the replica fully up to date.
  • Simulate a primary outage and confirm the replica works as a standalone Jenkins instance.
  • Restore the primary and document the manual switchover procedure for your team.
Run a fresh sync before testing

Before simulating an outage, you want the replica to have the absolute latest data from the primary. Any jobs created or builds triggered since the last automatic sync would be missing otherwise.

  • In your terminal on the primary server (192.168.1.253), trigger a manual sync by running this command:
sudo /opt/jenkins-sync/sync-jenkins.sh

The script stops Jenkins on the replica, runs rsync, and starts Jenkins again. This may take a minute or two depending on the size of your JENKINS_HOME.

  • Once the command returns, check the sync log to confirm it completed successfully:
tail /var/log/jenkins-sync.log

You should see a timestamped line ending with Sync completed successfully.

Sync showing a FAILED message?

Check that SSH is still working by running ssh user@192.168.1.230 'hostname' from the primary. If that hangs or asks for a password, your SSH key setup needs to be re-checked.

Also confirm there is enough disk space on the replica by running df -h /var/lib/jenkins on 192.168.1.230.

Still stuck? Help me debug a failed rsync sync between my primary and replica Jenkins servers

Simulate a primary outage

Now for the real test. You are going to stop Jenkins on the primary to simulate what happens when your team loses the production server.

  • On the primary server (192.168.1.253), stop Jenkins by running:
sudo systemctl stop jenkins

Your primary Jenkins at http://jenkins.polus.com/ is now down. In a real outage, this is the moment your team would lose access to all CI/CD jobs.

Before you visit the replica, take a moment to think: what do you expect to see on the replica's Jenkins dashboard? Will your jobs be there? Will the build history match?

  • Open your browser and visit http://192.168.1.230:8080.

You should see all your jobs from the primary listed on the dashboard. The replica is serving as a fully functional Jenkins instance right now, with the primary completely offline.

  • Pick any job on the replica's dashboard and click into it.
  • Verify that the build history matches what you see on the primary. Check that the build numbers and results are identical.
  • Click Manage Jenkins in the left sidebar, then select Plugins. Confirm the installed plugins match the primary.
  • Go back to the dashboard, select a job, and click Build Now to trigger a test build. Confirm the build starts and completes.

Why trigger a test build?

Seeing your jobs on the dashboard proves the data synced. Triggering a build proves the replica can actually execute work. In a real outage, your team will not just browse jobs. They need to run them.

Replica not showing your jobs?

Go back and check that the manual sync completed successfully by running tail /var/log/jenkins-sync.log on the primary. If the log shows a failure, re-run the sync script.

If the sync succeeded but jobs are missing, the file ownership on the replica may be wrong. Run ls -la /var/lib/jenkins/jobs/ on the replica and check that the files are owned by the jenkins user.

Need more help? Help me troubleshoot why my replica Jenkins is not showing jobs from the primary

Restore the primary and document the switchover procedure

The replica passed the test. Now bring the primary back online and capture the switchover procedure so your team knows exactly what to do during a real outage.

  • On the primary server (192.168.1.253), start Jenkins back up by running:
sudo systemctl start jenkins
  • Wait for Jenkins to fully start. Check its status by running:
curl -s -o /dev/null -w "%{http_code}" http://192.168.1.253:8080/login

Keep running this until it returns 200. Jenkins can take a minute to fully start up.

  • Visit http://jenkins.polus.com/ in your browser. Confirm the primary is back to normal operation with all your jobs visible.

Your primary is restored. The last piece is to write down the switchover procedure so anyone on your team can follow it during a real outage.

Your manual switchover procedure

In a real outage, your team follows these three steps:

  • Confirm the primary is down. Check that http://jenkins.polus.com/ is unreachable or returning errors.
  • Confirm the latest sync completed. Check /var/log/jenkins-sync.log on the primary (if accessible) or check the replica's JENKINS_HOME timestamps to estimate how current the data is.
  • Direct users to the replica. Tell your team to access Jenkins at http://192.168.1.230:8080 until the primary is restored.

Because the cron job syncs every 3 hours, the worst-case data loss is 3 hours of build history and configuration changes. Any builds that were in progress at the time of the last sync may be in a partial state on the replica.

Secret mission

Add HAProxy for Automatic Failover

Your replication setup works, but switching to the replica still requires someone to notice the outage and tell everyone the new URL. What if the primary goes down at 2am? In this mission, you will install HAProxy to sit in front of both servers and automatically route traffic to the replica when the primary fails its health check.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely on your existing on-prem servers using open-source tools, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Sync script at /opt/jenkins-sync/sync-jenkins.sh on 192.168.1.253.
  • Cron job on 192.168.1.253 running every 3 hours.
  • SSH key pair at ~/.ssh/id_ed25519 on 192.168.1.253, with the public key in authorized_keys on 192.168.1.230.
  • Sync log at /var/log/jenkins-sync.log on 192.168.1.253.
  • HAProxy installation and configuration at /etc/haproxy/haproxy.cfg (Secret Mission only).

Keep everything running

No action needed. Choose this if your replication setup is actively protecting your production Jenkins.

  • The cron job continues syncing JENKINS_HOME every 3 hours automatically.
  • The replica at 192.168.1.230 stays within 3 hours of your primary's data at all times.
  • Monitor sync health periodically by checking /var/log/jenkins-sync.log on 192.168.1.253.
  • If you completed the Secret Mission, HAProxy remains running and handles automatic failover between the primary and replica.

Pause - I'll come back to this later

Pause the automated replication while keeping all scripts and configuration in place. The replica keeps its current data but stops receiving updates.

  • On 192.168.1.253, open the crontab for editing by running:
sudo crontab -e
  • Add a # at the beginning of the sync line to comment it out.
  • Save and close the editor.
  • To resume later, re-open the crontab with the same command and remove the # to re-enable the scheduled sync.

Delete - I don't want to use this again

Remove all replication resources and start fresh. This does not affect your primary Jenkins server or its data.

  • On 192.168.1.253, remove the cron entry by running:
sudo crontab -e
  • Delete the line that starts with 0 */3 * * *, then save and close the file.
  • Delete the sync script directory on 192.168.1.253 by running:
sudo rm -rf /opt/jenkins-sync/
  • Delete the sync log on 192.168.1.253 by running:
sudo rm /var/log/jenkins-sync.log
  • On 192.168.1.230, remove the SSH public key from the replica by running:
nano ~/.ssh/authorized_keys
  • Find and delete the line containing the ed25519 key from 192.168.1.253, then save the file.
  • Back on 192.168.1.253, delete the SSH key pair by running:
rm ~/.ssh/id_ed25519 ~/.ssh/id_ed25519.pub

The replica's Jenkins installation at 192.168.1.230 can be left in place or removed. It functions as a standalone Jenkins server and does not incur any cost.

If you completed the Secret Mission, remove HAProxy:

  • Update your DNS record for jenkins.polus.com to point back to 192.168.1.253 instead of the HAProxy IP.
  • Remove HAProxy on Debian/Ubuntu by running:
sudo apt-get remove --purge haproxy
  • On RHEL/CentOS, run this instead:
sudo yum remove haproxy

Nice Work!

Nice Work!

Nice work! You just built a fully automated disaster recovery replication pipeline for your production Jenkins server.

Your primary Jenkins at 192.168.1.253 now pushes a full copy of its jobs, workspaces, build history, plugins, and configuration to your replica at 192.168.1.230 every 3 hours. If the primary ever goes down, your team can keep working without skipping a beat.

You've learned how to:

  • Build automated rsync replication that keeps a warm standby Jenkins replica within a 3-hour recovery window using passwordless SSH and incremental file sync.
  • Create a production-ready sync script with error handling, runtime file exclusions, and timestamped audit logging, scheduled via cron to run every 3 hours with zero manual intervention.
  • Test a manual switchover by simulating a primary outage and confirming the replica serves as a fully functional Jenkins instance with all jobs, build history, and plugins intact.
  • Secret Mission: Added HAProxy for automatic failover so traffic routes to the replica without manual intervention when the primary goes down.

Ready to quiz yourself?