Linux Service Health Monitor

Build a self-healing Linux service with bash, systemd, and journalctl.

Introduction

30 Second Summary

Every running service will eventually crash. The difference between a minor hiccup and a 3 AM emergency page comes down to one thing: whether something is watching the service and can bring it back automatically.

In this project, you will build a complete service health monitoring system on Linux. You will create a custom HTTP service, write a Bash script that detects failures and restarts the service, and deploy it all on an automated timer using systemd.

What You'll Build

When you finish, you can break your HTTP server on purpose, walk away, and come back to find the health monitor has already detected the failure and restarted the service without any human intervention.

By the end of this project, you'll have:

  • A custom systemd service running a Python HTTP server that you can start, stop, and probe with curl to see a live JSON health response.
  • A bash health-check script that automatically detects when the service goes down and triggers a restart without human intervention.
  • A self-healing demo where you break the service on purpose, watch the systemd timer detect the failure within a minute, and trace the full incident trail through journalctl logs.
  • Secret Mission: Add structured alert logging with timestamps and severity levels to create a production-grade incident audit trail.

Are there any prerequisites?

This project assumes you have a Windows PC with WSL2 installed and systemd enabled. You should be comfortable with basic Linux commands like cd, ls, and sudo.

Before We Start

Before We Start...

Before we dive in, let's lock in what you're about to build and why it matters.

Set Up Your Project Environment

Before building anything, you need to confirm your environment has the right foundations. This project relies on systemd for service management and Python 3 for the HTTP server you will monitor.

In this step, you will verify both tools are available in your WSL2 environment, create a clean project directory, and confirm you have the permissions needed to register custom services.

In this step, get ready to:
  • Verify systemd and Python 3 are available.
  • Create the project directory structure.
  • Confirm write access to the systemd configuration directory.
Verify systemd and Python 3 are available

Your service monitoring system depends on systemd being active and Python 3 being installed. A quick version check for each will confirm they are ready to use.

  • In your WSL2 terminal, check that systemd is active by running this command:
systemctl --version

You should see output starting with systemd followed by a version number (for example, 249 or higher).

  • Confirm Python 3 is available by running:
python3 --version

You should see a version number like Python 3.10.x or similar.

Not seeing a version number?

If systemctl --version returns an error, systemd may not be enabled in your WSL2 configuration. Check that your /etc/wsl.conf file contains [boot] and systemd=true, then restart WSL from PowerShell with wsl --shutdown and reopen your terminal.

If python3 --version returns command not found, install it by running sudo apt update && sudo apt install python3.

Help me troubleshoot why systemd or Python 3 is not available in my WSL2 environment.

Create the project directory structure

You need a dedicated folder to hold your scripts and keep the project organized. The -p flag creates the parent and child directories in one command.

  • Create your project folder and scripts subdirectory by running this command:
mkdir -p ~/healthmon/scripts
  • Move into your new project folder by running:
cd ~/healthmon
  • Confirm the directory structure exists by running:
ls

You should see scripts listed in the output. This is where your health-check script and HTTP server will live.

Not seeing the scripts folder?

Make sure you ran mkdir -p ~/healthmon/scripts with the -p flag. Without it, the command fails if the parent healthmon directory does not already exist.

Help me figure out why my project directories were not created in WSL2.

Confirm write access to the systemd directory

Your custom service unit files will live in /etc/systemd/system/. You need sudo privileges to write there, so a quick listing confirms you have access.

Before you run the next command, what do you expect to see inside a system-level systemd directory?

  • Verify you can read the systemd unit directory by running:
sudo ls /etc/systemd/system/

You should see a list of existing .service and .target files. This is the directory where your custom unit files will live in later steps.

Permission denied?

If you see a permission error, make sure you included sudo at the start of the command. You may be prompted to enter your password. Type it and press Enter (the characters will not appear as you type).

Help me resolve a permission denied error when running sudo ls on the systemd directory.

Your environment is confirmed and your project directory is ready. Next up, you will create a custom HTTP service that gives you something real to monitor.

Create a Custom HTTP Service

Your environment is confirmed and ready. Now you need something to actually monitor.

In this step, you'll build a minimal HTTP server in Python and register it as a systemd service. This gives you a real network service you can probe, break on purpose, and watch recover.

In this step, get ready to:
  • Create a Python HTTP server that returns a JSON health status on port 8111.
  • Write a systemd service unit file to manage the server.
  • Start the service and confirm it responds to HTTP requests.
Create the Python HTTP server script

You need a service that listens on the network and responds to requests. A Python HTTP server does this with zero external dependencies since Python is already installed in your WSL2 environment.

Why a custom Python server instead of Nginx?

You already have experience with Nginx from earlier projects. A custom Python server teaches you to create services from scratch, including writing your own unit files instead of enabling pre-packaged ones.

It also makes it trivial to intentionally break and fix the service without touching system packages.

  • Create the server script by opening a new file in nano:
nano ~/healthmon/scripts/healthmon-server.py

What will I see?

Nano opens with a blank editing area and shows New File in the status bar at the bottom. You can paste code directly into this editor.

  • Copy and paste the following code into the nano editor:
#!/usr/bin/env python3
"""Minimal HTTP health-check server for systemd monitoring demo."""

import json
from http.server import HTTPServer, BaseHTTPRequestHandler

class HealthHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        response = {"status": "healthy", "service": "healthmon"}
        body = json.dumps(response).encode("utf-8")
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, format, *args):
        # Print to stdout so systemd journal captures it
        print(f"[healthmon] {args[0]}")

if __name__ == "__main__":
    server = HTTPServer(("0.0.0.0", 8111), HealthHandler)
    print("[healthmon] Server starting on port 8111")
    server.serve_forever()

What does this code do?

This script creates a minimal HTTP server that listens on port 8111 and responds to every GET request with a JSON health status.

  • The HealthHandler class defines what happens when a request arrives. It builds a JSON response body and sends it back with a 200 status.
  • The log_message override prints request logs to stdout, which systemd's journal will capture automatically.
  • The if __name__ block starts the server on all interfaces (0.0.0.0) so you can reach it from localhost.
  • Save the file by pressing Ctrl+O, then Enter to confirm the filename.
  • Exit nano by pressing Ctrl+X.
  • Make the script executable by running:
chmod +x ~/healthmon/scripts/healthmon-server.py
  • Confirm the file exists and is executable by running:
ls -l ~/healthmon/scripts/

You should see healthmon-server.py listed with -rwxr-xr-x permissions (the x means executable).

Don't see the x permission?

Make sure you ran chmod +x with the full path to the script. Run it again and check with ls -l.

If the file is empty or missing, re-open it with nano ~/healthmon/scripts/healthmon-server.py and paste the code again. Make sure you press Ctrl+O to save before exiting.

Help me fix file permissions on my healthmon-server.py script.

✔️ Awesome, I've got everything!

Great. Make sure you've saved the file and confirmed the executable permission before moving on.

ⓧ I'd like to double check the full code

Here is the complete healthmon-server.py file. Confirm yours matches:

#!/usr/bin/env python3
"""Minimal HTTP health-check server for systemd monitoring demo."""

import json
from http.server import HTTPServer, BaseHTTPRequestHandler

class HealthHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        response = {"status": "healthy", "service": "healthmon"}
        body = json.dumps(response).encode("utf-8")
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, format, *args):
        # Print to stdout so systemd journal captures it
        print(f"[healthmon] {args[0]}")

if __name__ == "__main__":
    server = HTTPServer(("0.0.0.0", 8111), HealthHandler)
    print("[healthmon] Server starting on port 8111")
    server.serve_forever()
Create the systemd service unit file

A systemd unit file tells systemd how to manage your service. It defines what command to run, when to restart, and where to send logs.

  • Open a new unit file in the systemd directory by running:
sudo nano /etc/systemd/system/healthmon.service
  • Paste the following service definition into the editor:
[Unit]
Description=Health Monitor Demo HTTP Server
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/python3 /root/healthmon/scripts/healthmon-server.py
Restart=on-failure
RestartSec=5
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

What does this unit file do?

  • Type=simple tells systemd the process runs in the foreground. Systemd considers the service started as soon as ExecStart begins.
  • ExecStart uses the absolute path to both Python and your script. Unit files require absolute paths because they run without a shell environment.
  • Restart=on-failure tells systemd to automatically restart the service if it exits with a non-zero exit code. RestartSec=5 waits 5 seconds before each restart attempt.
  • StandardOutput=journal routes all print statements to the systemd journal, making them queryable with journalctl.
  • Save the file by pressing Ctrl+O, then Enter to confirm.
  • Exit nano by pressing Ctrl+X.
  • Reload systemd and start the service by running these commands:
sudo systemctl daemon-reload
sudo systemctl start healthmon.service

What do these commands do?

The daemon-reload command tells systemd to re-read all unit files from disk. This is required every time you create or edit a unit file. The start command launches your service in the background.

If both commands return with no error output, the service started successfully. You'll verify it fully in the next substep.

Seeing a permission error?

Make sure you used sudo before both nano (when creating the file) and the systemctl commands. The /etc/systemd/system/ directory requires root access.

If you see an error mentioning the executable path, check that ExecStart in your unit file points to /usr/bin/python3 /root/healthmon/scripts/healthmon-server.py with no typos.

Help me troubleshoot my healthmon.service failing to start.

✔️ Awesome, I've got everything!

Great. Confirm you ran daemon-reload before starting the service.

ⓧ I'd like to double check the full code

Here is the complete healthmon.service unit file. Confirm yours matches:

[Unit]
Description=Health Monitor Demo HTTP Server
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/python3 /root/healthmon/scripts/healthmon-server.py
Restart=on-failure
RestartSec=5
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
Test the running service

Your service is started. Time to confirm it is actually running and responding to requests.

  • Check the service status by running:
sudo systemctl status healthmon.service --no-pager

You should see active (running) in green text, along with the PID and the log line showing your server started on port 8111.

Before you run the next command, what do you think curl http://localhost:8111 will return? Think about what your do_GET method sends back.

  • Probe the HTTP endpoint by running:
curl http://localhost:8111

You should see the JSON response:

{"status": "healthy", "service": "healthmon"}

Your custom service is live and queryable. You started a Python server, registered it with systemd, and confirmed it responds to network requests.

Seeing 'Connection refused' from curl?

Run sudo systemctl status healthmon.service --no-pager again. If it shows failed, the Python script crashed on startup.

Common causes: a typo in the Python script (check the port number is 8111), or the ExecStart path in the unit file does not match the actual script location at /root/healthmon/scripts/healthmon-server.py.

If another process is already using port 8111, you will see an Address already in use error in the journal. Run sudo journalctl -u healthmon.service --no-pager -n 10 to see the error details.

Help me debug why curl can't connect to my healthmon service.

Your HTTP service is running and managed by systemd. Next up, you'll intentionally break it and learn how to diagnose failures using journalctl. This is where the real sysadmin skills kick in.

Break It and Diagnose with journalctl

Your service is running and responding to requests. Starting a service is table stakes.

In production, services fail constantly. Knowing how to figure out WHY a service stopped is what separates a capable Linux admin from someone who just follows tutorials. In this step, you will intentionally break your service and use journalctl to find the root cause from the logs alone.

In this step, get ready to:
  • Introduce a deliberate misconfiguration to simulate a real production failure.
  • Practice a production troubleshooting workflow using systemctl status and journalctl filters.
  • Complete a full incident cycle: detect, diagnose, remediate, verify.
Stop the service and introduce a misconfiguration

A common production failure is a bad deployment that changes a service's executable path. You will simulate this by pointing ExecStart at a file that does not exist.

  • Stop the running service by running this command:
sudo systemctl stop healthmon.service
  • Open the unit file in nano by running:
sudo nano /etc/systemd/system/healthmon.service
  • Find this line in the [Service] section:
ExecStart=/usr/bin/python3 /root/healthmon/scripts/healthmon-server.py
  • Replace the path with a non-existent file, so the line looks like this:
ExecStart=/usr/bin/python3 /usr/local/bin/healthmon-server-WRONG.py
  • Save the file with Ctrl+O, press Enter, then exit with Ctrl+X.
  • Reload the unit file and attempt to start the service by running these commands:
sudo systemctl daemon-reload
sudo systemctl start healthmon.service

Before you look at the output, what do you think happened? The path you pointed to does not exist on your filesystem.

The start command will either fail immediately or return silently while the service enters a failed state. Either way, the service is now broken.

Why break it on purpose?

In the real world, you never get a friendly message saying "your deployment broke the path." You get a page at 2am saying the service is down. Practicing this deliberately builds muscle memory so you reach for the right diagnostic tool automatically.

Diagnose the failure with journalctl

Now you will step through the exact workflow a production admin uses when paged about a failed service. Each command reveals a different layer of information.

  • Start with the quick overview by running:
sudo systemctl status healthmon.service --no-pager

What should I see?

You should see Active: failed with a result like exit-code and an exit status of 203/EXEC. Exit code 203 means systemd could not execute the command in ExecStart because the file does not exist.

The status output is the "at a glance" view. It tells you THAT something failed and gives a category, but not always the full story.

  • Dig deeper with the full journal log and explanatory hints by running:
sudo journalctl -xeu healthmon.service --no-pager

What do the flags mean?

The -x flag adds explanatory help texts from the message catalog. The -e flag jumps to the end of the journal (most recent entries). The -u flag filters to only messages from your specific unit.

You should see a detailed log trail showing the failed start attempt and a hint pointing at the non-existent executable path.

  • Now filter to only error-priority messages in a recent time window by running:
sudo journalctl -u healthmon.service -p err --since "5 minutes ago"

Why filter by priority?

The -p err flag shows only messages at error priority or higher. The --since flag restricts the time window. In production, journals can be enormous. Combining these two filters cuts straight to the signal that reveals the root cause.

You should see the error message that confirms the misconfiguration: systemd tried to execute a file that does not exist at the path you specified.

Seeing no output from the journal commands?

Make sure you ran sudo systemctl start healthmon.service after the daemon-reload. If you only reloaded without attempting a start, there is no failure event in the journal yet.

Also verify the --since window is wide enough. If more than 5 minutes passed since the failed start, widen it to "10 minutes ago".

Need more help? Why is journalctl not showing any error logs for my failed healthmon service?

Fix the misconfiguration and verify recovery

You have detected and diagnosed the failure. Now complete the incident cycle by remediating the root cause and verifying recovery.

  • Open the unit file again by running:
sudo nano /etc/systemd/system/healthmon.service
  • Find this broken line:
ExecStart=/usr/bin/python3 /usr/local/bin/healthmon-server-WRONG.py
  • Replace it with the correct path, so the line looks like this:
ExecStart=/usr/bin/python3 /root/healthmon/scripts/healthmon-server.py
  • Save with Ctrl+O, press Enter, then exit with Ctrl+X.
  • Reload and restart the service by running:
sudo systemctl daemon-reload
sudo systemctl start healthmon.service

Before you check the result, do you think the service is healthy again? What would prove it?

  • Verify the service is back and responding by running:
curl http://localhost:8111

You should see {"status": "healthy", "service": "healthmon"} returned. Your service is live again.

What did you just practice?

You completed a full incident cycle: detect (the start failed), diagnose (journalctl revealed the bad path), remediate (restored the correct ExecStart), and verify (curl confirms the service responds). This is the exact workflow production teams repeat daily.

Still seeing a failed status after fixing the path?

Make sure you ran sudo systemctl daemon-reload before attempting to start. Without the reload, systemd still uses the old (broken) unit file from memory.

Also double-check the path matches exactly: /usr/bin/python3 /root/healthmon/scripts/healthmon-server.py. A single typo will cause another 203/EXEC failure.

Still stuck? Help me fix my healthmon.service that still shows failed after I restored the ExecStart path.

You now have the diagnostic muscle memory to tackle real service failures. Next up, you will build a script that automates this health-checking so you do not have to run curl manually every time.

Write a Health-Check Script

You have a running service you can check with curl. But manually running a command every time you suspect something is wrong does not scale.

In production, services fail at 3am when nobody is watching. You need an automated probe that checks the service on its own, decides whether it is healthy, and takes corrective action when it is not. That is the building block of every monitoring system.

In this step, get ready to:
  • Write a bash script that probes your HTTP service and logs the result.
  • Add automatic restart logic so the script self-heals a downed service.
  • Test the script in both healthy and failed scenarios.
Create the probe and success path

The health-check script's first job is simple: probe the HTTP endpoint and report whether the service responded with a 200 status code.

  • Create the health-check script by opening a new file in nano:
nano ~/healthmon/scripts/healthcheck.sh
  • Paste the following code into the file:
#!/usr/bin/env bash
# Health check script for healthmon service.
# Probes the HTTP endpoint and restarts the service if unreachable.

set -euo pipefail

SERVICE="healthmon.service"
URL="http://localhost:8111"
TIMEOUT=5

echo "[healthcheck] Probing ${URL}..."

HTTP_CODE=$(curl --silent --output /dev/null --write-out "%{http_code}" \
    --max-time "${TIMEOUT}" "${URL}" 2>/dev/null || echo "000")

if [[ "${HTTP_CODE}" == "200" ]]; then
    echo "[healthcheck] OK - service is healthy (HTTP ${HTTP_CODE})"
    exit 0
fi

What does this code do?

This is the probe half of your health check. It sends a request to your HTTP server and reads the response code.

  • set -euo pipefail makes the script exit immediately on any unexpected error, catching typos and misconfigurations early.
  • curl --write-out "%{http_code}" extracts just the HTTP status code from the response. If curl cannot connect at all, the || echo "000" fallback produces 000 instead of crashing the script.
  • --max-time prevents the probe from hanging forever if the service is unresponsive. Five seconds is generous for a localhost check.
  • If the response is 200, the script logs success and exits cleanly.
  • Save the file by pressing Ctrl+O, then Enter, then Ctrl+X to exit nano.
  • Make the script executable by running:
chmod +x ~/healthmon/scripts/healthcheck.sh
  • Test the probe with your service still running:
sudo bash ~/healthmon/scripts/healthcheck.sh

You should see output like:

[healthcheck] Probing http://localhost:8111...
[healthcheck] OK - service is healthy (HTTP 200)

Seeing "permission denied" or "command not found"?

  • Make sure you ran chmod +x on the script. Without it, bash may refuse to execute the file.
  • Confirm the shebang line at the top says #!/usr/bin/env bash with no typos.
  • help me debug my healthcheck script
Add the failure-recovery logic

The script can now detect a healthy service. But if the probe fails, it just exits silently. You need the script to take action: restart the service and verify the recovery.

  • Open the script again in nano:
nano ~/healthmon/scripts/healthcheck.sh
  • Add the following code directly below the closing fi line at the bottom of the file:
echo "[healthcheck] FAIL - service unreachable (HTTP ${HTTP_CODE})"
echo "[healthcheck] Attempting restart of ${SERVICE}..."

systemctl restart "${SERVICE}"
sleep 2

# Verify recovery
HTTP_CODE=$(curl --silent --output /dev/null --write-out "%{http_code}" \
    --max-time "${TIMEOUT}" "${URL}" 2>/dev/null || echo "000")

if [[ "${HTTP_CODE}" == "200" ]]; then
    echo "[healthcheck] RECOVERED - service restarted successfully"
    exit 0
else
    echo "[healthcheck] CRITICAL - restart failed, service still down (HTTP ${HTTP_CODE})"
    exit 1
fi

What does this code do?

This is the self-healing half. It only runs when the probe detected a failure (the earlier if block did not exit 0).

  • systemctl restart tells systemd to stop and then start the service fresh.
  • sleep 2 gives the service time to bind the port before the verification probe runs.
  • The script re-probes after the restart. If the service is back, it reports RECOVERED. If it is still down, it reports CRITICAL and exits with code 1 so upstream tooling (like a timer) can detect persistent failures.
  • Save and exit nano (Ctrl+O, Enter, Ctrl+X).

Now test the full failure-recovery path. Before you run the script, do you think it will succeed or fail when the service is already stopped?

  • Stop the service to simulate a crash:
sudo systemctl stop healthmon.service
  • Run the health check against the downed service:
sudo bash ~/healthmon/scripts/healthcheck.sh

You should see the script detect the failure, attempt a restart, and confirm recovery:

[healthcheck] Probing http://localhost:8111...
[healthcheck] FAIL - service unreachable (HTTP 000)
[healthcheck] Attempting restart of healthmon.service...
[healthcheck] RECOVERED - service restarted successfully

Script says CRITICAL instead of RECOVERED?

  • The sleep 2 might not be enough on your system. Try increasing it to sleep 4 in the script and running again.
  • Verify the service can start at all by running sudo systemctl start healthmon.service manually. If that fails, the service unit file may have a bad ExecStart path.
  • help me debug why my healthcheck script cannot restart the service
Verify the full self-healing loop

The script said it recovered, but you should independently confirm the service is actually responding. Trust but verify.

  • Confirm the service is responding by running:
curl http://localhost:8111

You should see {"status": "healthy", "service": "healthmon"} returned.

  • Check the journal to see the stop and restart events that systemd recorded:
sudo journalctl -u healthmon.service --since "2 minutes ago" --no-pager

You should see entries showing the service being stopped, then started again. This is the audit trail that proves your script triggered a real restart through systemd.

Why check the journal after a script-triggered restart?

In production, you often arrive after an incident has already been auto-resolved. The journal is your forensic record. It tells you when the service went down, what restarted it, and whether the restart stuck.

The --since flag narrows the output to a recent window. Without it, you would scroll through the entire service history looking for the relevant events.

✔️ Awesome, I've got everything!

Great. Make sure you have saved healthcheck.sh and that curl http://localhost:8111 still returns the healthy JSON response.

ⓧ I'd like to double check the full code

Your ~/healthmon/scripts/healthcheck.sh should contain two sections in order:

  • The probe and success path (shebang, set -euo pipefail, variables, the curl probe, and the if block that exits 0 on HTTP 200).
  • The failure-recovery logic (FAIL echo, systemctl restart, sleep 2, the recovery probe, and the final if/else that reports RECOVERED or CRITICAL).

Scroll up and confirm each section is present in your file, in order. The script should be 37 lines total (including blank lines and comments).

You now have an automated probe that can detect a downed service and bring it back without human intervention. Next up, you will deploy this script on a schedule using a systemd timer so it runs every minute without anyone having to trigger it.

Deploy as a Systemd Timer

In the previous step you built a health-check script that can probe your service and restart it on failure. But running that script by hand every time defeats the entire purpose of automation.

Systemd timers are the production-grade way to schedule recurring tasks on modern Linux. They give you journal logging, dependency ordering, and Persistent=true for catching up on missed runs. In this step, you will wire your health-check script into a timer that fires every minute and automatically restarts the service if it goes down.

In this step, get ready to:
  • Create a oneshot service unit and a timer unit to schedule your health check.
  • Enable and start the timer with systemctl enable --now and validate its schedule.
  • Test the full self-healing loop by stopping the HTTP service and watching the timer bring it back.
Create the oneshot service and timer units

Systemd timers work in pairs. A .timer unit controls the schedule, and it triggers a matching .service unit each time it fires. Your wrapper service uses Type=oneshot because it runs a single task and exits.

Why systemd timers instead of cron?

Production Linux teams prefer systemd timers because every execution is logged to the journal automatically. You can query timer runs with journalctl just like any other service.

Timers also support Persistent=true, which catches up on missed runs if the system was down when a timer should have fired. Cron jobs simply skip missed runs with no record they were missed.

  • Create the oneshot service unit by opening it in nano:
sudo nano /etc/systemd/system/healthmon-check.service
  • Paste the following content into the editor:
[Unit]
Description=Health Check for healthmon service
Wants=healthmon.service

[Service]
Type=oneshot
ExecStart=/bin/bash /root/healthmon/scripts/healthcheck.sh
StandardOutput=journal
StandardError=journal
  • Save the file with Ctrl+O, press Enter to confirm the filename, then exit with Ctrl+X.

What does this unit file do?

  • Wants=healthmon.service tells systemd this unit is related to your HTTP service, but does not require it to be running.
  • Type=oneshot means this service runs a single task and exits rather than staying resident.
  • ExecStart points at your health-check script using the absolute path. Systemd requires absolute paths in unit files.
  • Confirm the file was saved correctly by running:
cat /etc/systemd/system/healthmon-check.service

You should see the unit file content you just pasted printed back to the terminal.

See an empty file or 'No such file'?

  • Make sure you used sudo with the nano command. Without it, you cannot write to /etc/systemd/system/.
  • Check you saved with Ctrl+O before exiting. If you pressed Ctrl+X first without saving, the file will be empty.

Having trouble with nano permissions?

  • Now create the timer unit that controls when the health check runs:
sudo nano /etc/systemd/system/healthmon-check.timer
  • Paste the following content into the editor:
[Unit]
Description=Run health check for healthmon every minute

[Timer]
OnCalendar=*-*-* *:*:00
Persistent=true
AccuracySec=1s

[Install]
WantedBy=timers.target
  • Save with Ctrl+O, press Enter, then exit with Ctrl+X.

What do these timer directives mean?

  • OnCalendar=*-*-* *:*:00 is a calendar expression meaning "every minute at :00 seconds." The wildcards match any year, month, day, hour, and minute.
  • Persistent=true tells systemd to catch up on missed runs if the system was down when the timer should have fired.
  • AccuracySec=1s ensures the timer fires close to the target time instead of batching with other timers for power saving.
  • WantedBy=timers.target means this timer activates automatically at boot when enabled.
  • Verify both unit files exist by running:
ls /etc/systemd/system/healthmon-check*

You should see both healthmon-check.service and healthmon-check.timer listed.

Only see one file?

  • Re-run the sudo nano command for the missing file and make sure you save before exiting.
  • Double check the filename ends in .timer (not .time or .service).

Why are my unit files not appearing?

✔️ Awesome, I've got everything!

Great. Double check both files are saved before moving on.

ⓧ I'd like to double check the full code

Here is healthmon-check.service:

[Unit]
Description=Health Check for healthmon service
Wants=healthmon.service

[Service]
Type=oneshot
ExecStart=/bin/bash /root/healthmon/scripts/healthcheck.sh
StandardOutput=journal
StandardError=journal

Here is healthmon-check.timer:

[Unit]
Description=Run health check for healthmon every minute

[Timer]
OnCalendar=*-*-* *:*:00
Persistent=true
AccuracySec=1s

[Install]
WantedBy=timers.target
Enable and start the timer

With both unit files on disk, you need to tell systemd to re-read them and then activate the timer.

  • Reload systemd's unit file cache and enable the timer in one go by running these commands:
sudo systemctl daemon-reload
sudo systemctl enable --now healthmon-check.timer

What does enable --now do?

The enable part creates a symlink so the timer starts automatically at boot. The --now flag also starts the timer immediately so you do not have to run a separate start command.

  • Confirm the timer is scheduled by running:
sudo systemctl list-timers --all | grep healthmon

You should see a row with healthmon-check.timer showing values in the NEXT and LEFT columns. These tell you exactly when the next health check will fire.

  • Validate your OnCalendar expression to confirm it means "every minute" by running:
systemd-analyze calendar "*-*-* *:*:00"

The output normalizes the expression and shows you the next few trigger times, one minute apart.

Timer not listed or showing 'dead'?

  • Make sure you ran sudo systemctl daemon-reload before enabling. Without it, systemd does not know about the new files.
  • Check for typos in the timer filename. The timer must be named healthmon-check.timer to match the service healthmon-check.service (same prefix, different suffix).
  • Run sudo systemctl status healthmon-check.timer to see if there is an error message about the unit.

Why is my systemd timer not showing up in list-timers?

Test the full self-healing loop

The timer is now firing every minute. Time to prove the entire automation works end-to-end: you will break the service and watch the timer bring it back without any manual intervention.

  • First, confirm the timer has already fired at least once by checking its journal logs:
sudo journalctl -u healthmon-check.service --since "3 minutes ago"

You should see log entries from your health-check script showing [healthcheck] OK - service is healthy. This proves the timer triggered your script automatically.

What if I don't see any logs yet?

The timer fires at the top of each minute (:00 seconds). If you just enabled it, wait until the next full minute passes, then re-run the journalctl command.

Now for the real test. Before you run the next command, predict what will happen: if you stop the HTTP service, will curl http://localhost:8111 succeed or fail after one minute?

  • Stop the HTTP service to simulate a crash:
sudo systemctl stop healthmon.service
  • Confirm the service is down by running:
curl http://localhost:8111

You should see a connection refused error. The service is dead.

  • Now wait up to one minute for the timer to fire. You can watch the timer in real time by running:
sudo journalctl -u healthmon-check.service -f

This follows the health-check service journal live. When the timer fires, you will see your script detect the failure and restart the service. Press Ctrl+C to stop following once you see the recovery message.

  • Verify the service was automatically restarted by running:
curl http://localhost:8111

You should see {"status": "healthy", "service": "healthmon"} returned. The timer detected the outage and brought the service back without you lifting a finger.

What just happened?

You proved a complete self-healing loop. The timer fired on schedule, the oneshot service executed your bash script, the script detected the HTTP failure, issued a restart, and confirmed recovery.

This is exactly the pattern production teams use to reduce on-call pages. Instead of a human noticing the outage and SSHing in to restart, the system heals itself and logs the incident for review.

  • Confirm the full incident trail in the journal by running:
sudo journalctl -u healthmon-check.service --since "3 minutes ago" --no-pager

You should see the sequence: a healthy check, then the failure detection, the restart attempt, and the recovery confirmation.

Service still down after a minute?

  • Check the timer is actually active with sudo systemctl status healthmon-check.timer. If it shows inactive, re-enable it with sudo systemctl enable --now healthmon-check.timer.
  • Verify your health-check script path is correct in the oneshot unit. Run cat /etc/systemd/system/healthmon-check.service and confirm ExecStart points to /bin/bash /root/healthmon/scripts/healthcheck.sh.
  • Check for script errors with sudo journalctl -u healthmon-check.service -p err.

My systemd timer is not restarting the stopped service. What is wrong?

Your health monitor is now fully autonomous. The timer fires every minute, the health-check script probes the service, and if anything goes wrong, it restarts automatically with a full audit trail in the journal. Next up, you will extend this system with structured alert logging in the Secret Mission.

Secret mission

Add Structured Alert Logging

Your health check writes to the journal, but journal entries rotate and disappear. In production, you need a persistent audit trail of every incident. Can you add structured alerting with timestamps, severity levels, and failure details written to a dedicated log file?

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely inside your WSL2 instance, so there are no ongoing costs.

Resources you used:

  • /etc/systemd/system/healthmon.service (systemd service unit).
  • /etc/systemd/system/healthmon-check.service (oneshot wrapper unit).
  • /etc/systemd/system/healthmon-check.timer (timer unit).
  • ~/healthmon/ directory (Python HTTP server and health-check script).
  • /var/log/healthmon/ directory (structured alert logs).

Keep everything running

No action needed. Choose this if you are still actively building or want the timer to keep checking the service every minute.

  • The healthmon-check.timer will continue firing every minute, probing the HTTP server and logging results to the journal.
  • Alert entries will keep appending to /var/log/healthmon/alerts.log.
  • There is no cost. Everything runs locally inside WSL2.

Pause - I'll come back to this later

Shut down the running services to free up system resources. Your files and unit definitions stay on disk so you can re-enable everything later.

  • Stop the timer and the HTTP service by running:
sudo systemctl stop healthmon-check.timer && sudo systemctl stop healthmon.service
  • To resume later, run sudo systemctl start healthmon.service followed by sudo systemctl start healthmon-check.timer.

Delete - I don't want to use this again

Remove all project resources and start fresh. This disables the services, deletes the unit files, removes the project directory, and cleans up the log directory.

  • Disable and stop the timer and service, then remove the unit files by running:
sudo systemctl disable --now healthmon-check.timer healthmon.service && sudo rm /etc/systemd/system/healthmon* && sudo systemctl daemon-reload
  • Remove the project directory by running:
rm -rf ~/healthmon
  • Remove the alert log directory by running:
sudo rm -rf /var/log/healthmon
  • Verify everything is gone by running:
systemctl list-units --all | grep healthmon
  • You should see no output. All healthmon units have been removed from systemd.

Nice Work!

Nice Work!

Nice work! You just built a production-style service health monitoring system from scratch in your WSL2 environment.

You've learned how to:

  • Built a custom systemd service from scratch using a Python HTTP server and a .service unit file.
  • Diagnosed a broken service using journalctl filters (-xeu, -p err, and --since) to find the root cause and restore it through a full incident cycle.
  • Deployed a self-healing monitoring loop with a bash health-check script and a systemd timer using OnCalendar scheduling and Persistent=true for missed-run catch-up.
  • Secret Mission: Added structured alert logging with severity levels and timestamps for incident audit trails.

Ready to quiz yourself?