System Health Monitor with Bash

Write a production Bash script and deploy it as a managed systemd service.

Introduction

30 Second Summary

Every server has vital signs. When CPU, memory, or disk usage spikes, you need to know before your users do.

In this project, you will build a system health monitor that checks your machine's resources and reports the results automatically on a schedule. You'll write production-grade Bash scripts and deploy them as a managed Linux service.

What You'll Build

A complete system health monitoring solution deployed as a systemd service that checks CPU, memory, and disk usage on an automated schedule with structured logging and resource controls.

By the end of this project, you'll have:

  • A production-grade Bash monitoring script with structured logging, error handling, and trap-based cleanup.
  • Hands-on experience managing Linux processes with tools like ps, htop, nice, and renice.
  • A fully automated systemd service and timer that runs your monitor on a schedule with journal logging and resource limits.
  • Secret Mission: Add configurable threshold alerting and a --dry-run mode to your monitor.

Are there any prerequisites?

You should be comfortable with basic terminal commands like cd, ls, cp, and chmod. Familiarity with editing files in the terminal is also helpful.

Before We Start

Before writing any monitoring code, you need a working environment with the right tools installed.

In this step, you'll install three utilities that your health monitor depends on, confirm your system's Bash and systemd versions, and create a dedicated project directory.

In this step, get ready to:
  • Install htop, ShellCheck, and sysstat.
  • Verify your Bash and systemd versions.
  • Create the project directory.
Install the required tools

Your health monitor script will use htop for interactive process inspection, ShellCheck for static analysis of your Bash scripts, and sysstat for system performance utilities like iostat.

  • Update your package list and install all three tools by running this command:
sudo apt update && sudo apt install -y htop shellcheck sysstat

What does this command do?

This is actually two commands joined by &&. The first half (sudo apt update) refreshes your system's list of available packages.

The second half (sudo apt install -y) installs the three packages. The -y flag automatically confirms the installation so you don't have to type "yes" manually.

  • Verify each tool installed correctly by running these commands one at a time:
htop --version
shellcheck --version
iostat --version

Each command should print a version number without any errors.

Seeing 'command not found'?

If any of the three verification commands fails, the install may not have completed. Run the full sudo apt install -y htop shellcheck sysstat command again and check for error messages in the output.

Make sure you are running Ubuntu 26.04 LTS. Other distributions may use different package names.

Help me troubleshoot why my apt install failed on Ubuntu

Verify your Bash and systemd versions

Your monitoring script uses Bash features that require version 5.x or higher. The systemd service and timer units you'll create later require systemd 259.

  • Check your Bash version by running:
bash --version

You should see 5.x or higher in the first line of output.

  • Check your systemd version by running:
systemctl --version

You should see systemd 259 in the first line of output.

Why these versions matter

Bash 5.x introduced improvements to associative arrays and other features that production scripts rely on. Ubuntu 26.04 LTS ships with Bash 5.x by default.

systemd 259 supports all the directives you'll use in this project, including CPUWeight= for resource control under cgroups v2 (Ubuntu 26.04 removed cgroup v1 entirely).

Seeing an older version?

If bash --version shows a version below 5.x or systemctl --version shows something other than 259, confirm you are running Ubuntu 26.04 LTS (Resolute Raccoon). You can check with:

lsb_release -a

Help me figure out why my Ubuntu version doesn't match

Create the project directory

All of your script files will live in a single directory on your home folder.

  • Create the project directory and move into it by running these commands:
mkdir -p ~/system-health-monitor
cd ~/system-health-monitor

What does -p do?

The -p flag tells mkdir to create the directory only if it doesn't already exist. Without it, running the command twice would throw an error.

  • Confirm you're in the right place by running:
pwd

You should see /home/your-username/system-health-monitor in the output.

Seeing a different path?

If pwd doesn't show system-health-monitor at the end, run cd ~/system-health-monitor again. The ~ symbol is shorthand for your home directory.

Help me navigate to my project directory

Your environment is ready. Next up, you'll write the system health monitor script with production-grade error handling and structured logging.

Write the System Health Monitor Script

Your project directory is ready. Now it is time to write the actual monitoring script that checks your system's health.

Production Bash scripts follow specific safety patterns that prevent silent failures and make debugging easier. In this step, you will write a script that uses strict mode, structured logging, trap-based cleanup, and input validation.

In this step, get ready to:
  • Create the script file with production safety features, logging, and cleanup.
  • Implement system check functions for CPU, memory, disk, and top processes.
  • Validate the script with ShellCheck and run it.
Create the script file with the production safety preamble

Every production Bash script starts with safety features that catch errors early. Without these, a typo or missing variable can silently corrupt data or produce wrong output.

  • Create and open the script file by running:
nano system-health-monitor.sh

This opens the nano text editor with a new empty file.

  • Paste the following code to set up the script's safety features, configuration, logging, and cleanup:
#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'

# =============================================================================
# System Health Monitor
# Production-grade system monitoring script for Ubuntu
# =============================================================================

# Configuration (defaults)
readonly SCRIPT_NAME="system-health-monitor"
readonly LOG_DIR="/var/log/${SCRIPT_NAME}"
readonly LOG_FILE="${LOG_DIR}/${SCRIPT_NAME}.log"
readonly CPU_THRESHOLD="${CPU_THRESHOLD:-80}"
readonly MEM_THRESHOLD="${MEM_THRESHOLD:-80}"
readonly DISK_THRESHOLD="${DISK_THRESHOLD:-90}"
readonly TOP_PROCESSES=5

# Ensure log directory exists
mkdir -p "${LOG_DIR}"

# -----------------------------------------------------------------------------
# Logging
# -----------------------------------------------------------------------------
log() {
    local level="${1}"
    shift
    local message="${*}"
    local timestamp
    timestamp="$(date '+%Y-%m-%d %H:%M:%S %Z')"
    printf '[%s] [%s] %s\n' "${timestamp}" "${level}" "${message}" | tee -a "${LOG_FILE}"
}

log_info()  { log "INFO"  "$@"; }
log_warn()  { log "WARN"  "$@"; }
log_error() { log "ERROR" "$@"; }

# -----------------------------------------------------------------------------
# Cleanup
# -----------------------------------------------------------------------------
cleanup() {
    local exit_code=$?
    log_info "Health check completed with exit code: ${exit_code}"
    exit "${exit_code}"
}
trap cleanup EXIT

# -----------------------------------------------------------------------------
# Input Validation
# -----------------------------------------------------------------------------
validate_dependencies() {
    local deps=("ps" "free" "df" "awk" "sort" "head" "top")
    for cmd in "${deps[@]}"; do
        if ! command -v "${cmd}" &>/dev/null; then
            log_error "Required command not found: ${cmd}"
            exit 1
        fi
    done
    log_info "All dependencies validated"
}

What does this code do?

  • set -euo pipefail enables Bash strict mode. -e exits on any error. -u treats unset variables as errors. -o pipefail catches failures inside piped commands.
  • IFS=$'\n\t' changes the field separator so that spaces in filenames do not cause word-splitting bugs.
  • The log() function creates timestamped entries with severity levels (INFO, WARN, ERROR). tee -a writes each entry to both the terminal and the log file.
  • trap cleanup EXIT ensures the cleanup() function runs whenever the script exits, whether from success, error, or a signal.
  • validate_dependencies() checks that every command the script needs is available before doing any real work.

You will not see output yet. The next section adds the functions that perform the actual system checks.

Implement the system check functions

Now you will add the functions that actually measure your system's CPU, memory, and disk usage. Each function parses output from standard Linux utilities and logs the results.

  • Below the validate_dependencies function, paste the CPU, memory, and disk check functions:
# -----------------------------------------------------------------------------
# CPU Check
# -----------------------------------------------------------------------------
check_cpu() {
    log_info "--- CPU Usage ---"
    local cpu_idle
    cpu_idle=$(top -bn1 | grep '%Cpu' | awk '{print $8}')
    local cpu_usage
    cpu_usage=$(awk "BEGIN {printf \"%.1f\", 100 - ${cpu_idle}}")
    log_info "CPU Usage: ${cpu_usage}%"

    if (( $(awk "BEGIN {print (${cpu_usage} > ${CPU_THRESHOLD}) ? 1 : 0}") )); then
        log_warn "CPU usage ${cpu_usage}% exceeds threshold of ${CPU_THRESHOLD}%"
    fi
}

# -----------------------------------------------------------------------------
# Memory Check
# -----------------------------------------------------------------------------
check_memory() {
    log_info "--- Memory Usage ---"
    local mem_total mem_used mem_percent
    mem_total=$(free -m | awk 'NR==2 {print $2}')
    mem_used=$(free -m | awk 'NR==2 {print $3}')
    mem_percent=$(awk "BEGIN {printf \"%.1f\", (${mem_used}/${mem_total})*100}")

    log_info "Memory: ${mem_used}MB / ${mem_total}MB (${mem_percent}%)"

    if (( $(awk "BEGIN {print (${mem_percent} > ${MEM_THRESHOLD}) ? 1 : 0}") )); then
        log_warn "Memory usage ${mem_percent}% exceeds threshold of ${MEM_THRESHOLD}%"
    fi
}

# -----------------------------------------------------------------------------
# Disk Check
# -----------------------------------------------------------------------------
check_disk() {
    log_info "--- Disk Usage ---"
    while IFS= read -r line; do
        local disk_percent filesystem mount_point
        disk_percent=$(echo "${line}" | awk '{print $5}' | tr -d '%')
        filesystem=$(echo "${line}" | awk '{print $1}')
        mount_point=$(echo "${line}" | awk '{print $6}')
        log_info "Disk ${filesystem} (${mount_point}): ${disk_percent}%"

        if (( disk_percent > DISK_THRESHOLD )); then
            log_warn "Disk ${filesystem} at ${disk_percent}% exceeds threshold of ${DISK_THRESHOLD}%"
        fi
    done < <(df -h --type=ext4 --type=xfs --type=btrfs 2>/dev/null | tail -n +2 || true)
}

What does this code do?

  • check_cpu() runs top -bn1 in batch mode for one iteration, extracts the idle percentage, and subtracts from 100 to get usage.
  • check_memory() uses free -m to read total and used memory in megabytes, then calculates the percentage.
  • check_disk() reads df -h output filtered to real filesystems (ext4, xfs, btrfs) and logs each mount point's usage.
  • Each function compares usage against its threshold constant and logs a WARN entry if exceeded.
  • Below the check_disk function, paste the top-processes reporter and the main function:
# -----------------------------------------------------------------------------
# Top Processes
# -----------------------------------------------------------------------------
check_top_processes() {
    log_info "--- Top ${TOP_PROCESSES} Processes by CPU ---"
    ps aux --sort=-%cpu | head -n $(( TOP_PROCESSES + 1 )) | tail -n "${TOP_PROCESSES}" | while IFS= read -r line; do
        log_info "  ${line}"
    done

    log_info "--- Top ${TOP_PROCESSES} Processes by Memory ---"
    ps aux --sort=-%mem | head -n $(( TOP_PROCESSES + 1 )) | tail -n "${TOP_PROCESSES}" | while IFS= read -r line; do
        log_info "  ${line}"
    done
}

# =============================================================================
# Main
# =============================================================================
main() {
    log_info "=========================================="
    log_info "System Health Check Started"
    log_info "Hostname: $(hostname)"
    log_info "=========================================="

    validate_dependencies
    check_cpu
    check_memory
    check_disk
    check_top_processes

    log_info "=========================================="
    log_info "System Health Check Complete"
    log_info "=========================================="
}

main "$@"

What does this code do?

  • check_top_processes() uses ps aux --sort=-%cpu and ps aux --sort=-%mem to list the top 5 processes consuming the most CPU and memory.
  • main() orchestrates the entire health check. It logs a header, calls each check function in order, and logs a footer.
  • main "$@" at the bottom invokes the main function, passing any command-line arguments through for future extensibility.
Make it executable and validate

The script is complete. Now you need to save it, make it executable, run ShellCheck for static analysis, and then execute it to see real output.

  • Save and exit nano by pressing Ctrl+O, then Enter to confirm the filename, then Ctrl+X to exit.
  • Make the script executable by running:
chmod +x system-health-monitor.sh
  • Run ShellCheck to validate your script for common bugs by running:
shellcheck system-health-monitor.sh

ShellCheck should produce no output, meaning zero warnings or errors.

What is ShellCheck?

ShellCheck is a static analysis and linting tool for sh/bash scripts. It catches typical beginner and intermediate level syntax errors and pitfalls like quoting issues, word-splitting bugs, and unsafe variable expansions.

Running it before executing your script is standard practice in production DevOps workflows.

  • Run the script to see your system's health report:
sudo ./system-health-monitor.sh

You should see timestamped log entries showing CPU usage, memory usage, disk usage, and the top 5 processes by CPU and memory.

Why sudo?

The script creates a log directory at /var/log/system-health-monitor/ which requires root permissions. Running with sudo grants the script permission to create that directory and write log files there.

ShellCheck reports warnings?

  • Check for missing quotes around variables. ShellCheck will tell you the exact line number and what to fix.
  • Make sure you copied the code exactly. A missing closing brace or mismatched quote will cause parse errors.
  • If you see a warning about tee -a or read -r, double-check that the flags are spelled correctly with the dash.

My script fails ShellCheck or won't run. What should I check?

Permission denied when running?

  • Make sure you ran chmod +x system-health-monitor.sh before executing.
  • Make sure you are using sudo when running the script, since it writes to /var/log/.

✔️ Awesome, I've got everything!

Great. Make sure you have saved system-health-monitor.sh and confirmed it runs with timestamped output.

ⓧ I'd like to double check the full code

Here is the complete system-health-monitor.sh file. Compare your version against this:

#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'

# =============================================================================
# System Health Monitor
# Production-grade system monitoring script for Ubuntu
# =============================================================================

# Configuration (defaults)
readonly SCRIPT_NAME="system-health-monitor"
readonly LOG_DIR="/var/log/${SCRIPT_NAME}"
readonly LOG_FILE="${LOG_DIR}/${SCRIPT_NAME}.log"
readonly CPU_THRESHOLD="${CPU_THRESHOLD:-80}"
readonly MEM_THRESHOLD="${MEM_THRESHOLD:-80}"
readonly DISK_THRESHOLD="${DISK_THRESHOLD:-90}"
readonly TOP_PROCESSES=5

# Ensure log directory exists
mkdir -p "${LOG_DIR}"

# -----------------------------------------------------------------------------
# Logging
# -----------------------------------------------------------------------------
log() {
    local level="${1}"
    shift
    local message="${*}"
    local timestamp
    timestamp="$(date '+%Y-%m-%d %H:%M:%S %Z')"
    printf '[%s] [%s] %s\n' "${timestamp}" "${level}" "${message}" | tee -a "${LOG_FILE}"
}

log_info()  { log "INFO"  "$@"; }
log_warn()  { log "WARN"  "$@"; }
log_error() { log "ERROR" "$@"; }

# -----------------------------------------------------------------------------
# Cleanup
# -----------------------------------------------------------------------------
cleanup() {
    local exit_code=$?
    log_info "Health check completed with exit code: ${exit_code}"
    exit "${exit_code}"
}
trap cleanup EXIT

# -----------------------------------------------------------------------------
# Input Validation
# -----------------------------------------------------------------------------
validate_dependencies() {
    local deps=("ps" "free" "df" "awk" "sort" "head" "top")
    for cmd in "${deps[@]}"; do
        if ! command -v "${cmd}" &>/dev/null; then
            log_error "Required command not found: ${cmd}"
            exit 1
        fi
    done
    log_info "All dependencies validated"
}

# -----------------------------------------------------------------------------
# CPU Check
# -----------------------------------------------------------------------------
check_cpu() {
    log_info "--- CPU Usage ---"
    local cpu_idle
    cpu_idle=$(top -bn1 | grep '%Cpu' | awk '{print $8}')
    local cpu_usage
    cpu_usage=$(awk "BEGIN {printf \"%.1f\", 100 - ${cpu_idle}}")
    log_info "CPU Usage: ${cpu_usage}%"

    if (( $(awk "BEGIN {print (${cpu_usage} > ${CPU_THRESHOLD}) ? 1 : 0}") )); then
        log_warn "CPU usage ${cpu_usage}% exceeds threshold of ${CPU_THRESHOLD}%"
    fi
}

# -----------------------------------------------------------------------------
# Memory Check
# -----------------------------------------------------------------------------
check_memory() {
    log_info "--- Memory Usage ---"
    local mem_total mem_used mem_percent
    mem_total=$(free -m | awk 'NR==2 {print $2}')
    mem_used=$(free -m | awk 'NR==2 {print $3}')
    mem_percent=$(awk "BEGIN {printf \"%.1f\", (${mem_used}/${mem_total})*100}")

    log_info "Memory: ${mem_used}MB / ${mem_total}MB (${mem_percent}%)"

    if (( $(awk "BEGIN {print (${mem_percent} > ${MEM_THRESHOLD}) ? 1 : 0}") )); then
        log_warn "Memory usage ${mem_percent}% exceeds threshold of ${MEM_THRESHOLD}%"
    fi
}

# -----------------------------------------------------------------------------
# Disk Check
# -----------------------------------------------------------------------------
check_disk() {
    log_info "--- Disk Usage ---"
    while IFS= read -r line; do
        local disk_percent filesystem mount_point
        disk_percent=$(echo "${line}" | awk '{print $5}' | tr -d '%')
        filesystem=$(echo "${line}" | awk '{print $1}')
        mount_point=$(echo "${line}" | awk '{print $6}')
        log_info "Disk ${filesystem} (${mount_point}): ${disk_percent}%"

        if (( disk_percent > DISK_THRESHOLD )); then
            log_warn "Disk ${filesystem} at ${disk_percent}% exceeds threshold of ${DISK_THRESHOLD}%"
        fi
    done < <(df -h --type=ext4 --type=xfs --type=btrfs 2>/dev/null | tail -n +2 || true)
}

# -----------------------------------------------------------------------------
# Top Processes
# -----------------------------------------------------------------------------
check_top_processes() {
    log_info "--- Top ${TOP_PROCESSES} Processes by CPU ---"
    ps aux --sort=-%cpu | head -n $(( TOP_PROCESSES + 1 )) | tail -n "${TOP_PROCESSES}" | while IFS= read -r line; do
        log_info "  ${line}"
    done

    log_info "--- Top ${TOP_PROCESSES} Processes by Memory ---"
    ps aux --sort=-%mem | head -n $(( TOP_PROCESSES + 1 )) | tail -n "${TOP_PROCESSES}" | while IFS= read -r line; do
        log_info "  ${line}"
    done
}

# =============================================================================
# Main
# =============================================================================
main() {
    log_info "=========================================="
    log_info "System Health Check Started"
    log_info "Hostname: $(hostname)"
    log_info "=========================================="

    validate_dependencies
    check_cpu
    check_memory
    check_disk
    check_top_processes

    log_info "=========================================="
    log_info "System Health Check Complete"
    log_info "=========================================="
}

main "$@"

Your monitoring script is working and producing structured log output. Next up, you will explore process management and learn how to control process priority with nice and renice.

Manage Processes and Tune Performance

Your health monitor script is now checking CPU, memory, and disk usage with proper thresholds and warnings. That gives you a solid baseline view of system health.

But knowing that CPU is at 85% doesn't tell you which process is responsible. In this step, you'll learn to identify resource-hungry processes, control their priority, and verify your script's top-processes reporting.

In this step, get ready to:
  • Inspect running processes with ps and htop.
  • Control process priority with nice and renice.
  • Verify the top-processes reporting in your health monitor script.
Explore running processes interactively

Every program running on your Linux system is a process. When your health monitor reports high CPU or memory, you need a way to find out exactly which process is consuming those resources.

  • View the top 20 CPU-consuming processes by running this command:
ps aux --sort=-%cpu | head -20

What does this command do?

The ps aux command lists all running processes with details like CPU usage, memory usage, and the command name.

The --sort=-%cpu flag sorts by CPU percentage in descending order. Piping to head -20 limits the output to the top 20 results.

You should see a table with columns like USER, PID, %CPU, %MEM, and COMMAND showing your system's busiest processes.

  • View the top 10 memory-consuming processes by running:
ps aux --sort=-%mem | head -10

This shows the same process table sorted by memory usage instead. You now have two quick one-liners for diagnosing resource spikes.

  • Launch the interactive process viewer by running:
htop

What is htop?

htop is a real-time, interactive process viewer that provides a colour-coded overview of your system's resources. Unlike ps, it updates continuously and lets you sort, filter, and send signals to processes without leaving the tool.

  • Press F6 to open the sort menu. Use the arrow keys to select PERCENT_CPU and press Enter to sort processes by CPU usage.
  • Press F4 to open the filter. Type the name of a process you saw in the list to narrow down results.
  • Press Esc to clear the filter.
  • Press F9 to open the signal menu. This is how you send signals like SIGTERM or SIGKILL to a process. Press Esc to cancel without sending a signal.
  • Press q to quit htop and return to your terminal.

Use htop for real-time debugging

When a server is slow, htop is usually the first tool DevOps engineers open. It gives you an instant visual of what's consuming resources without needing to remember ps flags.

Control process priority with nice and renice

Linux assigns every process a nice value that determines its scheduling priority. The range goes from -20 (highest priority) to 19 (lowest priority), with a default of 0.

By adjusting nice values, you can ensure critical processes get more CPU time while background tasks stay out of the way.

  • Start a background process to experiment with by running:
sleep 300 &

What does the & do?

The & at the end runs the command in the background so your terminal stays free. The shell prints the process ID (PID) when it starts.

  • Check the nice value of your background process by running:
ps -o pid,ni,comm -p $!

What does this show?

The $! variable holds the PID of the last background process you started. The -o pid,ni,comm flag formats the output to show only the PID, nice value (NI), and command name.

You should see NI showing 0, which is the default nice value.

  • Lower the priority of your background process by running:
renice 10 -p $!

You should see a confirmation message showing the new priority value of 10. This means the process now yields CPU time to other tasks more willingly.

  • Verify the change took effect by running:
ps -o pid,ni,comm -p $!

The NI column should now show 10 instead of 0.

  • Start a new process with a custom priority from the start by running:
nice -n 15 sleep 300 &

nice vs renice

Use nice to launch a process with a specific priority. Use renice to change the priority of a process that is already running.

Setting a nice value higher than 0 (like 10 or 15) makes the process lower priority. Setting it below 0 (like -5) makes it higher priority, but that requires root permissions.

  • Clean up your background processes by running:
kill %1 %2 2>/dev/null; echo "Background processes stopped"

Getting a "no such job" error?

If one of the sleep processes already completed or was killed, you'll see a "no such job" message. That's fine. The 2>/dev/null silences most of these, but you may still see a notification from the shell.

help me clean up background processes

Verify top-processes reporting in your script

Your health monitor script already includes a check_top_processes function that uses the same ps aux --sort technique you just practiced. Let's verify it passes linting and produces the expected output.

  • Run ShellCheck against your script to confirm there are no issues:
shellcheck system-health-monitor.sh

ShellCheck should report no errors or warnings.

Seeing ShellCheck warnings?

If ShellCheck reports warnings, read the error code it gives you (e.g., SC2086). Each code links to a wiki page explaining the issue and the fix.

help me fix ShellCheck warnings in my script

  • Run your health monitor script to verify the top-processes output:
sudo ./system-health-monitor.sh

Scroll through the output. You should see sections labelled --- Top 5 Processes by CPU --- and --- Top 5 Processes by Memory --- with process details listed below each heading.

What does check_top_processes do?

The function runs ps aux --sort=-%cpu and ps aux --sort=-%mem to find the top 5 processes by each metric. It pipes the results through head and tail to skip the header row and limit to exactly 5 entries.

This is the same technique you used manually earlier. Now it runs automatically every time the health monitor executes.

Your process management skills are ready. Next up, you'll deploy your script as a managed systemd service with resource controls and sandboxing.

Deploy as a systemd Service

In Step 3, you explored running processes and learned how to control their priority with nice and renice. Your health monitor script now reports the top resource-consuming processes on the system.

Right now, you have to manually run the script every time you want a health check. In a real production environment, monitoring needs to happen automatically and reliably. That is exactly what systemd services are for.

In this step, you will deploy your script as a managed systemd service with resource controls that limit how much CPU and memory the monitor can consume.

In this step, get ready to:
  • Create a systemd service unit file with resource controls and sandboxing.
  • Install the script system-wide and run it as a managed service.
  • Inspect journal logs and verify resource limits are applied.
Create the systemd service unit file

A systemd service unit is a configuration file that tells systemd how to run, stop, and manage a process. It also lets you apply resource controls so the service cannot consume more than its fair share of the system.

  • Open the service unit file for editing by running this command:
sudo nano /etc/systemd/system/system-health-monitor.service
  • Paste the following service unit configuration into the file:
[Unit]
Description=System Health Monitor
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/system-health-monitor.sh
Nice=10
CPUWeight=50
MemoryMax=128M
StandardOutput=journal
StandardError=journal
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=/var/log/system-health-monitor

[Install]
WantedBy=multi-user.target

What does this service file do?

The file is split into three sections that each control a different aspect of the service:

  • [Unit] names the service and tells systemd to wait until the network is ready before starting it.
  • [Service] defines how to run the script. Type=oneshot means systemd waits for the script to finish before marking the service as complete.
  • [Install] tells systemd when this service should be available (at multi-user boot).

What are the resource controls doing?

systemd uses cgroups (control groups) to enforce resource limits on services. Ubuntu 26.04 uses cgroup v2 exclusively.

  • Nice=10 lowers the scheduling priority so the monitor does not compete with important workloads.
  • CPUWeight=50 gives the service a smaller share of CPU time relative to other services (default weight is 100).
  • MemoryMax=128M hard-caps memory usage at 128 megabytes. If the service tries to exceed this, it gets killed.
  • ProtectSystem=strict makes most of the filesystem read-only, and ReadWritePaths= grants write access only to the log directory.
  • Save and exit nano by pressing Ctrl+O, then Enter, then Ctrl+X.
  • Confirm the file was created by running:
cat /etc/systemd/system/system-health-monitor.service

You should see the full unit file contents printed in your terminal, matching what you just pasted.

File is empty or missing?

Make sure you used sudo before nano. The /etc/systemd/system/ directory requires root permissions to write to.

Also check that you pressed Ctrl+O (the letter O, not zero) to save before exiting.

Help me troubleshoot my systemd service file creation.

Install the script system-wide and test the service

The service unit file points to /usr/local/bin/system-health-monitor.sh as its ExecStart path. You need to copy your script there so systemd can find it.

  • Copy the script to the system-wide bin directory and create the log directory by running these commands:
sudo cp system-health-monitor.sh /usr/local/bin/
sudo mkdir -p /var/log/system-health-monitor

Why copy to /usr/local/bin/?

/usr/local/bin/ is the standard location for locally installed scripts and binaries on Linux. It is in the system PATH, so systemd can find and execute the script without needing an absolute path to your home directory.

  • Verify the copy and directory creation by running:
ls /usr/local/bin/system-health-monitor.sh /var/log/system-health-monitor

You should see /usr/local/bin/system-health-monitor.sh listed in the output, confirming both the script and log directory exist.

  • Reload the systemd daemon so it picks up the new service file, then start the service by running these commands:
sudo systemctl daemon-reload
sudo systemctl start system-health-monitor.service
  • Check the service status by running:
sudo systemctl status system-health-monitor.service

You should see the service listed as inactive (dead) with status=0/SUCCESS. This is correct. Because the service is Type=oneshot, it runs once and exits. The inactive (dead) state with exit code 0 means it completed successfully.

Seeing a failed status?

If the status shows failed, check that:

  • The script exists at /usr/local/bin/system-health-monitor.sh and is executable (chmod +x was applied in Step 2).
  • The log directory /var/log/system-health-monitor exists. The ProtectSystem=strict setting prevents the script from creating it at runtime.
  • There are no typos in the ExecStart path. It must be an absolute path.

Help me debug my failed systemd service.

Inspect resource controls and logs

One of the biggest advantages of running scripts through systemd is that all output goes to the systemd journal. You can query it with journalctl instead of hunting through log files manually.

  • View the health check output from the journal by running:
journalctl -u system-health-monitor.service --no-pager

You should see the timestamped log entries from your health check, including CPU usage, memory usage, disk usage, and the top processes report.

What is journalctl?

journalctl is the tool for querying the systemd journal. The -u flag filters output to a specific unit (your service). The --no-pager flag prints everything directly to the terminal instead of opening a scrollable viewer.

  • Verify the resource control properties are applied by running:
systemctl show system-health-monitor.service -p Nice,CPUWeight,MemoryMax

You should see output confirming the three resource properties:

Nice=10
CPUWeight=50
MemoryMax=134217728

Why does MemoryMax show 134217728?

systemd stores memory values in bytes internally. 134217728 bytes is exactly 128 megabytes (128 x 1024 x 1024). The 128M shorthand in the unit file is a convenience that systemd converts to bytes.

Properties show infinity or 0?

If MemoryMax shows infinity or CPUWeight shows a different value, double check your service file for typos. Make sure there are no spaces around the = signs.

After fixing the file, always run sudo systemctl daemon-reload before starting the service again.

Help me fix my systemd resource control properties.

✔️ Awesome, I've got everything!

Great. Make sure you have saved your service file and confirmed the resource properties match.

ⓧ I'd like to double check the full code

Here is the complete service unit file at /etc/systemd/system/system-health-monitor.service:

[Unit]
Description=System Health Monitor
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/system-health-monitor.sh
Nice=10
CPUWeight=50
MemoryMax=128M
StandardOutput=journal
StandardError=journal
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=/var/log/system-health-monitor

[Install]
WantedBy=multi-user.target

Your health monitor is now a proper systemd service with resource guardrails and sandboxing. Next up, you will create a systemd timer to run this service automatically on a schedule.

Automate with a systemd Timer

Your service works perfectly when you start it manually. In a real production environment, you need it to run automatically on a recurring schedule without human intervention.

In this step, you will create a systemd timer unit that triggers your health monitor every 5 minutes. The timer is the scheduling companion to your service unit, and together they replace the older cron approach with a modern, journal-integrated solution.

In this step, get ready to:
  • Create the timer unit file with calendar-based scheduling.
  • Enable and start the timer so it survives reboots.
  • Monitor automated execution through journal logs.
Create the timer unit file

A systemd timer is a separate unit file that tells systemd when to activate a corresponding service. The timer file lives alongside your service file and shares the same base name.

  • Open the timer unit file for editing by running this command:
sudo nano /etc/systemd/system/system-health-monitor.timer
  • Paste the following timer configuration into the file:
[Unit]
Description=Run System Health Monitor every 5 minutes

[Timer]
OnCalendar=*:0/5
Persistent=true
RandomizedDelaySec=30

[Install]
WantedBy=timers.target

What does this configuration do?

  • OnCalendar=*:0/5 means "every 5 minutes, on the 0th, 5th, 10th, 15th... minute of every hour." The asterisk matches every day.
  • Persistent=true catches up on missed runs. If your system was off when the timer should have fired, it triggers immediately on boot.
  • RandomizedDelaySec=30 adds up to 30 seconds of random delay. This prevents all timers on the system from firing at exactly the same moment.
  • WantedBy=timers.target ensures the timer activates automatically when the system reaches the timers target during boot.
  • Save the file by pressing Ctrl+O, then Enter to confirm the filename.
  • Exit nano by pressing Ctrl+X.

File not saving?

Make sure you ran the nano command with sudo. Files in /etc/systemd/system/ require root permissions to write.

Help me troubleshoot file permission issues with nano and sudo

✔️ Awesome, I've got everything!

Great. Double check you have saved the file and exited nano.

ⓧ I'd like to double check the full code

Here is the complete timer unit file. Make sure yours matches exactly:

[Unit]
Description=Run System Health Monitor every 5 minutes

[Timer]
OnCalendar=*:0/5
Persistent=true
RandomizedDelaySec=30

[Install]
WantedBy=timers.target
Enable and start the timer

Creating the file is not enough. You need to tell systemd to reload its configuration, then enable the timer (so it starts on boot) and start it (so it begins now).

  • Reload systemd to pick up the new timer file by running this command:
sudo systemctl daemon-reload
  • Enable the timer so it persists across reboots by running:
sudo systemctl enable system-health-monitor.timer
  • Start the timer immediately by running:
sudo systemctl start system-health-monitor.timer

What is the difference between enable and start?

enable creates a symlink so the timer activates every time the system boots. start activates it right now without waiting for a reboot. You need both to get immediate scheduling that also survives restarts.

  • Verify the timer is active and scheduled by running:
sudo systemctl list-timers --all | grep system-health

You should see a row showing your timer with a NEXT column displaying the upcoming trigger time and a LEFT column showing how long until it fires.

Timer not showing in the list?

Check that the file name ends in .timer (not .service). Also confirm you ran sudo systemctl daemon-reload after saving the file. Without the reload, systemd does not know the timer exists.

Help me troubleshoot my systemd timer not appearing

Monitor automated execution

Your timer is now counting down to the next 5-minute mark. You can either wait for it to fire naturally, or trigger a manual run to confirm everything works end-to-end.

  • Trigger the service manually to test without waiting by running:
sudo systemctl start system-health-monitor.service
  • Check the recent journal logs for your service by running:
journalctl -u system-health-monitor.service --since "5 minutes ago"

You should see timestamped health check entries showing CPU, memory, disk usage, and top processes. Each run produces a full set of log lines between the "Started" and "Complete" markers.

  • View the timer's current status and scheduling details by running:
systemctl status system-health-monitor.timer

The output shows whether the timer is active, when it last triggered, and when the next trigger is scheduled.

Why systemd timers instead of cron?

Cron is the traditional Linux scheduler, but systemd timers are the modern standard for Ubuntu systems. Timers offer four key advantages that cron cannot provide.

  • Persistent catch-up: missed runs fire after reboot (cron silently skips them).
  • Dependency awareness: timers can wait for network or other services before firing.
  • Journal integration: all output flows through journalctl with timestamps and filtering.
  • Per-unit resource controls: each timer's service gets its own CPU, memory, and sandboxing limits.

Not seeing any log entries?

If journalctl returns empty output, confirm the service ran successfully with systemctl status system-health-monitor.service. If it shows a failure, check that /usr/local/bin/system-health-monitor.sh exists and is executable.

Help me debug why my systemd service isn't producing journal output

Secret mission

Add Threshold Alerting and Dry-Run Mode

Your monitor works, but production scripts need safe testing modes and actionable alerts. In this secret mission, you'll add a --dry-run flag that lets you verify behavior without writing files, configurable thresholds via environment variables, and a separate alerts log that captures only the moments something goes wrong.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally with no paid services, so there are no ongoing costs.

Resources you used:

  • Script installed at /usr/local/bin/system-health-monitor.sh.
  • Systemd service unit at /etc/systemd/system/system-health-monitor.service.
  • Systemd timer unit at /etc/systemd/system/system-health-monitor.timer.
  • Log directory at /var/log/system-health-monitor/.
  • Project directory at ~/system-health-monitor/.
  • Packages installed via apt: htop, shellcheck, sysstat.

Keep everything running

No action needed. Choose this if you are still actively building or want to keep the health monitor running on its 5-minute schedule.

  • The timer will continue running every 5 minutes, writing small log entries to the journal and /var/log/system-health-monitor/.
  • Log files will grow slowly over time. You can check their size periodically with du -sh /var/log/system-health-monitor/.
  • The installed packages (htop, shellcheck, sysstat) remain available as general-purpose tools on your system.

Pause - I'll come back to this later

Stop the timer so no new health checks run, but keep all files in place so you can restart later.

  • Stop the timer by running:
sudo systemctl stop system-health-monitor.timer
  • This stops scheduling but keeps the unit files, script, and logs intact.
  • To resume later, run:
sudo systemctl start system-health-monitor.timer

Delete - I don't want to use this again

Remove all project resources and start fresh. This disables the timer, removes all unit files, deletes the script, and clears the logs.

  • Disable and stop the timer, remove unit files, the installed script, and the log directory by running:
sudo systemctl disable --now system-health-monitor.timer && sudo rm /etc/systemd/system/system-health-monitor.{service,timer} && sudo rm /usr/local/bin/system-health-monitor.sh && sudo rm -rf /var/log/system-health-monitor && sudo systemctl daemon-reload
  • Remove the project directory by running:
rm -rf ~/system-health-monitor
  • Uninstall the packages you installed for this project by running:
sudo apt remove -y htop shellcheck sysstat
  • Confirm the timer is gone by running:
sudo systemctl list-timers --all | grep system-health
  • You should see no output, confirming the timer has been fully removed.

Nice Work!

Nice Work!

You just built a complete system health monitoring solution on Ubuntu and deployed it as a managed systemd service. These are the exact shell scripting and Linux service management skills that DevOps teams rely on every day.

You've learned how to:

  • Built a production-grade Bash script using strict mode, structured logging, trap cleanup, and input validation.
  • Managed Linux processes with ps, htop, nice, and renice to monitor resource usage and control process priority.
  • Deployed the script as a systemd service with resource controls (Nice, CPUWeight, MemoryMax) and automated it with a persistent systemd timer for scheduled execution and journal logging.
  • Secret Mission: Added configurable threshold alerting, environment variable overrides, and a --dry-run mode to your monitor.

Ready to quiz yourself?