Shell Scripting Part 3: Production Tools

Build a modular ops-toolkit with ShellCheck, parallel processing, and API monitoring.

Introduction

30 Second Summary

Every DevOps team has a folder of scripts that grew organically and became impossible to maintain. One developer's quick fix becomes another's mystery when there are no flags, no shared functions, and no linting.

In this project, you will build an ops-toolkit: a modular, production-grade monitoring system with a professional CLI, parallel log processing, and automated API health checks.

What You'll Build

When you run bin/healthcheck -v -e prod, your toolkit sources a shared library, parses flags, checks system health, and saves a timestamped report that any teammate could read and trust.

By the end of this project, you'll have:

  • A shared library (lib/common.sh) with logging, validation, and a dry-run wrapper that every script in your toolkit sources automatically.
  • A professional CLI with getopts-parsed flags (-v, -d, -e, -h) that produces formatted system health reports saved to timestamped log files.
  • A parallel log processor that crunches sample logs with xargs -P and displays a timing comparison proving the multi-core speedup.
  • Secret Mission: Build an API health monitor with exponential backoff retry logic, and integrate ShellCheck into a Git pre-commit hook so no unclean script can ever be committed to your ops-toolkit repo.

Do I need to complete Parts 1 and 2 first?

Yes. This project builds on the strict mode, trap cleanup, log function, arrays, and config file patterns from Part 2. You should also have Linux Mint XFCE with Bash, Git, and a GitHub account already set up.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters for your DevOps toolkit.

Set Up Your Project Scaffold

Before writing production scripts, you need two things: tools that enforce code quality, and a project layout that real teams recognize.

Without this structure, scripts become unmaintainable single-file blobs. In this step, you will install ShellCheck (a static analysis tool for Bash) and jq (a JSON processor), then scaffold the ops-toolkit project with a professional directory structure.

In this step, get ready to:
  • Install ShellCheck, jq, and tree via apt.
  • Create the ~/ops-toolkit project with bin/, lib/, config/, and logs/ directories.
  • Initialize a Git repository with an initial commit.
Install ShellCheck and jq

ShellCheck catches unquoted variables, word-splitting bugs, and common pitfalls that manual testing misses. jq lets your scripts parse and construct JSON payloads cleanly.

  • Install all three tools by running this command:
sudo apt install shellcheck jq tree -y

What does this command do?

  • sudo runs the command with administrator privileges, which is required for installing system packages.
  • apt install downloads and installs packages from the Linux Mint repositories.
  • -y automatically confirms the install prompt so you do not have to type "yes" manually.
  • tree is a utility that displays directory structures visually. You will use it to verify your project layout.
  • Verify ShellCheck installed correctly by running:
shellcheck --version
  • Verify jq installed correctly by running:
jq --version

✔️ I see version numbers for both

Both tools are ready to go. shellcheck --version prints a version like 0.9.0 and jq --version prints something like jq-1.6. The exact numbers may differ on your system.

ⓧ Command not found

The apt install may not have completed successfully.

  • Scroll up in your terminal and look for any error messages from the apt install output.
  • Try running the install command again without -y to see the full prompts:
sudo apt install shellcheck jq tree
  • If you see an "Unable to locate package" error, run sudo apt update first to refresh your package list, then retry the install.
Create the project directory structure

Production toolkits separate concerns into directories: executables in bin/, shared code in lib/, settings in config/, and output in logs/.

  • Create the ops-toolkit project folder with all subdirectories by running these commands:
mkdir -p ~/ops-toolkit/{bin,lib,config,logs}
cd ~/ops-toolkit

What does this command do?

  • mkdir -p creates the directory and any parent directories that do not exist yet.
  • The curly brace syntax {bin,lib,config,logs} expands into four separate mkdir calls, creating all subdirectories in one shot.
  • cd ~/ops-toolkit moves your terminal into the new project folder so all future commands run from there.
  • Initialize a Git repository in the project by running:
git init

You should see a message like Initialized empty Git repository in /home/.../ops-toolkit/.git/.

  • Confirm the directory structure exists by running:
ls

You should see four directories listed: bin, config, lib, and logs.

Not seeing four directories?

  • Make sure you ran the mkdir command with the -p flag and the curly brace syntax exactly as shown.
  • Confirm you are in the right directory by running pwd. It should print /home/your-username/ops-toolkit.
  • help me debug my directory structure
Create the configuration placeholder and make your initial commit

A placeholder config file gives your scripts a known location to read settings from, even before you wire up the full logic.

  • Create config/healthcheck.conf by running:
cat << 'EOF' > config/healthcheck.conf
# ops-toolkit healthcheck configuration placeholder
ALERT_THRESHOLD=5
CHECK_INTERVAL=30
EOF

What does this command do?

  • cat << 'EOF' is a here document. Everything between the two EOF markers is written into the file as-is.
  • ALERT_THRESHOLD and CHECK_INTERVAL are placeholder values that your healthcheck script will read later.
  • Verify the file was created correctly by running:
cat config/healthcheck.conf

You should see the comment line followed by the two variable assignments.

  • Stage all files and make your initial commit by running:
git add .
git commit -m "Initial project scaffold"

Git confirms the commit with a summary showing files changed.

Before you run the final verification, what directories do you expect to see in the tree output?

  • Verify the complete project layout by running:
tree ~/ops-toolkit

You should see a tree with four directories (bin/, config/, lib/, logs/) and one file (config/healthcheck.conf).

✔️ Awesome, I've got everything!

Your project scaffold is in place and version-controlled.

ⓧ I'd like to double check the full code

Here is what config/healthcheck.conf should contain:

# ops-toolkit healthcheck configuration placeholder
ALERT_THRESHOLD=5
CHECK_INTERVAL=30

And your project tree should look like this:

ops-toolkit/
├── bin/
├── config/
│   └── healthcheck.conf
├── lib/
└── logs/

Tree command not found?

  • If tree is not recognized, it may not have installed correctly. Run sudo apt install tree again.
  • As an alternative, run find ~/ops-toolkit -type f to list all files, or ls -R ~/ops-toolkit to see the recursive directory listing.
  • help me verify my ops-toolkit project structure

Your environment is locked in and your project has a professional foundation. Next up, you will build the shared library that every script in this toolkit sources.

Build the Shared Library

In Step 1, you scaffolded a professional project layout. But right now those folders are empty shells.

In Part 2, your log function and helpers lived inside a single script. That works for one-off automation, but real teams extract shared code into a library file that every script sources. A change to the library instantly propagates to every script that uses it.

In this step, you will create lib/common.sh and prove it works by sourcing it from a minimal healthcheck stub.

In this step, get ready to:
  • Create a shared library with logging, error handling, and a dry-run wrapper.
  • Add validation functions that check for required commands and environment variables.
  • Wire up a healthcheck stub that sources the library and proves the path resolution works.
Create the shared library with core functions

The library file is the foundation everything else builds on. It provides a consistent logging format, a way to bail out on errors, and a dry-run mode for safe testing.

  • Create the shared library file by running:
cat > lib/common.sh << 'EOF'
#!/usr/bin/env bash
# ops-toolkit shared library
# Source this file; do not execute directly.

set -euo pipefail

readonly OPS_TOOLKIT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
readonly OPS_LOGS_DIR="${OPS_TOOLKIT_DIR}/logs"

# Globals set by CLI parsing in the calling script
DRY_RUN="${DRY_RUN:-0}"
VERBOSE="${VERBOSE:-0}"

log() {
    local level="$1"
    shift
    local message="$*"
    local timestamp
    timestamp="$(date '+%Y-%m-%d %H:%M:%S')"
    printf '[%s] [%s] %s\n' "$timestamp" "$level" "$message" >&2
}
EOF

What does this code do?

This is the first half of your shared library. Here is what each section handles:

  • set -euo pipefail activates strict mode. Any unset variable, failed command, or broken pipe halts the script immediately.
  • OPS_TOOLKIT_DIR resolves the project root using BASH_SOURCE and dirname. This works regardless of where the script is called from.
  • DRY_RUN and VERBOSE use the ${VAR:-default} pattern so calling scripts can override them before sourcing.
  • log() uses local variables (no namespace pollution) and printf to stderr so log output never mixes with command output.
  • Test the library so far by sourcing it and calling log:
source lib/common.sh && log INFO "Library loaded"

You should see a timestamped line like [2026-08-07 14:23:01] [INFO] Library loaded printed to your terminal.

Don't see any output?

  • Make sure you are in the ~/ops-toolkit directory. The BASH_SOURCE path resolution needs to find lib/common.sh relative to where you run it.
  • Check for typos in the heredoc. The EOF closing marker must be on its own line with no leading spaces.

Stuck? Help me debug why source lib/common.sh is not printing a log line

Now add the error-handling and dry-run functions below the log() function.

  • Append the die() and run_cmd() functions to lib/common.sh by running:
cat >> lib/common.sh << 'EOF'

die() {
    log ERROR "$@"
    exit 1
}

run_cmd() {
    if [[ "$DRY_RUN" -eq 1 ]]; then
        log DRY-RUN "Would execute: $*"
        return 0
    fi
    "$@"
}
EOF

What do these functions do?

  • die() logs an ERROR-level message and immediately exits the script. This gives you a one-liner for fatal failures.
  • run_cmd() is a dry-run wrapper. When DRY_RUN=1, it logs what it would do without actually executing. When dry-run is off, it runs the command normally.
  • Verify dry-run mode works by running:
source lib/common.sh && DRY_RUN=1 run_cmd echo "hello world"

You should see [DRY-RUN] Would execute: echo hello world (with a timestamp prefix). The echo command was not actually run.

Seeing 'hello world' printed instead?

  • If you see the actual echo output, DRY_RUN is not being set to 1. Make sure DRY_RUN=1 appears before run_cmd on the same line (no semicolon between them).
  • Check that the if [[ "$DRY_RUN" -eq 1 ]] comparison uses double brackets and the -eq operator.

Still stuck? Help me debug why run_cmd is executing the command instead of printing the dry-run message

Add validation functions

Production scripts need to fail fast when a required tool is missing or an environment variable is unset. These two functions handle both cases.

  • Append the validation functions to lib/common.sh by running:
cat >> lib/common.sh << 'EOF'

validate_env() {
    local cmd
    for cmd in "$@"; do
        if ! command -v "$cmd" > /dev/null 2>&1; then
            die "Required command not found: $cmd"
        fi
    done
}

require_var() {
    local var_name="$1"
    local var_value="${!var_name:-}"
    if [[ -z "$var_value" ]]; then
        die "Required environment variable not set: $var_name"
    fi
}
EOF

What do these functions do?

  • validate_env() loops over any command names you pass it and checks each with command -v. If a tool is missing, it calls die() with a clear message.
  • require_var() uses Bash indirect expansion (${!var_name:-}) to check if a named variable has a value. This catches misconfigured environments before the script does real work.
  • Verify validate_env() works by checking for tools you already installed:
source lib/common.sh && validate_env git shellcheck jq && log INFO "All tools present"

You should see [INFO] All tools present (with a timestamp). If any of those three tools were missing, the script would have exited with an error.

Seeing 'Required command not found'?

  • If it reports shellcheck or jq not found, those tools may not have installed correctly in Step 1. Re-run sudo apt install shellcheck jq and try again.
  • If it reports git not found, confirm Git is installed with git --version.

Need more help? Help me debug why validate_env says a command is not found even though I installed it

✔️ Awesome, I've got everything!

Great. Double check you have saved lib/common.sh.

ⓧ I'd like to double check the full code

Your lib/common.sh should contain these sections in order. Scroll up and confirm each chunk is present:

  • Shebang, comments, and set -euo pipefail.
  • readonly OPS_TOOLKIT_DIR and OPS_LOGS_DIR variables.
  • DRY_RUN and VERBOSE globals with defaults.
  • log() function using local variables and printf to stderr.
  • die() function calling log ERROR then exit 1.
  • run_cmd() function checking DRY_RUN before executing.
  • validate_env() looping over arguments with command -v.
  • require_var() using indirect expansion ${!var_name:-}.
Create the healthcheck stub to prove sourcing works

A library is only useful if other scripts can source it reliably. The trickiest part is resolving the path back to lib/common.sh regardless of where the script is invoked from.

  • Create the healthcheck stub in bin/healthcheck by running:
cat > bin/healthcheck << 'EOF'
#!/usr/bin/env bash
# ops-toolkit healthcheck stub

set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=../lib/common.sh
source "${SCRIPT_DIR}/../lib/common.sh"

log INFO "Starting"
exit 0
EOF

How does the path resolution work?

  • BASH_SOURCE[0] always holds the path to the currently executing script, even if it was called via a symlink or from another directory.
  • dirname strips the filename, giving you the directory. Wrapping it in cd ... && pwd resolves any relative path to an absolute one.
  • The # shellcheck source= comment tells ShellCheck where to find the sourced file for static analysis. Without it, ShellCheck cannot verify the library's functions.
  • Make the script executable by running:
chmod +x bin/healthcheck
  • Run the stub to confirm it sources the library correctly:
bin/healthcheck

You should see a timestamped log line: [INFO] Starting. The script sourced lib/common.sh from bin/ and the path resolution worked.

Seeing 'No such file or directory'?

  • Make sure you ran the command from ~/ops-toolkit. Run pwd to confirm.
  • Check that lib/common.sh exists by running ls lib/common.sh. If missing, re-run the earlier cat > commands.
  • Verify the SCRIPT_DIR line uses BASH_SOURCE[0] (with the index). A missing [0] can cause unexpected behavior.

Still stuck? Help me debug why bin/healthcheck cannot find lib/common.sh when I run it

✔️ Awesome, I've got everything!

Great. Confirm you have saved bin/healthcheck and that it is executable.

ⓧ I'd like to double check the full code

Your bin/healthcheck should look exactly like this:

#!/usr/bin/env bash
# ops-toolkit healthcheck stub

set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=../lib/common.sh
source "${SCRIPT_DIR}/../lib/common.sh"

log INFO "Starting"
exit 0
Verify the end-to-end library workflow

Before you run the final check, predict what you expect to see. The command sources the library and calls log with an INFO level and a custom message. What format will the output be in?

  • Run the final verification from the project root:
source lib/common.sh && log INFO "test message"

You should see output like [2026-08-07 14:30:55] [INFO] test message printed to stderr. The timestamp, level, and message are all formatted by your library.

Why does this print to stderr?

Production scripts separate log output (stderr) from data output (stdout). This means you can pipe a script's real output to another command or file without log noise contaminating it.

Your shared library is complete. Next up, you will turn the healthcheck stub into a professional CLI with flags, environment selection, and a full system report.

Build a Professional CLI with getopts

Your shared library is ready, but the healthcheck stub is a dead end. It runs one log message and exits. Nobody on your team can control what it does without editing source code directly.

In this step, you'll add professional command-line flag parsing using getopts. When you're done, anyone can run bin/healthcheck -v -e prod to get a verbose health report for the production environment.

In this step, get ready to:
  • Add getopts flag parsing with -v, -d, -e, and -h options.
  • Wire flags into script behavior with verbose tracing and config loading.
  • Generate a formatted health report using mktemp and trap cleanup.
Add getopts parsing to bin/healthcheck

A script without flags is a script nobody else can use. getopts is a Bash builtin that parses short options (-v, -d, -e, -h) without any external dependencies.

  • Open bin/healthcheck in your text editor and replace the entire contents with this script header and global variables:
#!/usr/bin/env bash
# ops-toolkit healthcheck - system health report with professional CLI

set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=../lib/common.sh
source "${SCRIPT_DIR}/../lib/common.sh"

ENV=""
DRY_RUN=0
VERBOSE=0

What does this code do?

This replaces the old stub entirely. The three new globals (ENV, DRY_RUN, VERBOSE) are initialized with safe defaults. CLI parsing will set them later. They override the defaults from lib/common.sh so this script owns its own state.

  • Below VERBOSE=0, add the usage function that prints help text using a here document:
usage() {
    cat <<EOF
Usage: $(basename "$0") [OPTIONS]

Generate a system health report.

Options:
  -e ENV    Environment name (required: dev, staging, prod)
  -v        Verbose mode (enables debug tracing)
  -d        Dry-run mode (no file writes)
  -h        Show this help and exit

Examples:
  $(basename "$0") -e prod
  $(basename "$0") -v -d -e dev
EOF
    exit 0
}

What is a here document?

The cat <<EOF pattern prints multi-line text until it hits a line containing only EOF. Shell variables like $(basename "$0") expand inside, so the usage text always shows the correct script name.

The exit 0 ensures the script stops after printing help. No one wants a health check to run when they asked for -h.

  • Below usage(), add the parse_args function and a temporary main wrapper so the script is runnable:
parse_args() {
    while getopts ":e:vdh" opt; do
        case "$opt" in
            e) ENV="$OPTARG" ;;
            v) VERBOSE=1 ;;
            d) DRY_RUN=1 ;;
            h) usage ;;
            :) die "Option -${OPTARG} requires an argument." ;;
            \?) die "Unknown option: -${OPTARG}. Use -h for help." ;;
        esac
    done
    shift $((OPTIND - 1))

    if [[ -z "$ENV" ]]; then
        die "Environment (-e) is required. Use -h for help."
    fi
}

main() {
    parse_args "$@"
}

main "$@"

What does this code do?

The optstring ":e:vdh" uses a leading colon for silent-error mode. This suppresses Bash's default error messages so you handle them yourself in the : and \? cases.

The colon after e means -e requires an argument (stored in $OPTARG). Flags without colons (v, d, h) are boolean toggles.

After the loop, shift $((OPTIND - 1)) removes all processed options so any remaining positional arguments stay accessible via $@.

  • Save bin/healthcheck.
  • Test the help flag by running:
bin/healthcheck -h

You should see your formatted usage text with the options list and examples.

  • Test the environment flag by running:
bin/healthcheck -e dev

The script exits cleanly with no output. That's correct because main() only calls parse_args so far.

  • Test an invalid flag by running:
bin/healthcheck -z

You should see an error message: Unknown option: -z. Use -h for help. followed by a non-zero exit.

Seeing 'Permission denied'?

The script needs execute permission. Run chmod +x bin/healthcheck and try again. If you see a syntax error instead, check that the EOF closing the here document starts at column 1 with no leading spaces.

help me debug a permission denied or syntax error in my getopts script

Wire the parsed flags into script behavior

The flags parse correctly, but they don't do anything yet. Verbose mode should enable debug tracing, and the -e flag should load an environment-specific config file.

  • Create the dev environment config by running:
cat > config/dev.conf << 'EOF'
# ops-toolkit configuration for dev environment
ALERT_THRESHOLD=5
CHECK_INTERVAL=30
ALERT_WEBHOOK_URL="https://httpbin.org/post"
EOF
  • Create the prod environment config by running:
cat > config/prod.conf << 'EOF'
# ops-toolkit configuration for prod environment
ALERT_THRESHOLD=3
CHECK_INTERVAL=10
ALERT_WEBHOOK_URL="https://httpbin.org/post"
EOF
  • Confirm both config files exist by running:
ls config/

You should see dev.conf, healthcheck.conf, and prod.conf listed.

Why separate config files per environment?

Production environments need tighter thresholds and faster check intervals than dev. By storing settings in config/${ENV}.conf, you swap behavior with a flag (-e prod vs -e dev) instead of editing code.

  • Return to bin/healthcheck in your editor. Replace the temporary main() function and the main "$@" line at the bottom with this expanded version:
main() {
    parse_args "$@"

    if [[ "$VERBOSE" -eq 1 ]]; then
        export PS4='+(${BASH_SOURCE}:${LINENO}): '
        set -x
    fi

    local config_file="${OPS_TOOLKIT_DIR}/config/${ENV}.conf"
    if [[ -f "$config_file" ]]; then
        log INFO "Loading config: ${config_file}"
        # shellcheck source=/dev/null
        source "$config_file"
    fi

    validate_env hostname df free uptime
}

main "$@"

What does this code do?

When VERBOSE is 1, the script sets PS4 to show the source file and line number, then enables set -x trace mode. Every command the script runs prints its location before executing.

The config-loading block sources config/${ENV}.conf only if the file exists. This brings in environment-specific variables like ALERT_THRESHOLD without crashing if someone passes an environment that has no config yet.

Finally, validate_env ensures all the system commands the script needs (hostname, df, free, uptime) are available before doing any real work.

  • Save bin/healthcheck and test verbose mode with config loading by running:
bin/healthcheck -v -e dev

You should see debug trace lines prefixed with +(bin/healthcheck:LINE): and a log line reading Loading config: followed by the path to dev.conf.

Not seeing debug trace output?

Make sure the PS4 line uses single quotes (not double quotes). Double quotes would expand the variables at assignment time instead of at each traced line. Also check that set -x appears after the PS4 assignment.

help me debug why my verbose tracing isn't showing file and line numbers

Add the health check report body

The CLI is wired up and loading config. Now the script needs to actually do something useful. It will collect system stats into a temporary file, display them, and save a timestamped copy.

  • In bin/healthcheck, find the validate_env hostname df free uptime line inside main(). Replace everything from the closing } of main and the main "$@" call with this report generation body:
    local tmp_report
    tmp_report="$(mktemp "${OPS_LOGS_DIR}/report.XXXXXX")"
    trap 'rm -f "$tmp_report"' EXIT

    log INFO "Generating health report for environment: ${ENV}"

    {
        printf '=== SYSTEM HEALTH REPORT ===\n'
        printf 'Environment: %s\n' "$ENV"
        printf 'Hostname:    %s\n' "$(hostname)"
        printf 'Date:        %s\n' "$(date '+%Y-%m-%d %H:%M:%S')"
        printf 'Uptime:      %s\n' "$(uptime -p)"
        printf '\n--- Disk Usage ---\n'
        df -h /
        printf '\n--- Memory Usage ---\n'
        free -h
        printf '\n=== END REPORT ===\n'
    } > "$tmp_report"

    cat "$tmp_report"

    local final_report="${OPS_LOGS_DIR}/healthcheck_$(date '+%Y%m%d_%H%M%S').log"
    run_cmd cp "$tmp_report" "$final_report"

    if [[ "$DRY_RUN" -eq 0 ]]; then
        log INFO "Report saved to: ${final_report}"
    fi
}

main "$@"

What does this code do?

  • mktemp creates a temporary file with a random suffix in your logs/ directory. The XXXXXX is replaced by 6 random characters.
  • trap 'rm -f "$tmp_report"' EXIT guarantees the temp file is cleaned up when the script exits, even on errors.
  • The brace group { ... } > "$tmp_report" redirects all output from multiple commands into one file in a single write operation.
  • run_cmd cp uses your library's dry-run wrapper. When -d is passed, it logs what it would do instead of actually copying.
  • Save bin/healthcheck.

Before you run the full report, what do you expect to see when you pass -v -e prod? Think about which config file loads and whether you'll see trace output mixed in.

  • Run the full health check by running:
bin/healthcheck -v -e prod

You should see debug trace lines (from -v), a Loading config: log line pointing at prod.conf, and a formatted report showing your hostname, uptime, disk usage, and memory stats.

  • Test the help flag by running:
bin/healthcheck -h

You should see the full usage text with options and examples.

  • Test an unknown option by running:
bin/healthcheck -z

You should see the error Unknown option: -z. Use -h for help. and a non-zero exit code.

  • Test dry-run mode by running:
bin/healthcheck -d -e dev

You should see the report printed to the terminal, plus a DRY-RUN log line showing Would execute: cp instead of actually saving the log file.

Seeing 'Required command not found'?

The validate_env call checks for hostname, df, free, and uptime. On Linux Mint these are all available by default. If one is missing, install it via sudo apt install with the package name matching the command.

help me figure out why validate_env is failing in my healthcheck script

✔️ Awesome, I've got everything!

Great. Double check you've saved bin/healthcheck and that all four test runs above matched the expected output.

ⓧ I'd like to double check the full code

Here is the complete bin/healthcheck at this point. Scroll up and confirm each section is present in this order: header, globals, usage(), parse_args(), main(), and the main "$@" call.

  • Header and globals (shebang through VERBOSE=0).
  • The usage() function with the here document.
  • The parse_args() function with the getopts loop and ENV validation.
  • The main() function calling parse_args, setting up verbose/config, running validate_env, creating the temp report with mktemp and trap, writing stats, catting to terminal, and copying via run_cmd.
  • The final main "$@" call at the very bottom.

Your healthcheck script now has a professional interface that anyone can use. Next up, you'll run ShellCheck on this code and discover quoting bugs hiding beneath the surface.

Lint and Debug with ShellCheck

Your healthcheck script from Step 3 runs perfectly when you test it by hand. But "works on my machine" is not a production standard.

Hidden quoting bugs and word-splitting traps lurk in scripts that appear functional. A filename with a space, an empty variable, or a masked return value can break your script silently in production. ShellCheck catches these before your users do.

In this step, get ready to:
  • Run ShellCheck on your scripts and understand the warnings it catches.
  • Learn the full debugging workflow: syntax checking, static analysis, and runtime tracing.
  • Create a Git pre-commit hook that blocks unclean code from being committed.
Run ShellCheck on your scripts

ShellCheck is a static analysis tool for shell scripts. It reads your code without executing it and flags patterns that lead to bugs at runtime.

  • Run ShellCheck on both your scripts by running this command:
shellcheck bin/healthcheck lib/common.sh

Before you hit Enter, what do you expect to see? Given that you carefully quoted variables throughout Steps 2 and 3, do you think there will be warnings?

You should see no output at all. ShellCheck only prints something when it finds a problem, so silence means your code is clean.

Why is the code already clean?

The patterns you followed in Steps 2 and 3 are exactly what ShellCheck enforces. Separating local tmp_report from its assignment, quoting every variable expansion, and quoting command substitutions all prevent the three most common ShellCheck warnings.

Understanding what these warnings mean helps you recognise bad patterns in other people's scripts and in code you inherit.

Understand the warnings ShellCheck catches

Even though your code passes, knowing what ShellCheck looks for is essential for reading existing scripts and reviewing pull requests. Here are the three most common warnings.

What do the common ShellCheck codes mean?

  • SC2086 warns: "Double quote to prevent globbing and word splitting." It fires when you use $var instead of "$var". Without quotes, a variable containing a space splits into multiple arguments, and * expands to every filename in the directory.
  • SC2046 warns: "Quote this to prevent word splitting." It triggers on unquoted command substitutions like $(somecommand). The output undergoes word splitting just like a variable, so it needs double quotes.
  • SC2155 warns: "Declare and assign separately to avoid masking return values." It fires on local var=$(cmd) because local always returns 0, masking the exit code of the command. The fix is local var then var=$(cmd) on a separate line.
  • See how your code already applies the SC2155 fix by finding these lines in bin/healthcheck:
    local tmp_report
    tmp_report="$(mktemp "${OPS_LOGS_DIR}/report.XXXXXX")"

What does this pattern do?

Line one declares tmp_report as a local variable. Line two assigns the result of mktemp separately. If mktemp fails (disk full, permissions issue), the exit code propagates and set -e halts the script immediately.

A combined local tmp_report=$(mktemp ...) would have hidden that failure because local always succeeds and overwrites the exit code.

Learn the full debugging workflow

ShellCheck handles static analysis, but you have three more tools in your debugging toolkit. Together they cover syntax errors, logic errors, and runtime failures.

  • Check for pure syntax errors (missing fi, unmatched quotes) by running:
bash -n bin/healthcheck

No output means the syntax is valid. This parses the script without executing any commands.

  • Run ShellCheck for static analysis (you just did this):
shellcheck bin/healthcheck lib/common.sh
  • Use runtime tracing for logic bugs by running your script with the -v flag you wired in Step 3:
bin/healthcheck -v -e dev

You should see every line prefixed with +(bin/healthcheck:LINE): showing the exact file and line number being executed. This is the PS4 debug format you set up in Step 3.

What is the debugging workflow summary?

  • bash -n catches syntax errors (missing keywords, unmatched brackets) without running the script.
  • shellcheck catches semantic bugs (unquoted variables, masked return values) via static analysis.
  • PS4 + set -x (the -v flag) traces execution line by line with source file and line numbers.
  • The ERR trap with BASH_LINENO captures exactly which line caused a failure when set -e terminates the script.

Not seeing debug trace output?

Make sure you pass the -v flag when running the healthcheck. Without -v, the VERBOSE variable stays 0 and set -x is never activated.

help me debug why bin/healthcheck -v is not showing trace output

Create a ShellCheck pre-commit hook

Running ShellCheck manually is fine for you. But on a team, someone will forget. A Git pre-commit hook enforces the check automatically: no script can be committed unless it passes ShellCheck.

  • Create the first part of the pre-commit hook (file detection logic) by running:
cat > .git/hooks/pre-commit << 'HOOK'
#!/usr/bin/env bash
# ShellCheck pre-commit hook for ops-toolkit
set -euo pipefail

STAGED_FILES="$(git diff --cached --name-only --diff-filter=ACM)"
SHELL_FILES=""

for file in $STAGED_FILES; do
    case "$file" in
        *.sh|lib/*|bin/*)
            if [[ -f "$file" ]] && head -1 "$file" | grep -q 'bash'; then
                SHELL_FILES="${SHELL_FILES} ${file}"
            fi
            ;;
    esac
done

if [[ -z "$SHELL_FILES" ]]; then
    exit 0
fi
HOOK

What does this section do?

  • git diff --cached --name-only --diff-filter=ACM lists only files that are staged for commit (Added, Copied, or Modified).
  • The case block filters for shell scripts: anything in bin/, lib/, or with a .sh extension.
  • If no shell files are staged, the hook exits cleanly with code 0 and the commit proceeds normally.
  • Append the ShellCheck validation logic by running:
cat >> .git/hooks/pre-commit << 'HOOK'

log() {
    printf '[pre-commit] %s\n' "$*" >&2
}

FAILED=0
for file in $SHELL_FILES; do
    if ! shellcheck "$file"; then
        log "FAILED: ${file}"
        FAILED=1
    fi
done

if [[ "$FAILED" -eq 1 ]]; then
    log "Commit blocked. Fix ShellCheck warnings above."
    exit 1
fi

log "All shell scripts pass ShellCheck."
exit 0
HOOK

What does this section do?

  • Each shell file gets run through shellcheck. If any file fails, FAILED is set to 1.
  • When FAILED is 1, the hook exits with code 1. Git aborts the commit whenever a pre-commit hook exits non-zero.
  • If everything passes, a success message prints and the commit goes through.
  • Make the hook executable by running:
chmod +x .git/hooks/pre-commit
  • Verify the hook file exists and is executable by running:
ls -la .git/hooks/pre-commit

You should see -rwxr-xr-x permissions (the x means executable).

Now test the hook by staging a deliberately broken file.

  • Create a test script with an intentional unquoted variable by running:
cat > bin/bad-test.sh << 'EOF'
#!/usr/bin/env bash
echo $UNQUOTED_VAR
EOF
  • Stage the broken file by running:
git add bin/bad-test.sh

Before you run the commit, what do you predict will happen? The file has $UNQUOTED_VAR without double quotes.

  • Attempt the commit by running:
git commit -m "test: should be blocked"

The hook fires, ShellCheck finds the SC2086 warning, and the commit is blocked. You should see output ending with [pre-commit] Commit blocked. Fix ShellCheck warnings above. and the commit does not go through.

What just happened?

The pre-commit hook ran shellcheck on every staged shell script. Because bin/bad-test.sh contains an unquoted variable, ShellCheck printed the SC2086 warning and the hook returned exit code 1. Git aborted the commit.

  • Clean up the test file by running:
git reset HEAD bin/bad-test.sh
rm bin/bad-test.sh
  • Confirm your clean scripts still commit successfully by staging and committing them:
git add bin/healthcheck lib/common.sh
git commit -m "feat: ShellCheck-clean scripts with pre-commit hook"

You should see [pre-commit] All shell scripts pass ShellCheck. followed by the commit confirmation message.

Commit still blocked on your clean scripts?

If ShellCheck reports a warning on bin/healthcheck or lib/common.sh, read the SC code in the output and fix the quoted line it points to. The most common cause is a missing double quote around a variable expansion.

help me fix ShellCheck warnings blocking my commit in ops-toolkit

✔️ Awesome, I've got everything!

Great. Make sure your commit went through successfully and you see the success message from the pre-commit hook.

ⓧ I'd like to double check the full code

The only new file in this step is the pre-commit hook. Your bin/healthcheck and lib/common.sh are unchanged from Step 3. Scroll up and confirm your hook contains both chunks in order:

  • Chunk 1: the shebang, set -euo pipefail, staged file detection with the case block, and early exit 0 if no shell files are staged.
  • Chunk 2: the log() function, the shellcheck loop setting FAILED, and the pass/fail exit logic.

Verify it is executable with ls -la .git/hooks/pre-commit and confirm you see the x permission flag.

Your scripts are now ShellCheck-clean and protected by an automated gatekeeper. Next up, you will put your parallelism skills to work processing thousands of log lines faster than a sequential loop ever could.

Process Logs in Parallel

In production, your team processes gigabytes of log files daily. Sequential processing wastes CPU cores that could be working simultaneously.

In this step, you will use xargs -P to process logs across multiple cores at once. You will also hit a real race condition when multiple workers write to the same file, and fix it with per-worker temp files.

In this step, get ready to:
  • Create a log generator that produces 200,000 lines of sample data across 4 files.
  • Build a parallel log processor that demonstrates measurable speedup over sequential.
  • Hit a race condition from concurrent file writes and fix it with per-worker temp files.
Create the log generator script

Before you can test parallel processing, you need realistic log data. The generator creates multiple files with randomized entries so you can measure real timing differences.

  • Create a new executable file called bin/gen-logs by running:
touch bin/gen-logs
chmod +x bin/gen-logs
  • Open bin/gen-logs in your text editor and paste the script header, setup, and data arrays:
#!/usr/bin/env bash
# Generate sample log files for parallel processing exercise

set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=../lib/common.sh
source "${SCRIPT_DIR}/../lib/common.sh"

SAMPLE_DIR="${OPS_LOGS_DIR}/sample"
NUM_FILES=4
LINES_PER_FILE=50000

main() {
    mkdir -p "$SAMPLE_DIR"
    log INFO "Generating ${NUM_FILES} log files with ${LINES_PER_FILE} lines each..."

    local levels=("INFO" "INFO" "INFO" "INFO" "WARN" "ERROR")
    local messages=(
        "Request processed successfully"
        "User authenticated"
        "Cache hit for key"
        "Database query completed"
        "Connection pool running low"
        "Failed to connect to upstream"
    )

What does this code do?

The levels array is weighted toward INFO (4 out of 6 entries), so roughly 1 in 6 lines will be ERROR. That gives you a realistic distribution to count against.

The messages array provides variety so the generated logs look plausible rather than identical repeating lines.

  • Below the arrays (still inside main()), paste the generation loop and closing:
    local i
    for i in $(seq 1 "$NUM_FILES"); do
        local outfile="${SAMPLE_DIR}/app${i}.log"
        log INFO "Generating ${outfile}..."
        local line
        for line in $(seq 1 "$LINES_PER_FILE"); do
            local idx=$((RANDOM % ${#levels[@]}))
            printf '2026-08-07 %02d:%02d:%02d [%s] %s\n' \
                $((RANDOM % 24)) $((RANDOM % 60)) $((RANDOM % 60)) \
                "${levels[$idx]}" "${messages[$idx]}"
        done > "$outfile"
    done

    log INFO "Done. Files in ${SAMPLE_DIR}:"
    ls -lh "$SAMPLE_DIR"
}

main "$@"

What does this code do?

The nested loop generates 50,000 lines per file. Each line gets a random timestamp, a random log level picked from the weighted array, and a corresponding message.

Redirecting the entire inner loop with > "$outfile" is faster than appending line by line because it opens the file once for the whole block.

  • Save bin/gen-logs.
  • Run the generator to produce your test data:
bin/gen-logs

You should see log messages as each file is generated, then a listing of 4 files in logs/sample/ each roughly 2-4 MB in size. This takes 30-60 seconds because generating 200,000 random lines in pure Bash is CPU-intensive.

Permission denied when running bin/gen-logs?

Make sure you ran chmod +x bin/gen-logs earlier. If you still see errors, verify you are in the ~/ops-toolkit directory.

Help me debug this

  • Confirm the files exist by running:
ls -lh logs/sample/

You should see app1.log, app2.log, app3.log, and app4.log listed.

Build the log processor and hit the race condition

Now you will write a script that processes these logs two ways: first sequentially (one file at a time), then in parallel (4 workers at once). The first parallel attempt will use a shared output file, which introduces a race condition you need to see for yourself.

  • Create and make executable bin/logproc by running:
touch bin/logproc
chmod +x bin/logproc
  • Open bin/logproc and paste the header, setup, and count_errors function:
#!/usr/bin/env bash
# Parallel log processor - demonstrates xargs -P speedup and race condition fix

set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=../lib/common.sh
source "${SCRIPT_DIR}/../lib/common.sh"

SAMPLE_DIR="${OPS_LOGS_DIR}/sample"
REPORT_FILE="${OPS_LOGS_DIR}/error_report.txt"

# Function for counting errors in a single file
count_errors() {
    local file="$1"
    local output_file="$2"
    local count
    count="$(grep -c 'ERROR' "$file" || true)"
    printf '%s: %s errors\n' "$(basename "$file")" "$count" >> "$output_file"
}
export -f count_errors

What does this code do?

The count_errors() function takes a log file and an output file, counts ERROR lines with grep -c, and appends the result. The || true prevents grep from returning a non-zero exit code when no matches are found (which would kill the script under set -e).

The export -f makes the function visible to child bash processes spawned by xargs. Without it, parallel workers cannot call the function.

  • Below export -f count_errors, paste the main() function with sequential processing:
main() {
    validate_env find xargs grep

    if [[ ! -d "$SAMPLE_DIR" ]]; then
        die "Sample logs not found. Run bin/gen-logs first."
    fi

    log INFO "=== Sequential Processing ==="
    local seq_report="${OPS_LOGS_DIR}/seq_report.txt"
    : > "$seq_report"
    SECONDS=0
    local file
    for file in "${SAMPLE_DIR}"/*.log; do
        local count
        count="$(grep -c 'ERROR' "$file" || true)"
        printf '%s: %s errors\n' "$(basename "$file")" "$count" >> "$seq_report"
    done
    local seq_time=$SECONDS
    log INFO "Sequential time: ${seq_time}s"
    cat "$seq_report"
}

main "$@"

What does this code do?

The SECONDS built-in variable resets to 0 and then counts elapsed seconds automatically. After the loop finishes processing all 4 files one by one, $SECONDS holds the total wall-clock time.

The : > "$seq_report" idiom truncates the file to zero bytes before writing. This ensures a clean report even if you run the script multiple times.

  • Save bin/logproc and run the sequential-only version to verify it works:
bin/logproc

You should see each file's error count and a sequential timing value. This confirms the script runs correctly before you add parallelism.

  • Now open bin/logproc again. Find the closing } of main() and the main "$@" call. Replace them both with a naive parallel section that uses the shared REPORT_FILE:
    log INFO ""
    log INFO "=== Parallel Processing (xargs -P 4) ==="
    SECONDS=0
    : > "$REPORT_FILE"
    find "$SAMPLE_DIR" -name '*.log' -print0 | \
        xargs -0 -P 4 -I {} bash -c 'count_errors "$1" "$2"' _ {} "$REPORT_FILE"
    local par_time=$SECONDS
    log INFO "Parallel time: ${par_time}s"
    cat "$REPORT_FILE"
}

main "$@"

What does this code do?

This spawns 4 parallel workers (-P 4) that each call count_errors on a different log file. All 4 workers append their results to the same REPORT_FILE using >>.

The -print0 and -0 pair handles filenames with spaces safely by using null bytes as delimiters instead of newlines.

  • Save bin/logproc.

Before you run this, think about what happens when 4 processes all append to the same file simultaneously. Do you expect clean, complete output every time?

  • Run the log processor:
bin/logproc

You should see the sequential results (each file's error count), followed by the parallel results. With only 4 short writes the output may look correct this time. But POSIX does not guarantee that concurrent appends to a regular file are atomic.

Why is this a race condition?

When multiple processes append to the same file, the OS may interleave bytes from different writers at arbitrary boundaries. With 4 short lines you might get lucky today.

At production scale (hundreds of files, longer output per worker), lines interleave and corrupt your report. You cannot rely on luck for correctness. The safe pattern is: each worker writes to its own temp file, then you aggregate after all workers finish.

Fix the race condition with per-worker temp files

The fix gives each parallel worker its own exclusive output file. After all workers finish, a single aggregation step combines the results safely. No concurrent writes, no corruption.

  • In bin/logproc, find the naive parallel section that starts with:
    log INFO ""
    log INFO "=== Parallel Processing (xargs -P 4) ==="
    SECONDS=0
    : > "$REPORT_FILE"
    find "$SAMPLE_DIR" -name '*.log' -print0 | \
        xargs -0 -P 4 -I {} bash -c 'count_errors "$1" "$2"' _ {} "$REPORT_FILE"
    local par_time=$SECONDS
    log INFO "Parallel time: ${par_time}s"
    cat "$REPORT_FILE"
}

main "$@"
  • Replace that entire block with the fixed version using per-worker temp files and a timing comparison:
    log INFO ""
    log INFO "=== Parallel Processing (xargs -P 4) ==="
    local tmp_dir
    tmp_dir="$(mktemp -d "${OPS_LOGS_DIR}/parallel.XXXXXX")"
    trap 'rm -rf "$tmp_dir"' EXIT

    SECONDS=0
    find "$SAMPLE_DIR" -name '*.log' -print0 | \
        xargs -0 -P 4 -I {} bash -c '
            file="$1"
            tmp_dir="$2"
            count=$(grep -c "ERROR" "$file" || true)
            printf "%s: %s errors\n" "$(basename "$file")" "$count" \
                > "${tmp_dir}/$(basename "$file").result"
        ' _ {} "$tmp_dir"
    local par_time=$SECONDS

    cat "${tmp_dir}"/*.result > "$REPORT_FILE"
    log INFO "Parallel time: ${par_time}s"
    cat "$REPORT_FILE"

    log INFO ""
    log INFO "=== Timing Comparison ==="
    log INFO "Sequential: ${seq_time}s | Parallel: ${par_time}s"
    if [[ "$seq_time" -gt 0 && "$par_time" -gt 0 ]]; then
        log INFO "Speedup: roughly $(( seq_time / par_time ))x (with 4 workers)"
    fi
}

main "$@"

What changed?

Each worker now writes to its own file inside a tmp_dir (created with mktemp -d). The > redirect (not >>) is safe because each worker owns its file exclusively.

After xargs exits (all workers done), a single cat aggregates the results into REPORT_FILE. No concurrent writes, no race condition.

The trap 'rm -rf "$tmp_dir"' EXIT ensures the temp directory is cleaned up even if the script fails partway through.

  • Save bin/logproc.
  • Run the fixed version:
bin/logproc

You should see the sequential timing, then the parallel timing, followed by a comparison line showing the parallel approach is faster. The report now has clean, complete per-file error counts.

  • Verify both scripts pass shellcheck with zero warnings by running:
shellcheck bin/gen-logs bin/logproc

What do you expect to see? If your code matches the chunks above, you should see no output at all. That silence means zero warnings.

Seeing ShellCheck warnings?

If you see SC2086 about unquoted variables, make sure every variable reference in your parallel bash -c block is quoted. If you see SC2155, separate your local declarations from assignments (declare on one line, assign on the next).

Help me fix my ShellCheck warnings

✔️ Awesome, I've got everything!

Great. Double check you have saved both bin/gen-logs and bin/logproc.

ⓧ I'd like to double check the full code

Both files are over 30 lines, so confirm your code by scrolling up and checking each chunk is present in order.

For bin/gen-logs, confirm you have:

  • The header, strict mode, SCRIPT_DIR, source, and the SAMPLE_DIR/NUM_FILES/LINES_PER_FILE variables.
  • The main() opening with mkdir -p, the levels array, and the messages array.
  • The nested for loop generating lines, the ls -lh confirmation, the closing }, and main "$@".

For bin/logproc, confirm you have:

  • The header, strict mode, SCRIPT_DIR, source, SAMPLE_DIR, and REPORT_FILE variables.
  • The count_errors() function with export -f count_errors.
  • The main() sequential section using a for loop with SECONDS timing.
  • The fixed parallel section using mktemp -d, trap, per-worker .result files, cat aggregation, and the timing comparison.
  • The closing } and main "$@".

Your ops-toolkit now crunches logs in parallel. Next up, you will extend this toolkit with an API health monitor that uses curl and jq to poll endpoints with retry logic.

Secret mission

API Health Monitor with Retry Logic

Real DevOps automation talks to APIs. Can you build a health monitor that polls endpoints, retries failures with exponential backoff, and fires JSON alerts when things go down? This combines curl, jq, and retry logic into a miniature version of what production monitoring tools do internally.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • The ~/ops-toolkit project directory (scripts, config, logs, and Git history).
  • ShellCheck (installed via apt).
  • jq (installed via apt).

Keep everything running

No action needed. Choose this if you plan to keep extending the toolkit with new scripts in bin/.

  • Your ~/ops-toolkit directory stays as-is, ready to add new scripts.
  • ShellCheck and jq remain installed and available system-wide for any future projects.
  • The generated sample logs in logs/sample/ take up some disk space (around 40 MB). Delete them with rm -rf ~/ops-toolkit/logs/sample if you want to reclaim that space while keeping everything else.

Pause - I'll come back to this later

Archive the project to free up disk space while keeping a backup you can restore later.

  • Create a compressed archive by running:
tar -czf ~/Desktop/ops-toolkit-backup.tar.gz ~/ops-toolkit
  • Remove the project directory by running:
rm -rf ~/ops-toolkit
  • To restore later, extract the archive:
tar -xzf ~/Desktop/ops-toolkit-backup.tar.gz -C ~/
  • ShellCheck and jq stay installed. They are small packages and useful for other projects.

Delete - I don't want to use this again

Remove all project files and uninstall the tools you added during this project.

  • Delete the entire project directory by running:
rm -rf ~/ops-toolkit
  • Uninstall ShellCheck and jq by running:
sudo apt remove shellcheck jq
  • Confirm the project directory is gone by running:
ls ~/ops-toolkit
  • You should see a "No such file or directory" message confirming the removal.

Nice Work!

Nice Work!

Nice work. You just built a production-grade ops-toolkit that any DevOps team would recognize and commit to their shared repository.

You've learned how to:

  • Build a modular script architecture with a shared library (lib/common.sh) sourced across scripts, a professional CLI parsed with getopts, and a dry-run wrapper that makes every command safe to test.
  • Master the ShellCheck debugging workflow by fixing real static analysis warnings (SC2086, SC2046, SC2155) and wiring a Git pre-commit hook that blocks unclean code from ever being committed.
  • Build a parallel log processor with xargs -P that demonstrated measurable speedup over sequential processing, hit a real race condition from interleaved writes, and fixed it with per-worker temp files.
  • Secret Mission: Created an API health monitor using curl -w format strings, jq JSON parsing, and exponential backoff retry logic to poll endpoints and fire alerts when thresholds are breached.

Ready to quiz yourself?