Production Monitoring and Log Analysis

Deploy a containerized monitoring stack and build Bash ops automation scripts.

Introduction

30 Second Summary

When a website goes down at 2 AM, someone has to figure out what broke, why it broke, and how to fix it fast. That someone is a System Operations Engineer, and their secret weapon is a toolkit of monitoring dashboards, log analyzers, and automated health checks.

In this project, you will build a complete production operations toolkit on your local machine using Docker containers. You'll deploy a monitoring stack, write automation scripts in Bash, and investigate incidents the same way real ops teams do every day.

What You'll Build

A containerized monitoring environment with automated log analysis and health checking, mirroring the tools production support teams rely on.

By the end of this project, you'll have:

  • A Docker Compose stack running Nginx (generating logs), Prometheus (collecting metrics), and Grafana (visualizing dashboards).
  • A Bash log analyzer script that scans for errors, counts severity levels, and outputs structured reports.
  • A health check automation script scheduled via cron that monitors service availability.
  • Secret Mission: Build an incident response runbook script that simulates a production incident, walks through the investigation, and auto-generates a post-incident report.

Are there any prerequisites?

You only need basic command-line familiarity (cd, ls, mkdir). No prior Docker or Bash scripting experience is required.

Before We Start

Before building your operations toolkit, you need three things in place: Docker Desktop to run your containers, a project folder to organize your configs and scripts, and a code editor to write everything in.

This step gets your workstation ready. Once it's done, you'll have a clean project structure and all the tools you need to start deploying containers.

In this step, get ready to:
  • Verify Docker Desktop is installed and running.
  • Create the project directory structure.
  • Open the project in your code editor.
Verify Docker Desktop is installed and running

Docker Desktop is the application that lets you run containers on your Mac. Your entire monitoring stack will run inside Docker containers, so this needs to be working first.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal and press Enter to open it.
  • Check if Docker is installed by running this command:
docker --version

✔️ I see a version number

Docker is installed. Move on to the next check below.

ⓧ Command not found

Docker Desktop isn't installed yet. Follow these steps to install it:

  • Download Docker Desktop from the official Docker website.
  • Open the downloaded .dmg file and drag Docker to your Applications folder.
  • Open Docker Desktop from your Applications folder.
  • Wait for Docker Desktop to finish starting up. You'll see a whale icon appear in your menu bar.
  • Return to your terminal and run docker --version again to confirm.
  • Verify Docker Compose is available by running:
docker compose version

What is Docker Compose?

Docker Compose lets you define and run multiple containers together using a single configuration file. In this project, you'll use it to spin up Nginx, Prometheus, and Grafana all at once.

  • Make sure Docker Desktop is running. Look for the whale icon in your menu bar at the top of your screen.
  • If you don't see the whale icon, press Cmd+Space, type Docker, and press Enter to launch it.

Docker command not found after installing?

Close your terminal completely and open a new one. Docker Desktop adds its CLI tools to your path on launch, but existing terminal sessions won't pick up the change until reopened.

Also confirm Docker Desktop is fully started. The whale icon in your menu bar should be still, not animating.

Help me troubleshoot Docker on my Mac.

Create the project directory structure

Your toolkit needs separate folders for monitoring configs, automation scripts, log output, and web server configuration. Setting this structure up now keeps everything organized as the project grows.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and all subdirectories by running these commands:
mkdir -p ops-toolkit/{monitoring,scripts,logs,nginx}
cd ops-toolkit
  • Confirm the directories exist by running:
ls

You should see four folders listed: logs, monitoring, nginx, and scripts.

What are these folders for?

  • monitoring/ holds your Prometheus configuration file.
  • scripts/ holds your Bash automation scripts (log analyzer, health checks).
  • logs/ stores output from your scripts and copied container logs.
  • nginx/ holds your custom Nginx web server configuration.

Don't see the four folders?

Make sure you ran both lines. The first line creates the directories, and the second line moves you inside ops-toolkit. If ls shows nothing, you may still be in ~/Desktop instead of inside the project folder.

Run pwd to check your current location. It should end with /Desktop/ops-toolkit.

Help me fix my project directory setup.

Verify your code editor

You'll write all your configuration files and scripts in VS Code (or Cursor). Opening the project folder in your editor gives you a file explorer sidebar and an integrated terminal in one window.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Visual Studio Code (or Cursor) and press Enter to open it.
  • Click File in the top menu bar.
  • Click Open Folder.
  • Navigate to your Desktop, select the ops-toolkit folder, then click Open.

You should see ops-toolkit at the top of the Explorer sidebar on the left, with your four subfolders nested inside.

  • Open the integrated terminal by pressing Ctrl+` (backtick key, below Escape).
  • Confirm you're in the right folder by running:
pwd

The output should end with /Desktop/ops-toolkit. This terminal is where you'll run all your commands for the rest of the project.

Terminal opens in the wrong directory?

If pwd shows a different path, run cd ~/Desktop/ops-toolkit to move to the project folder manually.

Help me fix my VS Code terminal directory.

Your environment is ready. Next up, you'll deploy a full monitoring stack with Nginx, Prometheus, and Grafana running together in Docker containers.

Deploy the Application and Monitoring Stack

Your environment is set up and ready. Now it's time to deploy the application you'll be monitoring throughout this project.

In production, operations teams need to observe what's happening inside their applications at all times. You'll create a multi-container stack using Docker Compose that runs three services: Nginx (generating web traffic logs), Prometheus (collecting metrics), and Grafana (visualizing dashboards).

In this step, get ready to:
  • Create a Docker Compose file that defines all three services.
  • Configure Prometheus to scrape metrics.
  • Set up Nginx with structured logging and start the full stack.
Create the Docker Compose file

Docker Compose lets you define multiple containers in a single YAML file and start them all with one command. Instead of running three separate docker run commands, you declare what you need and Compose handles the rest.

  • In VS Code, right-click inside the Explorer sidebar (on the empty area below your folders).
  • Select New File.
  • Name the file docker-compose.yml and press Enter.
  • Paste the following configuration into the file:
services:
  nginx:
    image: nginx:1.30.2-alpine
    container_name: ops-nginx
    ports:
      - "8080:80"
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - nginx-logs:/var/log/nginx
    restart: unless-stopped

  prometheus:
    image: prom/prometheus:v3.12.0
    container_name: ops-prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
    restart: unless-stopped

  grafana:
    image: grafana/grafana-enterprise:latest
    container_name: ops-grafana
    ports:
      - "3000:3000"
    volumes:
      - grafana-storage:/var/lib/grafana
    restart: unless-stopped

volumes:
  nginx-logs:
  prometheus-data:
  grafana-storage:

What does this file do?

This file defines three services that will each run in their own container:

  • nginx runs a lightweight web server on port 8080. It mounts your custom config file and stores its logs in a named volume.
  • prometheus collects metrics from services every 15 seconds. It reads its config from your monitoring/ folder and persists data in its own volume.
  • grafana provides a browser-based dashboard for visualizing metrics on port 3000.

The volumes: section at the bottom declares named volumes so data persists even if you restart containers.

  • Save docker-compose.yml.

You won't see a change yet. This file needs the Prometheus and Nginx configs to exist before you can start the stack.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

services:
  nginx:
    image: nginx:1.30.2-alpine
    container_name: ops-nginx
    ports:
      - "8080:80"
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - nginx-logs:/var/log/nginx
    restart: unless-stopped

  prometheus:
    image: prom/prometheus:v3.12.0
    container_name: ops-prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
    restart: unless-stopped

  grafana:
    image: grafana/grafana-enterprise:latest
    container_name: ops-grafana
    ports:
      - "3000:3000"
    volumes:
      - grafana-storage:/var/lib/grafana
    restart: unless-stopped

volumes:
  nginx-logs:
  prometheus-data:
  grafana-storage:
Create the Prometheus configuration

Prometheus needs a configuration file that tells it which targets to scrape for metrics and how often to check them.

  • In VS Code, right-click on the monitoring folder in the Explorer sidebar.
  • Select New File and name it prometheus.yml.
  • Paste the following configuration:
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

What does this config do?

The scrape_interval: 15s tells Prometheus to pull metrics from its targets every 15 seconds. The single scrape job named prometheus points at Prometheus itself, so you can monitor the monitoring tool's own health.

  • Save monitoring/prometheus.yml.

You won't see a change yet. The next config file completes the set.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
Create a custom Nginx config with structured logging

The default Nginx logging format is basic. For production log analysis, you need structured logs that include the request method, status code, and response time in a consistent, parseable format.

  • Right-click on the nginx folder in the Explorer sidebar.
  • Select New File and name it nginx.conf.
  • Paste the following configuration:
user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format main '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent" '
                    'rt=$request_time';

    access_log /var/log/nginx/access.log main;

    sendfile        on;
    keepalive_timeout  65;

    server {
        listen       80;
        server_name  localhost;

        location / {
            root   /usr/share/nginx/html;
            index  index.html index.htm;
        }

        location /health {
            access_log off;
            return 200 'OK\n';
            add_header Content-Type text/plain;
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/share/nginx/html;
        }
    }
}

What does this config do?

The log_format main directive defines a custom log format that includes the client IP, request details, HTTP status code, and response time (rt=$request_time). This gives your log analyzer script structured data to parse later.

The /health endpoint returns a simple OK response without logging. Your health check script will use this endpoint to verify Nginx is alive without polluting the access logs.

  • Save nginx/nginx.conf.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format main '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent" '
                    'rt=$request_time';

    access_log /var/log/nginx/access.log main;

    sendfile        on;
    keepalive_timeout  65;

    server {
        listen       80;
        server_name  localhost;

        location / {
            root   /usr/share/nginx/html;
            index  index.html index.htm;
        }

        location /health {
            access_log off;
            return 200 'OK\n';
            add_header Content-Type text/plain;
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/share/nginx/html;
        }
    }
}

All three configuration files are in place. Time to bring the stack online.

  • Start all three containers in the background by running this command in your terminal:
docker compose up -d

Docker will pull the images for Nginx, Prometheus, and Grafana, create the containers, and start them. This may take a minute on the first run while images download.

  • Verify all three containers are running by running:
docker ps

You should see three containers listed: ops-nginx, ops-prometheus, and ops-grafana, all with a status of Up.

  • Open your browser and visit http://localhost:8080 to confirm Nginx is serving its default welcome page.

Containers not starting?

  • Make sure Docker Desktop is running. Check for the whale icon in your menu bar.
  • If you see a port conflict error, another application may already be using port 8080, 9090, or 3000. Stop that application and try again.
  • If you see a file mount error, double-check that nginx/nginx.conf and monitoring/prometheus.yml exist in the correct folders and are saved.

Still stuck? Why won't my Docker Compose stack start?

Your monitoring stack is now live. Next up, you'll write a Bash script that analyzes the logs Nginx generates.

Build the Log Analysis Script

Your monitoring stack is live and generating data. Now you need the tools to make sense of that data when something goes wrong at 2 AM.

In this step, you'll write a production-grade Bash script that parses Nginx access logs, counts requests by status code, identifies problem URLs, and outputs a structured report. This is the exact workflow ops teams use to investigate incidents.

In this step, get ready to:
  • Write a Bash log analyzer script with proper error handling and argument parsing.
  • Implement analysis logic that groups requests by status code and finds top error paths.
  • Test the script against live Nginx logs from your running container.
Create the log analyzer script skeleton

Production Bash scripts follow strict conventions that prevent silent failures. The set -euo pipefail pattern ensures the script exits on any error, unset variable, or failed pipe command.

  • Create a new file called log_analyzer.sh in your scripts/ folder.
  • Add the script header with strict mode, constants, and variable declarations:
#!/bin/bash
set -euo pipefail

# Log Analyzer - Production Operations Toolkit
# Analyzes Nginx access logs and generates structured reports

readonly SCRIPT_NAME="$(basename "$0")"
readonly TIMESTAMP="$(date +%Y%m%d_%H%M%S)"

# Colors for terminal output
readonly RED='\033[0;31m'
readonly GREEN='\033[0;32m'
readonly YELLOW='\033[1;33m'
readonly BLUE='\033[0;34m'
readonly NC='\033[0m'

LOG_FILE=""
OUTPUT_FILE=""
FILTER_STATUS=""

What does this code do?

  • set -euo pipefail enables strict mode. The script exits immediately on errors (-e), undefined variables (-u), and failed pipes (-o pipefail).
  • readonly makes constants immutable. The color codes let the report highlight errors in red and successes in green.
  • The three empty variables (LOG_FILE, OUTPUT_FILE, FILTER_STATUS) will be populated by command-line arguments.
  • Below the variable declarations, add the cleanup trap, usage function, and argument parsing logic:
cleanup() {
    # Cleanup temporary files if any
    :
}
trap cleanup EXIT

usage() {
    echo "Usage: ${SCRIPT_NAME} -f <log_file> [-l <status_filter>] [-o <output_file>]"
    echo ""
    echo "Options:"
    echo "  -f    Path to the Nginx access log file (required)"
    echo "  -l    Filter by HTTP status code prefix (e.g., 4 for 4xx, 5 for 5xx)"
    echo "  -o    Output report to file (default: stdout)"
    echo "  -h    Show this help message"
    exit 1
}

while getopts "f:l:o:h" opt; do
    case ${opt} in
        f) LOG_FILE="${OPTARG}" ;;
        l) FILTER_STATUS="${OPTARG}" ;;
        o) OUTPUT_FILE="${OPTARG}" ;;
        h) usage ;;
        *) usage ;;
    esac
done

if [[ -z "${LOG_FILE}" ]]; then
    echo -e "${RED}Error: Log file is required. Use -f <path>${NC}" >&2
    usage
fi

if [[ ! -f "${LOG_FILE}" ]]; then
    echo -e "${RED}Error: File '${LOG_FILE}' not found${NC}" >&2
    exit 1
fi

What does this code do?

  • trap cleanup EXIT ensures the cleanup() function runs when the script exits, regardless of how it exits. This prevents leftover temp files in production.
  • getopts is the standard Bash utility for parsing command-line flags. The string "f:l:o:h" means -f, -l, and -o each take an argument (the colon after the letter), while -h is a standalone flag.
  • The two if blocks validate that the user provided a log file and that the file actually exists on disk.
  • Save scripts/log_analyzer.sh.

Seeing a syntax error?

Make sure each ;; in the case block has two semicolons, not one. Also check that esac and done are on their own lines.

help me debug a syntax error in my getopts case statement

Implement the analysis logic

The analysis function uses standard Unix text-processing tools like grep, awk, and sort to extract meaningful patterns from raw log lines.

  • Below the file validation checks in log_analyzer.sh, add the analyze() function and the output logic:
analyze() {
    local report=""

    report+="$(echo -e "${BLUE}============================================${NC}")\n"
    report+="  Log Analysis Report\n"
    report+="  Generated: $(date)\n"
    report+="  File: ${LOG_FILE}\n"
    report+="$(echo -e "${BLUE}============================================${NC}")\n\n"

    # Total requests
    local total
    total=$(wc -l < "${LOG_FILE}" | tr -d ' ')
    report+="$(echo -e "${GREEN}Total Requests: ${total}${NC}")\n\n"

    # Status code breakdown
    report+="--- Status Code Breakdown ---\n"
    local status_2xx status_3xx status_4xx status_5xx
    status_2xx=$(grep -c '" 2[0-9][0-9] ' "${LOG_FILE}" 2>/dev/null || echo "0")
    status_3xx=$(grep -c '" 3[0-9][0-9] ' "${LOG_FILE}" 2>/dev/null || echo "0")
    status_4xx=$(grep -c '" 4[0-9][0-9] ' "${LOG_FILE}" 2>/dev/null || echo "0")
    status_5xx=$(grep -c '" 5[0-9][0-9] ' "${LOG_FILE}" 2>/dev/null || echo "0")

    report+="$(echo -e "  ${GREEN}2xx (Success):     ${status_2xx}${NC}")\n"
    report+="  3xx (Redirect):    ${status_3xx}\n"
    report+="$(echo -e "  ${YELLOW}4xx (Client Error): ${status_4xx}${NC}")\n"
    report+="$(echo -e "  ${RED}5xx (Server Error): ${status_5xx}${NC}")\n\n"

    # Top 5 error paths (4xx and 5xx)
    report+="--- Top 5 Error Paths (4xx/5xx) ---\n"
    report+="$(grep -E '" [45][0-9][0-9] ' "${LOG_FILE}" | awk '{print $7}' | sort | uniq -c | sort -rn | head -5)\n\n"

    # Top 5 client IPs
    report+="--- Top 5 Client IPs ---\n"
    report+="$(awk '{print $1}' "${LOG_FILE}" | sort | uniq -c | sort -rn | head -5)\n\n"

    # Apply status filter if specified
    if [[ -n "${FILTER_STATUS}" ]]; then
        report+="\n--- Filtered Results (Status ${FILTER_STATUS}xx) ---\n"
        local filtered_count
        filtered_count=$(grep -c "\" ${FILTER_STATUS}[0-9][0-9] " "${LOG_FILE}" 2>/dev/null || echo "0")
        report+="  Matching entries: ${filtered_count}\n"
    fi

    report+="$(echo -e "${BLUE}============================================${NC}")\n"
    report+="  Analysis complete.\n"

    echo -e "${report}"
}

# Run analysis
if [[ -n "${OUTPUT_FILE}" ]]; then
    analyze | tee "${OUTPUT_FILE}"
    echo -e "\n${GREEN}Report saved to: ${OUTPUT_FILE}${NC}"
else
    analyze
fi

What does this code do?

  • wc -l < "${LOG_FILE}" counts total lines in the log file. Each line is one HTTP request.
  • grep -c '" 2[0-9][0-9] ' counts lines matching HTTP 2xx status codes. The pattern matches the Nginx log format where the status code appears after a quote and space.
  • The top error paths pipeline (grep | awk | sort | uniq -c | sort -rn | head -5) filters for 4xx/5xx responses, extracts the URL path, and ranks them by frequency.
  • The || echo "0" pattern prevents grep from crashing the script when no matches are found. Without it, set -e would exit the script on zero matches.
  • Save scripts/log_analyzer.sh.

Script exits unexpectedly?

The most common cause is a missing || echo "0" after a grep -c call. When grep finds zero matches it returns exit code 1, which set -e treats as a fatal error.

help me fix my Bash script that exits early

✔️ Awesome, I've got everything!

Great. Double check you've saved scripts/log_analyzer.sh before moving on.

ⓧ I'd like to double check the full code

#!/bin/bash
set -euo pipefail

# Log Analyzer - Production Operations Toolkit
# Analyzes Nginx access logs and generates structured reports

readonly SCRIPT_NAME="$(basename "$0")"
readonly TIMESTAMP="$(date +%Y%m%d_%H%M%S)"

# Colors for terminal output
readonly RED='\033[0;31m'
readonly GREEN='\033[0;32m'
readonly YELLOW='\033[1;33m'
readonly BLUE='\033[0;34m'
readonly NC='\033[0m'

LOG_FILE=""
OUTPUT_FILE=""
FILTER_STATUS=""

cleanup() {
    # Cleanup temporary files if any
    :
}
trap cleanup EXIT

usage() {
    echo "Usage: ${SCRIPT_NAME} -f <log_file> [-l <status_filter>] [-o <output_file>]"
    echo ""
    echo "Options:"
    echo "  -f    Path to the Nginx access log file (required)"
    echo "  -l    Filter by HTTP status code prefix (e.g., 4 for 4xx, 5 for 5xx)"
    echo "  -o    Output report to file (default: stdout)"
    echo "  -h    Show this help message"
    exit 1
}

while getopts "f:l:o:h" opt; do
    case ${opt} in
        f) LOG_FILE="${OPTARG}" ;;
        l) FILTER_STATUS="${OPTARG}" ;;
        o) OUTPUT_FILE="${OPTARG}" ;;
        h) usage ;;
        *) usage ;;
    esac
done

if [[ -z "${LOG_FILE}" ]]; then
    echo -e "${RED}Error: Log file is required. Use -f <path>${NC}" >&2
    usage
fi

if [[ ! -f "${LOG_FILE}" ]]; then
    echo -e "${RED}Error: File '${LOG_FILE}' not found${NC}" >&2
    exit 1
fi

analyze() {
    local report=""

    report+="$(echo -e "${BLUE}============================================${NC}")\n"
    report+="  Log Analysis Report\n"
    report+="  Generated: $(date)\n"
    report+="  File: ${LOG_FILE}\n"
    report+="$(echo -e "${BLUE}============================================${NC}")\n\n"

    # Total requests
    local total
    total=$(wc -l < "${LOG_FILE}" | tr -d ' ')
    report+="$(echo -e "${GREEN}Total Requests: ${total}${NC}")\n\n"

    # Status code breakdown
    report+="--- Status Code Breakdown ---\n"
    local status_2xx status_3xx status_4xx status_5xx
    status_2xx=$(grep -c '" 2[0-9][0-9] ' "${LOG_FILE}" 2>/dev/null || echo "0")
    status_3xx=$(grep -c '" 3[0-9][0-9] ' "${LOG_FILE}" 2>/dev/null || echo "0")
    status_4xx=$(grep -c '" 4[0-9][0-9] ' "${LOG_FILE}" 2>/dev/null || echo "0")
    status_5xx=$(grep -c '" 5[0-9][0-9] ' "${LOG_FILE}" 2>/dev/null || echo "0")

    report+="$(echo -e "  ${GREEN}2xx (Success):     ${status_2xx}${NC}")\n"
    report+="  3xx (Redirect):    ${status_3xx}\n"
    report+="$(echo -e "  ${YELLOW}4xx (Client Error): ${status_4xx}${NC}")\n"
    report+="$(echo -e "  ${RED}5xx (Server Error): ${status_5xx}${NC}")\n\n"

    # Top 5 error paths (4xx and 5xx)
    report+="--- Top 5 Error Paths (4xx/5xx) ---\n"
    report+="$(grep -E '" [45][0-9][0-9] ' "${LOG_FILE}" | awk '{print $7}' | sort | uniq -c | sort -rn | head -5)\n\n"

    # Top 5 client IPs
    report+="--- Top 5 Client IPs ---\n"
    report+="$(awk '{print $1}' "${LOG_FILE}" | sort | uniq -c | sort -rn | head -5)\n\n"

    # Apply status filter if specified
    if [[ -n "${FILTER_STATUS}" ]]; then
        report+="\n--- Filtered Results (Status ${FILTER_STATUS}xx) ---\n"
        local filtered_count
        filtered_count=$(grep -c "\" ${FILTER_STATUS}[0-9][0-9] " "${LOG_FILE}" 2>/dev/null || echo "0")
        report+="  Matching entries: ${filtered_count}\n"
    fi

    report+="$(echo -e "${BLUE}============================================${NC}")\n"
    report+="  Analysis complete.\n"

    echo -e "${report}"
}

# Run analysis
if [[ -n "${OUTPUT_FILE}" ]]; then
    analyze | tee "${OUTPUT_FILE}"
    echo -e "\n${GREEN}Report saved to: ${OUTPUT_FILE}${NC}"
else
    analyze
fi
Test the script against live Nginx logs

Your script is complete. Now you need real log data to test it against. You'll generate traffic to Nginx (including some requests that trigger 404 errors), then pull the resulting logs out of the container.

  • Make the script executable by running this command:
chmod +x scripts/log_analyzer.sh
  • Generate a mix of successful and failing requests to Nginx by running these commands:
curl http://localhost:8080/
curl http://localhost:8080/
curl http://localhost:8080/nonexistent
curl http://localhost:8080/missing-page
curl http://localhost:8080/another-404
curl http://localhost:8080/

Why trigger 404 errors on purpose?

A log analyzer that only processes successful requests isn't very useful. By hitting URLs that don't exist (/nonexistent, /missing-page, /another-404), you create 4xx entries in the log so the script's error-detection logic has data to work with.

  • Copy the Nginx access log from inside the container to your local logs/ directory by running:
docker cp ops-nginx:/var/log/nginx/access.log ./logs/access.log

Why copy logs out of the container?

The Nginx logs live inside a Docker volume, which isn't directly accessible from your host machine. The docker cp command copies files between a container and your local filesystem. In production, you'd typically ship logs to a centralized system. For local testing, this copy step gives your script a file to analyze.

  • Run the log analyzer against your copied logs:
./scripts/log_analyzer.sh -f logs/access.log
  • You should see a structured report with a status code breakdown showing 2xx and 4xx counts, plus a list of the top error paths.

Permission denied when running the script?

Make sure you ran chmod +x scripts/log_analyzer.sh before executing the script. Without the executable permission, your shell won't run it.

If you see an error about the log file not being found, confirm you ran the docker cp command first and that logs/access.log exists.

help me fix permission or file-not-found errors with my log analyzer script

Your log analyzer is working. You can now parse real application logs and spot problems in seconds. Next, you'll automate health checks so your system monitors itself without manual intervention.

Automate Health Checks with Cron

In the previous step, you built a log analyzer that gives you a snapshot of what happened in the past. But in production, you need to know the moment something goes wrong.

In this step, you'll write a health check script that probes each service and logs whether it's alive or down. Then you'll schedule it with cron so it runs automatically every 5 minutes, just like a real ops team monitors production services.

In this step, get ready to:
  • Write a health check script that pings Nginx, Prometheus, and Grafana.
  • Schedule the script with cron to run every 5 minutes.
  • Simulate a failure and verify the script catches it.
Write the health check script

Your monitoring stack is running, but you have no automated way to detect when a service goes down. The health check script will use curl to hit each service's health endpoint and log the result with a timestamp.

  • Create the health check script by running:
touch scripts/health_check.sh
  • Open scripts/health_check.sh in your code editor.
  • Add the script header and configuration by pasting this code:
#!/bin/bash
set -uo pipefail

# Health Check Script - Production Operations Toolkit
# Checks service availability and logs results

readonly SCRIPT_NAME="$(basename "$0")"
readonly LOG_DIR="$(cd "$(dirname "$0")/../logs" && pwd)"
readonly LOG_FILE="${LOG_DIR}/health_check.log"
readonly TIMESTAMP="$(date '+%Y-%m-%d %H:%M:%S')"
readonly TIMEOUT=5

# Service definitions: name|url
declare -a SERVICES=(
    "nginx|http://localhost:8080/health"
    "prometheus|http://localhost:9090/-/healthy"
    "grafana|http://localhost:3000/api/health"
)

FAILED=0

What does this code do?

This header sets up the script's configuration and the services it will check.

  • set -uo pipefail enables strict mode. The u flag catches unset variables and pipefail propagates pipe errors. Notice there's no -e flag here because you don't want the script to exit on the first failed curl. It needs to check all services before reporting.
  • LOG_DIR resolves to the logs/ directory relative to the script, so the script works from any working directory.
  • The SERVICES array stores each service as a name|url pair. Each URL points to that service's official health endpoint.
  • FAILED is a counter that tracks how many services are down.
  • Below the configuration block, add the two functions that do the actual checking:
log_message() {
    local level="$1"
    local message="$2"
    echo "[${TIMESTAMP}] [${level}] ${message}" | tee -a "${LOG_FILE}"
}

check_service() {
    local name="$1"
    local url="$2"

    if curl --silent --fail --max-time "${TIMEOUT}" "${url}" > /dev/null 2>&1; then
        log_message "OK" "${name} is healthy (${url})"
    else
        log_message "ALERT" "${name} is DOWN or unreachable (${url})"
        FAILED=$((FAILED + 1))
    fi
}

What does this code do?

  • log_message() writes a timestamped entry to both the terminal and the log file using tee -a.
  • check_service() uses curl with --silent (no progress bar), --fail (return error on HTTP failures), and --max-time (5 second timeout) to probe a health endpoint.
  • If curl succeeds, the service is logged as OK. If it fails, the service is logged as ALERT and the FAILED counter increments.
  • Finally, add the main execution block that loops through all services and reports the result:
# Ensure log directory exists
mkdir -p "${LOG_DIR}"

log_message "INFO" "--- Health check started ---"

for service in "${SERVICES[@]}"; do
    IFS='|' read -r name url <<< "${service}"
    check_service "${name}" "${url}"
done

if [[ ${FAILED} -gt 0 ]]; then
    log_message "ALERT" "${FAILED} service(s) failed health check!"
    exit 1
else
    log_message "INFO" "All services healthy."
    exit 0
fi

What does this code do?

  • mkdir -p ensures the logs/ directory exists before writing to it.
  • The for loop splits each SERVICES entry on the | character to extract the name and URL, then passes them to check_service().
  • At the end, the script exits with code 1 if any service failed. This non-zero exit is what production alerting tools look for to trigger notifications.
  • Save scripts/health_check.sh.
  • Make the script executable by running:
chmod +x scripts/health_check.sh
  • Test the script by running it directly:
./scripts/health_check.sh

You should see output showing all three services as healthy, with timestamps and OK status for each.

Seeing permission denied or command not found?

  • Make sure you ran chmod +x scripts/health_check.sh to mark the file as executable.
  • Confirm you're in the ops-toolkit directory when running the script.
  • If a service shows as DOWN, verify all Docker containers are running with docker ps.

Help me troubleshoot my health check script.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

#!/bin/bash
set -uo pipefail

# Health Check Script - Production Operations Toolkit
# Checks service availability and logs results

readonly SCRIPT_NAME="$(basename "$0")"
readonly LOG_DIR="$(cd "$(dirname "$0")/../logs" && pwd)"
readonly LOG_FILE="${LOG_DIR}/health_check.log"
readonly TIMESTAMP="$(date '+%Y-%m-%d %H:%M:%S')"
readonly TIMEOUT=5

# Service definitions: name|url
declare -a SERVICES=(
    "nginx|http://localhost:8080/health"
    "prometheus|http://localhost:9090/-/healthy"
    "grafana|http://localhost:3000/api/health"
)

FAILED=0

log_message() {
    local level="$1"
    local message="$2"
    echo "[${TIMESTAMP}] [${level}] ${message}" | tee -a "${LOG_FILE}"
}

check_service() {
    local name="$1"
    local url="$2"

    if curl --silent --fail --max-time "${TIMEOUT}" "${url}" > /dev/null 2>&1; then
        log_message "OK" "${name} is healthy (${url})"
    else
        log_message "ALERT" "${name} is DOWN or unreachable (${url})"
        FAILED=$((FAILED + 1))
    fi
}

# Ensure log directory exists
mkdir -p "${LOG_DIR}"

log_message "INFO" "--- Health check started ---"

for service in "${SERVICES[@]}"; do
    IFS='|' read -r name url <<< "${service}"
    check_service "${name}" "${url}"
done

if [[ ${FAILED} -gt 0 ]]; then
    log_message "ALERT" "${FAILED} service(s) failed health check!"
    exit 1
else
    log_message "INFO" "All services healthy."
    exit 0
fi
Schedule with cron

Running the script manually works, but production environments need automated monitoring. Cron is a built-in macOS scheduler that runs commands on a recurring schedule.

  • First, get the full path to your script by running:
echo "$(cd scripts && pwd)/health_check.sh"

Copy the full path that's printed. You'll paste it into the cron entry in a moment.

  • Open the cron editor by running:
crontab -e

What is crontab -e?

This command opens your personal cron schedule in a text editor (usually nano on macOS). Cron reads this file and runs any commands at their scheduled times. Each line is one job.

  • If the editor opens as a blank file, that's normal. It means you have no existing cron jobs.
  • Add this line, replacing the path with the full path you copied earlier:
*/5 * * * * [[HEALTH_CHECK_PATH="full path to your health_check.sh (e.g. /Users/yourname/Desktop/ops-toolkit/scripts/health_check.sh)"]]

What does this cron expression mean?

  • */5 means every 5 minutes.
  • The five fields are: minute, hour, day-of-month, month, day-of-week. An asterisk (*) means "every" for that field.
  • The path after the time fields is the command that cron runs.
  • If you're in nano: save the file by pressing Ctrl+O, then press Enter to confirm. Exit with Ctrl+X.
  • If you're in vim: press Esc, type :wq, then press Enter.
  • Verify your cron entry was saved by running:
crontab -l

You should see your health check entry listed with the */5 * * * * schedule and the full path to your script.

Cron entry not showing up?

  • Make sure you saved the file in the editor before exiting. In nano, look for the "Wrote" confirmation at the bottom of the screen after pressing Ctrl+O.
  • If crontab -l shows "no crontab for user", run crontab -e again and add the line.

Help me set up my cron job for the health check script.

macOS may ask for permission

On recent macOS versions, cron jobs may require granting Full Disk Access to the cron daemon. If your scheduled health checks don't fire automatically, open System Settings > Privacy & Security > Full Disk Access and add /usr/sbin/cron.

Simulate a failure and verify alerting

The real test of a health check is whether it catches failures. You'll intentionally stop Nginx, run the health check, and confirm the script detects and logs the outage.

  • Stop the Nginx container by running:
docker stop ops-nginx
  • Run the health check manually to detect the failure:
./scripts/health_check.sh

You should see an ALERT entry for nginx being DOWN, while Prometheus and Grafana remain OK. The script also exits with a non-zero status.

  • Check the log file to confirm the failure was recorded:
cat logs/health_check.log

You should see timestamped entries including an ALERT line showing nginx is DOWN or unreachable.

  • Restart Nginx to restore the service:
docker start ops-nginx
  • Run the health check again to confirm recovery:
./scripts/health_check.sh

All three services should now show as OK. Your log file contains the full incident timeline: the failure, the ALERT, and the recovery.

  • View the full log history by running:
cat logs/health_check.log

You should see multiple entries showing the health check started, the nginx DOWN alert, and then healthy entries after the restart.

Nginx didn't come back as healthy?

  • After running docker start ops-nginx, wait 2-3 seconds before running the health check again. The container needs a moment to initialize.
  • Verify the container is running with docker ps and confirm ops-nginx is listed.

Help me troubleshoot why Nginx isn't recovering after restart.

Your health checks are automated and battle-tested. Next up, you'll explore the metrics flowing into Grafana and build a real-time monitoring dashboard.

Explore Metrics in Grafana

You now have a full monitoring stack running, health checks automated, and a log analyzer script ready. The final piece is connecting everything visually.

In this step, you will use Grafana to build a real-time dashboard showing the health of your services. This is how production support teams monitor systems at a glance without digging through terminal output.

In this step, get ready to:
  • Connect Prometheus as a data source in Grafana.
  • Create a monitoring dashboard with a PromQL query.
  • Verify that metrics flow through the full pipeline in real time.
Access Grafana and add Prometheus data source

Grafana is already running as part of your Docker Compose stack. You need to sign in and tell Grafana where to find your metrics by adding Prometheus as a data source.

  • Open your web browser and navigate to http://localhost:3000.
  • On the sign-in page, enter admin for both the username and password.
  • Click Sign in.

What about the password change prompt?

Grafana will prompt you to change the default password on first login. For a local development setup, you can click Skip or set a new password. Either way works for this project.

  • Click Connections in the left-side menu.
  • Click Add new connection.
  • Enter Prometheus in the search bar.
  • Click Prometheus data source.
  • Click Add new data source in the upper right.

Grafana opens the Settings tab for your new Prometheus data source.

  • In the Prometheus server URL field, enter http://prometheus:9090.
  • Scroll down and click Save & test.

Why prometheus instead of localhost?

Grafana and Prometheus run in separate Docker containers. Inside the Docker network, each container has its own localhost. Using http://prometheus:9090 refers to the Prometheus container by its Docker Compose service name, which Docker resolves automatically.

You should see a green success message confirming the data source is working.

Connection failed?

Make sure the URL is exactly http://prometheus:9090 with no trailing slash. If the test still fails, verify Prometheus is running by checking your terminal with docker ps. You should see the ops-prometheus container listed.

Help me debug my Grafana to Prometheus connection.

Create a monitoring dashboard

With the data source connected, you can now build a dashboard that visualizes whether your services are healthy. The up metric in Prometheus returns 1 for healthy targets and 0 for unreachable ones.

  • Click Dashboards in the main menu on the left.
  • Click New and select New Dashboard.
  • Click the Add new element icon (blue plus sign) and drag a panel onto the dashboard.
  • Select Configure visualization.

The panel editor opens with your Prometheus data source preselected.

  • In the query editor at the bottom, enter the PromQL query up.
  • Click Refresh to run the query.

You should see a time-series graph showing a value of 1 for the Prometheus target. This confirms Prometheus is reporting itself as healthy.

  • Under Panel options on the right side, enter Service Health as the panel title.
  • Click Save in the top right.
  • Enter Ops Toolkit Dashboard as the dashboard title.
  • Click Save.

What does the up metric show?

The up metric is automatically generated by Prometheus for every scrape target. A value of 1 means the target responded successfully to the last scrape. A value of 0 means it failed.

In production, ops teams use this exact metric to power alerting rules that fire when a service goes down.

Panel shows No data?

Make sure you typed up (lowercase) in the query field. Also confirm your data source is set to Prometheus in the Data source dropdown at the top of the query editor. If you still see nothing, click Refresh again to re-run the query.

Help me fix my Grafana panel that shows no data.

Verify the full pipeline

The final test brings everything together. You will generate fresh traffic to Nginx, re-run the log analyzer on the updated logs, and confirm the Grafana dashboard updates in real time.

  • Generate a batch of traffic to Nginx by running these commands:
# Generate a mix of successful and error requests
for i in {1..20}; do curl -s http://localhost:8080/ > /dev/null; done
for i in {1..5}; do curl -s http://localhost:8080/missing-page > /dev/null; done
for i in {1..3}; do curl -s http://localhost:8080/another-404 > /dev/null; done

What does this do?

These commands send 20 successful requests to the Nginx homepage and 8 requests to pages that do not exist. This generates a realistic mix of 200 and 404 status codes in the access log for the log analyzer to process.

  • Copy the fresh logs from the Nginx container to your local logs/ directory by running:
docker cp ops-nginx:/var/log/nginx/access.log logs/access.log
  • Re-run the log analyzer against the updated log file:
./scripts/log_analyzer.sh -f logs/access.log

You should see an updated report with higher request counts reflecting the traffic you just generated.

  • Switch back to your browser and open the Ops Toolkit Dashboard in Grafana.
  • Confirm the Service Health panel still shows a value of 1 (healthy) for the Prometheus target.

Why does the dashboard update automatically?

Prometheus scrapes its targets every 15 seconds (as configured in your prometheus.yml). Grafana polls Prometheus on a refresh interval, so the dashboard reflects the latest scraped data without any manual action from you.

Dashboard not updating?

Look for the refresh interval dropdown in the top-right corner of the Grafana dashboard (it may say Off). Set it to 5s or 10s to enable automatic polling. You can also click the refresh icon next to it to manually reload data.

Help me troubleshoot my Grafana dashboard refresh.

Your full monitoring pipeline is now operational. Nginx generates logs, Prometheus collects metrics, Grafana visualizes them, and your scripts automate analysis and health checks.

Secret mission

Build an Incident Response Script

Production incidents don't wait for you to figure out what to check. Real ops teams use runbook scripts that automatically capture system state, investigate the failure, and generate post-incident reports. Can you build one?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely on local Docker containers, so there are no ongoing cloud costs. The containers do consume about 1GB of RAM while running, so decide whether to keep them up, pause them, or delete everything.

Resources you used:

  • Docker containers (ops-nginx, ops-prometheus, ops-grafana) and their associated volumes.
  • Cron entry running health_check.sh every 5 minutes.
  • Local project files in the ops-toolkit folder on your Desktop.

Keep everything running

No action needed. Choose this if you're still actively building or experimenting with the monitoring stack.

  • Your containers will keep running and consuming memory until you manually stop them or restart your Mac.
  • The cron job will continue running the health check every 5 minutes in the background.
  • All scripts, configs, and logs remain in ~/Desktop/ops-toolkit.

Pause - I'll come back to this later

Stop the running containers to free up memory, but keep all files and volumes so you can resume later.

  • Stop the Docker Compose stack by running:
cd ~/Desktop/ops-toolkit
docker compose stop
  • When you're ready to resume, start the containers again by running:
docker compose start
  • The cron job will still run in the background, but the health checks will log alerts since the services are stopped. This is harmless.

Delete - I don't want to use this again

Remove all project containers, volumes, cron entries, and local files to start completely fresh.

  • Remove all Docker containers, networks, and volumes by running:
cd ~/Desktop/ops-toolkit
docker compose down -v
  • Remove the health check cron entry by running:
crontab -e
  • In the editor, find the line containing health_check.sh and delete it. Save and exit the editor.
  • Verify the cron entry is gone by running:
crontab -l
  • Delete the entire project folder by running:
rm -rf ~/Desktop/ops-toolkit
  • Your system is now back to its pre-project state. Docker Desktop itself remains installed for future use.

Nice Work!

Nice Work!

You've built a complete production operations toolkit from scratch! This project covered the exact skills production support teams use daily.

You've learned how to:

  • Deployed a multi-container monitoring stack with Docker Compose (Nginx + Prometheus + Grafana) and built a real-time metrics dashboard.
  • Written a production-grade Bash log analyzer with proper error handling, argument parsing, and structured report output.
  • Automated health checks with cron scheduling and failure detection that alerts when services go down.
  • Secret Mission: Built an incident response script that simulates production failures, investigates system state, and generates structured post-incident reports.

You can now talk confidently about container orchestration, monitoring, log analysis, and operational automation in interviews.

Ready to quiz yourself?