SRE Health Check Automation with Python

Introduction

30 Second Summary

Every online service eventually goes down. SRE teams handle this by building automated health checks that detect problems and fix them before users ever notice.

In this project, you will build a Python command-line tool that monitors Docker containers via HTTP health checks. Your tool will retry failing connections using exponential backoff and automatically restart any container that stays down.

What You'll Build

Stop a running container, run your health check tool, and watch it detect the failure, retry three times with increasing delays, restart the container on its own, and confirm recovery.

By the end of this project, you'll have:

• A working health check tool that scans your services and prints a timestamped status report showing which are healthy and which are down.
• Retry logic and auto-remediation that detects a downed container, retries with increasing wait times, and restarts it using the Docker SDK.
• A YAML configuration file that lets you add new services to monitor without changing any code.
• Secret Mission: Add a continuous monitoring loop with cooldown logic so your tool runs like a real background watchdog.

Are there any prerequisites?

This project assumes you're comfortable with basic Python concepts like variables, functions, loops, and imports. You'll also need Docker Desktop installed and running on your machine.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

Set Up Your Project and Target Services

Before you can build a tool that monitors services, you need services to monitor. You also need the Python packages that let your tool talk to those services over the network and manage their containers.

In this step, you'll create a Python project, install the libraries your tool depends on, and spin up two Docker containers as the live services it will watch.

In this step, get ready to:

• Create a project folder with a Python virtual environment and install packages.
• Start two nginx containers as target services.
• Verify everything is running and your packages are importable.

Create the project folder and install packages

A Python virtual environment keeps the packages for this project isolated from the rest of your system. The specific versions you install here won't collide with anything else on your machine.

• In VS Code, click Terminal in the top menu bar.
• Click New Terminal.
• Move to your Desktop and create the project folder by running these commands:

cd ~/Desktop
mkdir sre-health-checker
cd sre-health-checker

Your terminal prompt now shows sre-health-checker as the current directory.

• Create a Python virtual environment and activate it by running these commands:

macOS

python3 -m venv venv
source venv/bin/activate

Linux users: these same commands work on Linux.

Windows

python -m venv venv
venv\Scripts\activate

Your terminal prompt now starts with (venv), confirming the virtual environment is active.

Don't see (venv) in your prompt?

On Windows, PowerShell may block the activation script with an execution policy error. Run Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser and then try activating again.

Make sure you ran the activation command from inside the sre-health-checker folder. The venv folder must exist there before you can activate it.

Help me activate my Python virtual environment

• Install the three packages your project needs by running:

pip install requests==2.34.2 docker==7.2.0 PyYAML==6.0.3

What are these packages?

• requests sends HTTP requests to your services. Your tool uses it to check whether each service responds.
• docker is the official Docker SDK for Python. It lets your script restart containers programmatically instead of shelling out to the Docker CLI.
• PyYAML reads YAML configuration files. YAML is the standard config format in SRE and DevOps tooling.

You should see Successfully installed followed by the three package names near the end of the output.

Seeing a pip error?

If you see a "command not found" error for pip, your virtual environment may not be active. Check that your prompt starts with (venv) and try again.

If you see network or timeout errors, check your internet connection and re-run the install command.

Help me fix my pip install error

Start two nginx containers as target services

Your health check tool needs live services to monitor. You'll start two nginx containers. One will stay healthy for the entire project. The other, web-flaky, is the one you'll intentionally break later to test your tool's response.

• Start the first container by running:

docker run -d --name web-healthy -p 8081:80 nginx

The command prints a long container ID. That confirms the container started successfully.

• Start the second container by running:

docker run -d --name web-flaky -p 8082:80 nginx

What do these flags mean?

• -d runs the container in the background (detached mode) so your terminal stays free.
• --name web-healthy gives the container a human-readable name your script can reference later.
• -p 8081:80 maps port 8081 on your machine to port 80 inside the container. That's how you reach the service at http://localhost:8081.
• nginx is the official nginx image from Docker Hub. It starts in under a second and responds to HTTP requests immediately.

Seeing a port conflict or Docker error?

If you see "port is already allocated", another process is using that port. Run docker rm -f web-healthy web-flaky to remove leftover containers and try again.

If you see "Cannot connect to the Docker daemon", make sure Docker Desktop is running. Open it and wait for the engine to start before re-running the commands.

Help me troubleshoot my Docker container startup

Verify your containers and packages are ready

Before you run the next command, take a guess: what columns and rows do you expect to see when you list your running Docker containers?

• Confirm both containers are running by running:

docker ps

You should see two rows: one for web-healthy and one for web-flaky. Both should show a status starting with Up.

• Open your browser and go to http://localhost:8081.
• Open a second tab and go to http://localhost:8082.

Both pages should display the nginx default welcome page. This confirms your containers are serving HTTP responses, which is exactly what your health check tool will look for.

One last check. Do you think all three packages you installed will import successfully, or will something trip up?

• Confirm all three packages are importable by running:

python -c "import requests, docker, yaml; print('Ready')"

You should see Ready printed. This proves Python can find all three packages inside your virtual environment.

Seeing a ModuleNotFoundError?

Check that your prompt still shows (venv). If it doesn't, your virtual environment was deactivated. Reactivate it and re-run the pip install command.

If the error mentions yaml specifically, make sure you installed PyYAML (capital P, capital Y). The import name is yaml but the package name on PyPI is PyYAML.

Help me fix my Python import error

Your environment is ready with two live services running and all the Python packages installed. Next up, you'll write a script that checks whether those services are healthy.

Write a Basic Health Checker

Your environment is set up and both target services are running. Now you need a way to find out whether those services are actually up.

The simplest health check is an HTTP GET request. If a service responds with status code 200, it's healthy. In this step, you'll write a script that does exactly that for every service listed in a config file.

In this step, get ready to:

• Create a YAML config file defining the services to monitor.
• Write a Python script that checks each service's health via HTTP.
• Discover the script's fatal flaw when a service goes down.

Create the YAML config file

SRE tools read their settings from config files instead of hardcoding values into scripts. YAML is the standard format in DevOps tooling, used by Kubernetes, Ansible, and Prometheus.

• Create a new file called config.yaml in your project folder and add this configuration:

check_interval: 30
cooldown_period: 120
services:
  - name: web-healthy
    url: http://localhost:8081
    container_name: web-healthy
    max_retries: 3
  - name: web-flaky
    url: http://localhost:8082
    container_name: web-flaky
    max_retries: 3

What does this config do?

This file defines two services for the health checker to monitor. Each entry has a name for display, a url to send requests to, a container_name to target for restarts later, and max_retries for how many times to retry before giving up.

The check_interval and cooldown_period fields are used later in the Secret Mission. Having them here now means you can extend the tool without editing the config structure.

• Save config.yaml.
• Confirm the file exists by running:

ls

• You should see config.yaml listed in the output.

Don't see config.yaml?

Make sure you created the file inside your project folder, not in a parent directory. Run pwd to check which folder your terminal is in.

Check that the file name is exactly config.yaml with no extra spaces or typos.

Help me find my config.yaml file

Write the health check script

The script reads the config file, loops through each service, and sends an HTTP GET request to its URL. If the response comes back with status code 200, the service is alive.

• Create a new file called checker.py in your project folder and add the imports and config loader:

import requests
import yaml


def load_config(path):
    # Open and parse the YAML config file
    with open(path, "r") as f:
        return yaml.safe_load(f)

What does this code do?

• requests is the library that sends HTTP requests to your services.
• yaml.safe_load() parses the YAML config file into a Python dictionary without executing any embedded code.
• load_config() takes a file path, opens it, and returns the parsed config as a dictionary.

• Save checker.py.
• Test that the config loader works by running:

python -c "from checker import load_config; print(load_config('config.yaml'))"

• You should see a Python dictionary printed with your two service entries, their URLs, and container names.

Seeing a ModuleNotFoundError?

Make sure your virtual environment is activated. You should see its name in parentheses at the start of your terminal prompt.

If the environment is active but the error persists, check that you installed the packages inside the virtual environment, not outside of it.

Help me fix my import error

• Add the main function below load_config in checker.py:

def main():
    config = load_config("config.yaml")
    # Loop through each service and check its health
    for service in config["services"]:
        name = service["name"]
        url = service["url"]
        response = requests.get(url, timeout=5)
        print(f"{name}: HTTP {response.status_code}")


if __name__ == "__main__":
    main()

What does this code do?

• main() loads the config and loops through each service entry.
• requests.get(url, timeout=5) sends an HTTP GET request to the service URL. If the service doesn't respond within 5 seconds, the request fails.
• The script prints each service's name and HTTP status code so you can see at a glance which ones are up.
• if __name__ == "__main__": makes the script runnable from the command line.

• Save checker.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved checker.py before moving on.

ⓧ I'd like to double check the full code

Here's the complete checker.py so far:

import requests
import yaml


def load_config(path):
    # Open and parse the YAML config file
    with open(path, "r") as f:
        return yaml.safe_load(f)


def main():
    config = load_config("config.yaml")
    # Loop through each service and check its health
    for service in config["services"]:
        name = service["name"]
        url = service["url"]
        response = requests.get(url, timeout=5)
        print(f"{name}: HTTP {response.status_code}")


if __name__ == "__main__":
    main()

• Run the health checker by running:

python checker.py

You should see two lines printed, one for each service, both showing HTTP 200.

Seeing an error when you run the script?

If you see a FileNotFoundError, make sure config.yaml is in the same folder as checker.py.

If you see a connection error, confirm both containers are running with docker ps. You should see web-healthy and web-flaky listed.

Help me debug my health checker script

Break a service and watch the script crash

Both services are healthy right now, so the checker works fine. But in production, services fail all the time. What happens when your checker hits a downed service?

• Simulate a service failure by stopping the web-flaky container:

docker stop web-flaky

Before you run the checker again, think about what you expect to happen. The script sends an HTTP request to a service that no longer exists. Will it handle that gracefully, or will something go wrong?

• Run the health checker again:

python checker.py

The script reports web-healthy as HTTP 200, then crashes with a ConnectionError when it tries to reach web-flaky. The entire program dies on the spot.

Why does this crash matter?

A health checker that crashes when it finds an unhealthy service is useless. The whole point is to detect failures and report them. This script can't even survive the thing it's supposed to find.

This is the gap that retry logic with exponential backoff fixes in the next step.

• Restart web-flaky so it's ready for the next step:

docker start web-flaky

• Confirm it's back by running:

docker ps

• You should see both web-healthy and web-flaky listed with a status of Up.

You've proven the basic checker works, and you've seen exactly how it falls apart. Next up, you'll add retry logic with exponential backoff so the tool handles failures gracefully instead of crashing.

Add Retry Logic with Exponential Backoff

In the last step, you saw your health checker crash the moment web-flaky went down. That ConnectionError is exactly what happens in production when a naive health check hits a transient network blip.

The SRE-standard fix is retry logic with exponential backoff: wait 1 second, then 2, then 4, giving the service time to recover before declaring it truly down. In this step, you will wrap your health check in a retry loop so the script handles failures gracefully instead of crashing.

In this step, get ready to:

• Add structured logging so every check, retry, and failure includes a timestamp and service name.
• Build a retry loop with exponential backoff that catches connection errors instead of crashing.
• Test the retry pattern by stopping a container and watching the tool retry gracefully.

Set up structured logging

Right now checker.py uses bare print() calls. Production tools use Python's built-in logging module instead, because it automatically adds timestamps, severity levels, and consistent formatting to every message.

• In VS Code, open checker.py and replace everything above the load_config function with this code:

import time
import logging

import requests
import yaml


# Configure structured logging with timestamps
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(message)s",
    datefmt="%Y-%m-%d %H:%M:%S",
)
logger = logging.getLogger("healthchecker")

What does this code do?

The new imports bring in time (for sleep delays between retries) and logging (for structured output).

logging.basicConfig sets the output format so every log line includes a timestamp, a severity level like INFO or WARNING, and the message itself. logger is a named logger your functions will call instead of print().

• Save checker.py and run it to confirm nothing is broken:

python checker.py

You should still see the old print() output showing both services with their HTTP status codes. The new imports and logging config do not change behavior yet.

Seeing an ImportError?

Make sure your virtual environment is activated. If you see an error about requests or yaml not being found, your terminal session may have lost the virtual environment. Reactivate it and try again.

help me fix an ImportError in my Python health checker script

Build the check_health retry function

The core of retry logic is a loop that catches connection errors, waits an increasing amount of time, and tries again. Your check_health function will replace the bare requests.get() call in main() with this pattern.

• In checker.py, add this function below load_config and above main. This is the first half of check_health, covering the HTTP request and success path:

def check_health(service):
    # Read the service name, URL, and retry limit
    name = service["name"]
    url = service["url"]
    max_retries = service.get("max_retries", 3)

    for attempt in range(1, max_retries + 1):
        try:
            response = requests.get(url, timeout=5)
            if response.status_code == 200:
                logger.info("%s is healthy (HTTP %d)", name, response.status_code)
                return {"name": name, "status": "healthy", "attempts": attempt}
            else:
                logger.warning(
                    "%s returned HTTP %d (attempt %d/%d)",
                    name,
                    response.status_code,
                    attempt,
                    max_retries,
                )

What does this code do?

• max_retries reads the retry limit from config.yaml (defaulting to 3 if not set).
• The for loop runs up to max_retries times, trying the HTTP request each time.
• If the response is HTTP 200, it logs success and returns immediately with a "healthy" status.
• If the server responds with a non-200 code, it logs a warning with the attempt count and continues to the next attempt.

• Now add the rest of check_health directly below the code you just pasted. This handles connection errors, the backoff wait, and the final failure:

        except requests.exceptions.RequestException as e:
            logger.warning(
                "%s is unreachable (attempt %d/%d): %s",
                name,
                attempt,
                max_retries,
                type(e).__name__,
            )

        # Wait with exponential backoff before the next retry
        if attempt < max_retries:
            wait_time = 2 ** (attempt - 1)
            logger.info("Retrying %s in %d seconds...", name, wait_time)
            time.sleep(wait_time)

    logger.error("%s is unhealthy after %d attempts", name, max_retries)
    return {"name": name, "status": "unhealthy", "attempts": max_retries}

What does this code do?

• except requests.exceptions.RequestException catches any connection failure. This is the base class for all requests exceptions, so it handles timeouts, refused connections, and DNS errors.
• wait_time = 2 ** (attempt - 1) is the exponential backoff formula. On attempt 1 it waits 1 second, attempt 2 waits 2 seconds, attempt 3 waits 4 seconds.
• After all retries are exhausted, it logs the service as unhealthy and returns a result dict with "unhealthy" status.

Why exponential backoff?

If a service goes down for a few seconds (a restart, a network hiccup), hammering it with rapid retries makes things worse. Exponential backoff spaces out each retry so the service has time to recover.

The pattern (1s, 2s, 4s, 8s...) is standard across SRE tooling, cloud provider SDKs, and most API clients.

Now you need to update main() to call your new function instead of the old inline requests.get() logic.

• In checker.py, scroll down to main() and find this code:

def main():
    config = load_config("config.yaml")
    for service in config["services"]:
        name = service["name"]
        url = service["url"]
        response = requests.get(url, timeout=5)
        print(f"{name}: HTTP {response.status_code}")

• Replace the entire main() function with this simplified version that calls check_health:

def main():
    config = load_config("config.yaml")
    for service in config["services"]:
        check_health(service)

• Save checker.py.
• Make sure web-flaky is running by executing:

docker start web-flaky

• Run the health checker:

python checker.py

You should see timestamped log output with both services reported as healthy. The format now includes the date, time, log level, and message.

Seeing a NameError or IndentationError?

• A NameError: name 'logger' is not defined means the logging configuration block (from the first chunk) is missing or placed after the function. Make sure the logging.basicConfig and logger lines are at the top of the file, before any function definitions.
• An IndentationError usually means the except block is not aligned with the try block. Make sure the except keyword is at the same indentation level as try.

help me fix an error in my check_health function

✔️ Awesome, I've got everything!

Great. Double check you have saved checker.py and both services show as healthy in the log output.

ⓧ I'd like to double check the full code

Your checker.py has four sections. Scroll through the file and confirm each one is present in this order:

• Imports and logging config: import time, import logging, import requests, import yaml, then logging.basicConfig(...) and logger = logging.getLogger("healthchecker").
• load_config function (unchanged from the previous step).
• check_health function with the try/except block, exponential backoff wait, and final unhealthy return.
• main function that loads config and calls check_health(service) for each service, followed by the if __name__ == "__main__" guard.

Test the retry pattern with a downed container

The real test is what happens when a service goes down. In the last step, stopping web-flaky crashed the script. Now your retry logic should handle it gracefully.

• Stop the web-flaky container by running:

docker stop web-flaky

Before you run the health checker again, think about what you expect to see. Will the script crash like before, or will something different happen?

• Run the health checker:

python checker.py

No crash this time. Instead, you should see the tool detect that web-flaky is unreachable, retry 3 times with increasing wait times (1 second, then 2 seconds, then 4 seconds), and finally report the service as unhealthy.

What should I see?

Your log output should show web-healthy reported as healthy on the first attempt. Then for web-flaky, you will see three WARNING lines (one per failed attempt) with ConnectionError as the exception type, separated by INFO lines showing the retry waits.

The final ERROR line reads something like: web-flaky is unhealthy after 3 attempts. The total wait is about 7 seconds (1 + 2 + 4).

• Now restart web-flaky and confirm the tool reports it as healthy again:

docker start web-flaky

• Run the health checker one more time:

python checker.py

Both services should now show as healthy again in the log output.

Script still crashing with a ConnectionError?

• Make sure main() calls check_health(service) instead of the old requests.get() call. If the old inline code is still there, the retry logic never runs.
• Check that the except line catches requests.exceptions.RequestException (not requests.ConnectionError or a bare Exception). The base class RequestException covers all connection-related errors.

help me debug why my health checker still crashes when a service is down

Your health checker can now survive a downed service without crashing. Next up, you will take it one step further by adding auto-remediation. Instead of just reporting that a container is unhealthy, the tool will restart it automatically using the Docker SDK.

Add Auto-Remediation with Docker SDK

Your health check tool can now detect problems without crashing. It retries when services are unreachable and gracefully reports their status. But detecting a problem is only half the job.

In SRE, the real goal is auto-remediation. If a service is confirmed down after all retries, restart it automatically. This is the difference between getting paged at 3am and sleeping through while your tool fixes the issue.

In this step, get ready to:

• Add a restart_container function using the Docker SDK.
• Wire auto-remediation into the health check flow so unhealthy containers get restarted automatically.
• Add a summary report that shows each service's final status.

Import docker and add the restart_container function

The docker package provides a Python API for managing containers. Instead of shelling out to the docker CLI with subprocess calls, you call container.restart() directly.

Why the Docker SDK instead of subprocess?

You could shell out to the Docker CLI with subprocess.run(["docker", "restart", ...]), but that's fragile. You'd need to parse text output, handle exit codes, and deal with different CLI versions.

The Docker SDK gives you proper Python objects, typed exceptions like docker.errors.NotFound, and return values you can use directly. Using official SDKs instead of shelling out is the professional pattern.

• In checker.py, find the import block at the top of the file:

import time
import logging

import requests
import yaml

• Add import sys after import logging, and add import docker after import requests, so the block looks like this:

import time
import logging
import sys

import requests
import docker
import yaml

Now you need a function that takes a container name, finds the container through the Docker SDK, and restarts it.

• Add the following restart_container function below the check_health function in checker.py:

def restart_container(container_name):
    client = docker.from_env()
    try:
        # Look up the container by name
        container = client.containers.get(container_name)
        logger.info("Restarting container: %s", container_name)
        # Restart with a 10-second timeout before force-killing
        container.restart(timeout=10)
        logger.info("Container %s restarted successfully", container_name)
        return True
    except docker.errors.NotFound:
        logger.error("Container %s not found", container_name)
        return False
    except docker.errors.APIError as e:
        logger.error("Failed to restart %s: %s", container_name, e)
        return False

What does this code do?

• docker.from_env() creates a Docker client that connects to your locally running Docker Desktop.
• client.containers.get(container_name) looks up the container by its name (like web-flaky).
• container.restart(timeout=10) sends a restart signal, waiting up to 10 seconds before force-killing the container.
• The function catches two specific Docker errors. docker.errors.NotFound handles the case where the container name is wrong. docker.errors.APIError handles Docker server failures.

• Save checker.py.
• Verify the Docker SDK connects to your running Docker Desktop by running this command:

python -c "import docker; client = docker.from_env(); print(client.ping())"

You should see True printed. That confirms the SDK can talk to Docker Desktop.

Seeing an error instead of True?

• Make sure Docker Desktop is running. Check your system tray (Windows) or menu bar (macOS) for the Docker whale icon.
• Make sure your virtual environment is activated. You should see the environment name in your terminal prompt.
• help me connect the Docker SDK to Docker Desktop

Wire auto-remediation into the main flow

Right now your main function loops through services and calls check_health on each one. You need a new function that orchestrates the full flow: check health, and if a service is unhealthy, restart its container and verify it recovered.

• Add the following run_checks function below restart_container in checker.py:

def run_checks(config):
    results = []

    for service in config["services"]:
        result = check_health(service)

        # If the service is down, try to restart its container
        if result["status"] == "unhealthy":
            container_name = service.get("container_name")
            if container_name:
                restarted = restart_container(container_name)
                if restarted:
                    logger.info("Waiting 5 seconds for %s to start...", container_name)
                    time.sleep(5)
                    # Run one final health check to confirm recovery
                    verify = check_health(service)
                    if verify["status"] == "healthy":
                        result["status"] = "restarted"
                        logger.info("%s recovered after restart", service["name"])
                    else:
                        logger.error("%s still unhealthy after restart", service["name"])

        results.append(result)

    return results

What does this code do?

• For each service, it calls check_health first. If the service is healthy, it moves on.
• If a service is unhealthy (all retries exhausted), it looks up the container_name from config.yaml and calls restart_container.
• After a successful restart, it waits 5 seconds for the container to boot, then runs one final check_health to confirm the service actually recovered.
• If the verify passes, the status is updated to "restarted". This three-state result (healthy, unhealthy, restarted) tells you exactly what happened.

Now you need to update main to use run_checks instead of looping through services directly.

• Scroll down to the main function at the bottom of checker.py and find this code:

def main():
    config = load_config("config.yaml")
    for service in config["services"]:
        check_health(service)

• Replace the for loop and the check_health(service) call with a single call to run_checks, so the function looks like this:

def main():
    config = load_config("config.yaml")
    results = run_checks(config)

• Save checker.py.
• Verify the new flow works by running the script with both services still healthy:

python checker.py

You should see both web-healthy and web-flaky reported as healthy with HTTP 200 in the log output. No retries, no restarts.

Seeing an error when running the script?

• If you see a NameError for run_checks, make sure you added the run_checks function above main in the file, not inside another function.
• If both services show as unreachable, confirm both containers are running by running docker ps in your terminal. If web-flaky is stopped, start it with docker start web-flaky.
• help me debug my run_checks function

Add a summary report and test the full auto-remediation flow

The log output tells the full story, but a clean summary table at the end makes it easy to see results at a glance. Each service gets a status icon: OK, FAIL, or FIXED.

• Add the following print_summary function below run_checks and above main in checker.py:

def print_summary(results):
    # Print a formatted summary table
    print("\n" + "=" * 50)
    print("HEALTH CHECK SUMMARY")
    print("=" * 50)
    for r in results:
        status_icon = {"healthy": "OK", "unhealthy": "FAIL", "restarted": "FIXED"}.get(
            r["status"], "UNKNOWN"
        )
        print(f"  [{status_icon}] {r['name']} - {r['status']} ({r['attempts']} attempts)")
    print("=" * 50 + "\n")

What does this code do?

• The function maps each status to a human-readable icon: OK for healthy, FAIL for unhealthy, and FIXED for services that were restarted and recovered.
• Each line shows the status icon, service name, status text, and how many health check attempts were needed.

• Now find the main function, which currently looks like this:

def main():
    config = load_config("config.yaml")
    results = run_checks(config)

• Add print_summary(results) as the last line inside main, so it looks like this:

def main():
    config = load_config("config.yaml")
    results = run_checks(config)
    print_summary(results)

• Save checker.py.

Time for the real test. You are going to intentionally break a running service and watch your tool detect the failure, exhaust retries, restart the container, and verify recovery.

• Stop the web-flaky container to simulate a service failure by running:

docker stop web-flaky

Before you run the health checker, think about what you expect to happen. Will the tool crash like it did back in Step 2? Or will it handle the failure differently this time?

• Run the health checker by running:

python checker.py

Watch the output carefully. You should see the tool check web-healthy (healthy, HTTP 200), then try web-flaky and fail three times with increasing waits (1s, 2s, 4s).

After all retries are exhausted, the tool restarts the web-flaky container, waits 5 seconds, and runs one final health check. The summary table at the end shows [FIXED] for web-flaky and [OK] for web-healthy.

Still seeing web-flaky as unhealthy after restart?

• Make sure Docker Desktop is running. The Docker SDK cannot restart containers if the Docker engine is not active.
• Check that the container_name field in config.yaml exactly matches the container name. Run docker ps -a to see all container names.
• If the container restarts but the verify check still fails, the 5-second wait might not be long enough on your machine. This is normal. The container just needs a moment longer to respond.
• help me fix my auto-remediation flow

✔️ Awesome, I've got everything!

Great. Double check you have saved checker.py.

ⓧ I'd like to double check the full code

Your checker.py is now over 90 lines. Scroll through the file and confirm each section is present, in this order:

• Imports: time, logging, sys, requests, docker, yaml.
• Logging config: logging.basicConfig and logger = logging.getLogger("healthchecker").
• load_config: reads and returns YAML config.
• check_health: HTTP GET with retry loop and exponential backoff.
• restart_container: Docker SDK restart with error handling.
• run_checks: orchestrates health checks and auto-remediation for all services.
• print_summary: formatted table with OK/FAIL/FIXED status icons.
• main: calls load_config, run_checks, and print_summary.

Your tool now detects problems, retries gracefully, restarts failed containers, and verifies recovery. That is the core auto-remediation loop that SRE teams rely on every day. Up next, there is an optional Secret Mission to push your tool even further.

Secret mission

Add Continuous Monitoring with Cooldown

Right now your health checker runs once and exits. Real SRE watchdogs run continuously, checking every N seconds. But a naive loop would restart-storm a service that keeps crashing. Your mission: add a continuous monitoring loop with cooldown logic that prevents restart-looping, turning your one-shot script into a proper background watchdog.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing cloud costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

• Two Docker containers (web-healthy on port 8081 and web-flaky on port 8082).
• Python virtual environment with installed packages (requests, docker, PyYAML).
• Project files (checker.py and config.yaml).

Keep everything running

No action needed. Choose this if you are still actively building or experimenting with the health check tool.

• Your Docker containers will continue running and responding to HTTP requests on ports 8081 and 8082.
• Your virtual environment and project files remain on your Desktop.
• Docker Desktop will continue using system resources while the containers are active.

Pause - I'll come back to this later

Stop the running containers to free up memory and CPU. Your project files and virtual environment stay on your machine so you can pick up where you left off.

• If your health checker is still running in the terminal, press Ctrl+C to stop it.
• Stop both Docker containers by running:

docker stop web-healthy web-flaky

• Deactivate your virtual environment by running:

deactivate

When you come back, restart the containers and reactivate your virtual environment.

• Restart the containers by running:

docker start web-healthy web-flaky

• Reactivate your virtual environment from inside your project folder, then run your health checker again.

Delete - I don't want to use this again

Remove all project resources and start fresh. This deletes your containers, project files, and virtual environment.

• If your health checker is still running in the terminal, press Ctrl+C to stop it.
• Stop and remove both Docker containers by running:

docker stop web-healthy web-flaky
docker rm web-healthy web-flaky

• Deactivate your virtual environment by running:

deactivate

• Delete your project folder from the Desktop.

macOS

• Run this command to delete the project folder:

cd ~/Desktop
rm -rf [[PROJECT_FOLDER="your project folder name"]]

Windows

• Run this command to delete the project folder:

cd ~\Desktop
rmdir /s /q [[PROJECT_FOLDER="your project folder name"]]

• Confirm the folder is gone by running:

ls ~/Desktop

Your project folder should no longer appear in the output.

Nice Work!

Nice Work!

Nice work! You just built a production-grade SRE automation tool from scratch using Python. Your health checker monitors services, retries intelligently, and auto-heals failures without any human intervention.

What you learned:

• Built a Python health check tool that monitors Docker containers via HTTP and handles transient failures with retry logic and exponential backoff.
• Added auto-remediation that detects unhealthy containers and restarts them using the Docker SDK.
• Created a YAML-driven configuration so the tool works with any set of services without changing a single line of code.
• Secret Mission: Added continuous monitoring with cooldown logic to prevent restart loops and run the tool like a real background watchdog.

Ready to quiz yourself?