Dockerize, Break, and Fix a Multi-Container App

Build, break, and debug a 3-service Docker Compose stack like an SRE.

Introduction

30 Second Summary

When something breaks at work, the person who can diagnose and fix the problem is infinitely more valuable than the person who only knows how to follow a setup tutorial. The same is true for containerized applications.

In this project, you will build a multi-container application stack with Docker Compose, then deliberately break it in three ways that mirror real production failures and fix each one using structured debugging.

What You'll Build

A fully orchestrated local stack with a Python Flask API, a PostgreSQL database, and an Nginx reverse proxy, wired together with production patterns like healthchecks, custom networks, and named volumes.

By the end of this project, you'll have:

  • A production-quality multi-stage Dockerfile with layer caching, a non-root user, and minimal image size.
  • A running 3-service Docker Compose stack (Flask API + PostgreSQL + Nginx) with healthchecks, custom networks, and named volumes.
  • Diagnosed and fixed 3 real-world container failures using the same debugging workflow SREs use on-call.
  • Secret Mission: Simulate a container crash loop with memory constraints and diagnose it with restart policies and resource inspection.

Are there any prerequisites?

You should have basic Docker knowledge (images, containers, and Dockerfile basics) and some familiarity with Python. You also need Docker Desktop installed on Windows with the WSL2 backend.

Set Up the Project Structure

Before you can containerize anything, you need an application worth containerizing. A well-organized project structure is essential for Docker because the Dockerfile and Compose file need to reference specific paths to your code, config files, and dependencies.

In this step, you will create the full project directory and write a Flask API that connects to PostgreSQL. This API is what you will containerize, orchestrate, and debug throughout the rest of the project.

In this step, get ready to:
  • Create the project directory with all necessary files and subdirectories.
  • Write a Flask API with health and CRUD endpoints.
  • Pin your Python dependencies in a requirements file.
Create the project directory structure

Your project needs a clear separation between application code, reverse proxy config, and orchestration files. This mirrors how production repositories are organized.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir docker-debug-lab
cd docker-debug-lab
  • Create the subdirectories for your Flask app and Nginx config by running:
mkdir "app", "nginx"
  • Create the root-level project files by running:
New-Item -ItemType File -Path "Dockerfile", "compose.yaml", ".dockerignore", ".env"
  • Create the application and config files inside the subdirectories by running:
New-Item -ItemType File -Path "app/app.py", "app/requirements.txt", "nginx/nginx.conf"
  • Confirm the structure looks correct by running:
ls
ls app
ls nginx

What should I see?

The first ls should show app/, nginx/, Dockerfile, compose.yaml, .dockerignore, and .env. The second should show app.py and requirements.txt. The third should show nginx.conf.

Not seeing .env or .dockerignore?

Files starting with a dot are hidden by default in some file explorers. In PowerShell, ls shows them, but Windows File Explorer may not. If you only see them in the terminal, that is correct.

If the New-Item command failed, make sure you are inside the docker-debug-lab folder by running pwd to check your current location.

Why aren't my dotfiles showing up in Windows?

Write the Flask application and requirements

Your Flask API is a simple notes application with two key features: a health endpoint that checks database connectivity, and CRUD endpoints for creating and reading notes. The health endpoint is especially important because Docker will use it later to determine whether the container is ready to receive traffic.

  • Open app/app.py in your text editor and paste the following code:
import os
import psycopg2
from flask import Flask, jsonify, request

app = Flask(__name__)


def get_db_connection():
    # Connect to PostgreSQL using environment variables
    conn = psycopg2.connect(
        host=os.environ.get("POSTGRES_HOST", "db"),
        database=os.environ.get("POSTGRES_DB", "debuglab"),
        user=os.environ.get("POSTGRES_USER", "labuser"),
        password=os.environ.get("POSTGRES_PASSWORD", "labpass"),
    )
    return conn


def init_db():
    # Create the notes table if it doesn't exist yet
    conn = get_db_connection()
    cur = conn.cursor()
    cur.execute(
        """
        CREATE TABLE IF NOT EXISTS notes (
            id SERIAL PRIMARY KEY,
            content TEXT NOT NULL,
            created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
        )
        """
    )
    conn.commit()
    cur.close()
    conn.close()


@app.route("/health")
def health():
    # Health endpoint - Docker will use this to check container readiness
    try:
        conn = get_db_connection()
        cur = conn.cursor()
        cur.execute("SELECT 1")
        cur.close()
        conn.close()
        return jsonify({"status": "healthy", "database": "connected"})
    except Exception as e:
        return jsonify({"status": "unhealthy", "database": str(e)}), 500


@app.route("/notes", methods=["GET"])
def get_notes():
    # Retrieve all notes, newest first
    conn = get_db_connection()
    cur = conn.cursor()
    cur.execute("SELECT id, content, created_at FROM notes ORDER BY created_at DESC")
    notes = [
        {"id": row[0], "content": row[1], "created_at": row[2].isoformat()}
        for row in cur.fetchall()
    ]
    cur.close()
    conn.close()
    return jsonify(notes)


@app.route("/notes", methods=["POST"])
def create_note():
    # Create a new note from the JSON request body
    data = request.get_json()
    if not data or "content" not in data:
        return jsonify({"error": "content is required"}), 400
    conn = get_db_connection()
    cur = conn.cursor()
    cur.execute(
        "INSERT INTO notes (content) VALUES (%s) RETURNING id",
        (data["content"],),
    )
    note_id = cur.fetchone()[0]
    conn.commit()
    cur.close()
    conn.close()
    return jsonify({"id": note_id, "content": data["content"]}), 201


# Initialize the database table when the app starts
with app.app_context():
    init_db()

What does this code do?

  • get_db_connection() reads database credentials from environment variables. The defaults (like "db" for the host) match the service names you will define in compose.yaml later.
  • init_db() creates the notes table if it does not already exist. This runs automatically when the app starts.
  • /health tests the database connection and returns a JSON status. Docker Compose will call this endpoint to decide when the container is ready.
  • /notes handles GET (list all notes) and POST (create a new note) requests. This gives you real data to test end-to-end once the stack is running.
  • Save app/app.py.

You will not be able to run this app yet. It depends on a PostgreSQL database that will only exist once your Docker Compose stack is running in the next steps.

  • Now open app/requirements.txt and paste these pinned dependencies:
Flask==3.1.3
psycopg2-binary==2.9.10
gunicorn==23.0.0

Why pin exact versions?

Pinning versions ensures that your Docker image builds identically every time. Without pins, a new release of any package could introduce breaking changes and make debugging much harder.

  • Flask is the web framework that handles HTTP routing.
  • psycopg2-binary is the PostgreSQL driver. The binary variant avoids needing C build tools in the Docker image.
  • gunicorn is a production WSGI server that replaces Flask's built-in dev server.
  • Save app/requirements.txt.

✔️ Awesome, I've got everything!

Great. Make sure both app/app.py and app/requirements.txt are saved before continuing.

ⓧ I'd like to double check the full code

Here is the complete app/app.py for comparison:

import os
import psycopg2
from flask import Flask, jsonify, request

app = Flask(__name__)


def get_db_connection():
    conn = psycopg2.connect(
        host=os.environ.get("POSTGRES_HOST", "db"),
        database=os.environ.get("POSTGRES_DB", "debuglab"),
        user=os.environ.get("POSTGRES_USER", "labuser"),
        password=os.environ.get("POSTGRES_PASSWORD", "labpass"),
    )
    return conn


def init_db():
    conn = get_db_connection()
    cur = conn.cursor()
    cur.execute(
        """
        CREATE TABLE IF NOT EXISTS notes (
            id SERIAL PRIMARY KEY,
            content TEXT NOT NULL,
            created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
        )
        """
    )
    conn.commit()
    cur.close()
    conn.close()


@app.route("/health")
def health():
    try:
        conn = get_db_connection()
        cur = conn.cursor()
        cur.execute("SELECT 1")
        cur.close()
        conn.close()
        return jsonify({"status": "healthy", "database": "connected"})
    except Exception as e:
        return jsonify({"status": "unhealthy", "database": str(e)}), 500


@app.route("/notes", methods=["GET"])
def get_notes():
    conn = get_db_connection()
    cur = conn.cursor()
    cur.execute("SELECT id, content, created_at FROM notes ORDER BY created_at DESC")
    notes = [
        {"id": row[0], "content": row[1], "created_at": row[2].isoformat()}
        for row in cur.fetchall()
    ]
    cur.close()
    conn.close()
    return jsonify(notes)


@app.route("/notes", methods=["POST"])
def create_note():
    data = request.get_json()
    if not data or "content" not in data:
        return jsonify({"error": "content is required"}), 400
    conn = get_db_connection()
    cur = conn.cursor()
    cur.execute(
        "INSERT INTO notes (content) VALUES (%s) RETURNING id",
        (data["content"],),
    )
    note_id = cur.fetchone()[0]
    conn.commit()
    cur.close()
    conn.close()
    return jsonify({"id": note_id, "content": data["content"]}), 201


with app.app_context():
    init_db()

And here is app/requirements.txt:

Flask==3.1.3
psycopg2-binary==2.9.10
gunicorn==23.0.0

Seeing a file encoding issue?

If your editor saved the file with BOM (Byte Order Mark) encoding, Python may fail to parse it later. Make sure your editor uses UTF-8 without BOM. In VS Code, check the encoding in the bottom status bar.

Help me check that my Python files are saved with the right encoding.

Your application code is ready. Next up, you will write a production-grade Dockerfile that packages this Flask app into a minimal, secure container image using multi-stage builds.

Write a Production-Grade Dockerfile

Your Flask app and requirements are in place. Now you need to package them into a Docker image that follows production best practices.

A basic Dockerfile can get an app running, but a production-grade one goes further. It uses a multi-stage build to keep the final image small, orders layers to maximize cache efficiency, and runs as a non-root user for security.

In this step, get ready to:
  • Write a multi-stage Dockerfile with layer caching and a non-root user.
  • Create a .dockerignore to keep the build context small.
  • Build the image and verify its size and layer structure.
Write the multi-stage Dockerfile

A multi-stage build separates dependency installation from the final runtime image. The first stage (the builder) installs packages into a virtual environment. The second stage copies only the finished virtual environment, leaving behind pip caches and build tools that would bloat the image.

  • Open Dockerfile in your text editor (it is currently empty).
  • Add the builder stage by pasting this code:
# syntax=docker/dockerfile:1

# === Build stage: install dependencies ===
FROM python:3.13-slim AS builder

WORKDIR /app

RUN python -m venv /app/venv
ENV PATH="/app/venv/bin:$PATH"

COPY app/requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

What does this code do?

  • # syntax=docker/dockerfile:1 enables the latest stable Dockerfile syntax features.
  • FROM python:3.13-slim AS builder starts the builder stage using a minimal Python image and names it builder so the runtime stage can reference it.
  • The virtual environment at /app/venv keeps installed packages in a portable, copyable directory.
  • COPY app/requirements.txt . is placed BEFORE copying app code. Docker can cache the expensive pip install layer as long as requirements do not change.
  • --no-cache-dir tells pip not to store downloaded packages, keeping the layer smaller.

You will not see a change yet. The runtime stage below completes the Dockerfile.

  • Add the runtime stage below the builder stage in the same file:
# === Runtime stage: minimal production image ===
FROM python:3.13-slim

ENV PYTHONUNBUFFERED=1
ENV PYTHONDONTWRITEBYTECODE=1

RUN useradd --create-home appuser
WORKDIR /home/appuser/app

COPY --from=builder /app/venv /home/appuser/venv
ENV PATH="/home/appuser/venv/bin:$PATH"

COPY app/ .

USER appuser

EXPOSE 5000

CMD ["gunicorn", "--bind", "0.0.0.0:5000", "--workers", "2", "app:app"]

What does this code do?

  • A fresh FROM python:3.13-slim starts a clean image with no leftover build artifacts from the first stage.
  • PYTHONUNBUFFERED=1 ensures Python logs appear immediately in docker logs instead of being buffered.
  • PYTHONDONTWRITEBYTECODE=1 prevents Python from writing .pyc files, keeping the container filesystem clean.
  • useradd --create-home appuser creates a non-root user. Running as root inside a container is a security risk because a container escape gives the attacker root on the host.
  • COPY --from=builder pulls only the virtual environment from the builder stage. Nothing else from that stage makes it into the final image.
  • CMD runs Gunicorn (a production WSGI server) with 2 workers, binding to all interfaces on port 5000.
  • Save Dockerfile.

You will not see a visible change yet. The build command in the next substep confirms everything is correct.

Seeing a syntax error when you build later?

  • Make sure the # syntax=docker/dockerfile:1 line is the very first line of the file with no blank lines above it.
  • Check that COPY --from=builder in the runtime stage matches the AS builder alias in the first FROM line exactly.
  • Confirm quotes around ENV PATH values are straight double quotes, not curly smart quotes.

Help me debug my Dockerfile syntax error.

✔️ Awesome, I've got everything!

Great. Double check you have saved Dockerfile before moving on.

ⓧ I'd like to double check the full code

# syntax=docker/dockerfile:1

# === Build stage: install dependencies ===
FROM python:3.13-slim AS builder

WORKDIR /app

RUN python -m venv /app/venv
ENV PATH="/app/venv/bin:$PATH"

COPY app/requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# === Runtime stage: minimal production image ===
FROM python:3.13-slim

ENV PYTHONUNBUFFERED=1
ENV PYTHONDONTWRITEBYTECODE=1

RUN useradd --create-home appuser
WORKDIR /home/appuser/app

COPY --from=builder /app/venv /home/appuser/venv
ENV PATH="/home/appuser/venv/bin:$PATH"

COPY app/ .

USER appuser

EXPOSE 5000

CMD ["gunicorn", "--bind", "0.0.0.0:5000", "--workers", "2", "app:app"]
Write the .dockerignore

Docker sends everything in your project directory to the build engine as "context." A .dockerignore file excludes files that the build does not need, speeding up the transfer and keeping sensitive files out of the image.

  • Open .dockerignore in your text editor (it is currently empty).
  • Add these ignore patterns:
__pycache__
*.pyc
*.pyo
.git
.env
venv/
*.md
.dockerignore

What does this file do?

Each line tells Docker to skip matching files when building the context. __pycache__ and *.pyc exclude compiled Python bytecode. .env keeps your environment variables (which may hold passwords) out of the image layers.

venv/ excludes any local virtual environment since the Dockerfile creates its own inside the builder stage.

  • Save .dockerignore.
  • Confirm the file contents by running:
cat .dockerignore

You should see the list of patterns you just added printed to the terminal.

File appears empty?

  • Make sure you saved the file after adding the patterns.
  • Confirm you are editing .dockerignore (with the leading dot) and not a file named dockerignore without the dot.

Help me fix my .dockerignore file.

Build and verify the image

With the Dockerfile and .dockerignore in place, you can now build the image and confirm that the multi-stage approach produces a small, well-layered result.

  • Build the Docker image by running this command:
docker build -t debug-lab-api .

What does this command do?

  • docker build reads the Dockerfile in the current directory and executes each instruction.
  • -t debug-lab-api tags the resulting image with the name debug-lab-api so you can reference it easily later.
  • The . at the end sets the build context to the current directory.

You should see progress output as Docker processes each stage. The final lines will show the image being tagged as debug-lab-api.

Build failed?

  • If you see an error about requirements.txt not found, check that the COPY path in your Dockerfile says COPY app/requirements.txt . (not just COPY requirements.txt .).
  • If Docker cannot find the Dockerfile, make sure you are running the command from the docker-debug-lab directory.
  • If you see a network error during pip install, check that Docker Desktop is running and has internet access.

Help me fix my Docker build failure.

  • Check the image size by running:
docker images debug-lab-api

You should see a table with your image listed. The SIZE column should show a value under 200MB.

  • Inspect the image layers by running:
docker history debug-lab-api

You should see a table showing each layer, its size, and the command that created it. Only the runtime stage layers appear in the final image.

Why does multi-stage keep the image small?

The builder stage installs packages, downloads wheels, and runs pip. All of that work lives in temporary layers that never make it to the final image.

The runtime stage starts fresh and only copies the finished virtual environment directory. This is why your image is under 200MB instead of 400MB+.

Your production-quality image is built and verified. Next up, you will compose the full 3-service stack with PostgreSQL and Nginx.

Compose the Full Stack

Your API image is built and verified at under 200MB. A single container on its own is useful, but real applications need supporting services.

In this step, you will wire three containers together using Docker Compose. Your Flask API needs a PostgreSQL database for storage and an Nginx reverse proxy to handle incoming traffic. Compose lets you define all three services, their networks, and their startup dependencies in one file.

In this step, get ready to:
  • Write a compose.yaml defining 3 services, 2 networks, and a named volume.
  • Configure Nginx as a reverse proxy to your API.
  • Bring up the full stack and verify all containers are healthy.
Write compose.yaml with all 3 services

Docker Compose reads a single YAML file and creates all the containers, networks, and volumes your stack needs. Each service in the file maps to one running container.

  • Open compose.yaml in your text editor and paste the following configuration:
services:
  api:
    build:
      context: .
    networks:
      - frontend
      - backend
    depends_on:
      db:
        condition: service_healthy
    env_file:
      - .env
    expose:
      - "5000"
    restart: unless-stopped

  db:
    image: postgres:17
    networks:
      - backend
    volumes:
      - pgdata:/var/lib/postgresql/data
    env_file:
      - .env
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
      interval: 5s
      timeout: 5s
      retries: 5
      start_period: 10s
    expose:
      - "5432"
    restart: unless-stopped

  nginx:
    image: nginx:alpine
    networks:
      - frontend
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
    ports:
      - "8080:80"
    depends_on:
      api:
        condition: service_started
    restart: unless-stopped

networks:
  frontend:
  backend:

volumes:
  pgdata:

What does this configuration do?

This file defines three services that work together as a stack:

  • api builds from your Dockerfile and lives on both the frontend and backend networks. It waits for the database healthcheck to pass before starting.
  • db uses the official postgres:17 image and lives on the backend network only. The pgdata named volume persists data across container restarts.
  • nginx uses the nginx:alpine image on the frontend network. It is the only service that publishes a port (8080) to your host machine.

The expose directive makes a port available to other containers on the same network without publishing it to the host. Only ports publishes to your machine.

Why use depends_on with service_healthy?

A plain depends_on only waits for the container to start. It does not wait for the process inside to be ready. With condition: service_healthy, Compose waits until the database healthcheck passes before launching the API.

Without this, the API might try to connect to PostgreSQL before it finishes initializing, causing a crash on startup.

What is the double dollar sign in the healthcheck?

The $$ syntax escapes the dollar sign from Compose's variable interpolation. Without it, Compose would try to expand ${POSTGRES_USER} before passing the command to the shell. The double dollar ensures the shell inside the container resolves the variable at runtime.

  • Save compose.yaml.

You won't see a change yet. The next substep adds the Nginx configuration this file references.

✔️ Awesome, I've got everything!

Great. Double check you've saved the file before moving on.

ⓧ I'd like to double check the full code

services:
  api:
    build:
      context: .
    networks:
      - frontend
      - backend
    depends_on:
      db:
        condition: service_healthy
    env_file:
      - .env
    expose:
      - "5000"
    restart: unless-stopped

  db:
    image: postgres:17
    networks:
      - backend
    volumes:
      - pgdata:/var/lib/postgresql/data
    env_file:
      - .env
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
      interval: 5s
      timeout: 5s
      retries: 5
      start_period: 10s
    expose:
      - "5432"
    restart: unless-stopped

  nginx:
    image: nginx:alpine
    networks:
      - frontend
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
    ports:
      - "8080:80"
    depends_on:
      api:
        condition: service_started
    restart: unless-stopped

networks:
  frontend:
  backend:

volumes:
  pgdata:
Write the Nginx reverse proxy configuration

Nginx sits in front of your API and forwards incoming HTTP requests to the correct backend container. It resolves the service name api through Docker's internal DNS.

  • Open nginx/nginx.conf and paste the following configuration:
events {
    worker_connections 1024;
}

http {
    upstream api_server {
        server api:5000;
    }

    server {
        listen 80;

        location / {
            proxy_pass http://api_server;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }
    }
}

What does this configuration do?

  • upstream api_server defines a named group of backend servers. Here it points to api:5000, where api is the Docker Compose service name that resolves via internal DNS.
  • listen 80 tells Nginx to accept traffic on port 80 inside the container. Your compose.yaml maps host port 8080 to this container port.
  • The proxy_set_header lines forward the original client's IP and protocol to the API so your application logs show real visitor information, not Nginx's internal IP.
  • Save nginx/nginx.conf.

You won't see a change yet. The next substep creates the environment variables and brings up the stack.

✔️ Awesome, I've got everything!

Great. Double check you've saved the file before moving on.

ⓧ I'd like to double check the full code

events {
    worker_connections 1024;
}

http {
    upstream api_server {
        server api:5000;
    }

    server {
        listen 80;

        location / {
            proxy_pass http://api_server;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }
    }
}
Write the .env file and bring up the stack

Both the API and database services load environment variables from the same .env file. This keeps credentials in one place and out of your Compose file.

  • Open .env in your text editor and paste the following:
POSTGRES_USER=labuser
POSTGRES_PASSWORD=labpass
POSTGRES_DB=debuglab
POSTGRES_HOST=db

What do these variables do?

  • POSTGRES_USER, POSTGRES_PASSWORD, and POSTGRES_DB are read by the official PostgreSQL image on first startup to create the database and user.
  • POSTGRES_HOST=db tells your Flask app to connect to the db service by its Compose service name. Docker's internal DNS resolves this to the database container's IP.
  • Save .env.

All three configuration files are ready. Time to bring up the full stack.

  • Build and start all services in detached mode by running this command:
docker compose up -d --build

What does this command do?

  • up creates and starts all services defined in compose.yaml.
  • --build forces a fresh image build for the api service (since it uses build: instead of a pre-built image).
  • -d runs containers in the background so you get your terminal back.

Compose will pull the postgres:17 and nginx:alpine images, build your API image, create the networks and volume, then start all three containers in dependency order.

  • Check the status of all services by running:
docker compose ps

You should see all three services listed. The db service should show healthy in its status column. The api and nginx services should show running.

  • Verify the full request path works by running:
curl http://localhost:8080/health

You should see a JSON response like {"database":"connected","status":"healthy"}. This proves the request traveled from your machine through Nginx, to the API, and successfully queried PostgreSQL.

Services not starting or showing unhealthy?

  • Run docker compose logs to see error output from all services. Look for connection errors or missing files.
  • If db exits immediately, check that your .env file contains POSTGRES_PASSWORD. PostgreSQL refuses to start without it.
  • If nginx exits, check that nginx/nginx.conf exists at the path your volume mount expects. A typo in the path causes Nginx to fail on startup.

Still stuck? Help me debug why my Docker Compose services are not starting.

Your three-container stack is live. Next up, you will trace requests across the full path and inspect the network topology to see how Docker isolates your services.

Verify and Inspect the Running Stack

Your stack is running with all three containers healthy. Before you start breaking things, you need to understand exactly how requests flow through the system and how the containers communicate.

In this step, you will trace a request end-to-end from Nginx through the API to the database. You will also inspect the network topology and poke around inside a running container to see how your production patterns (non-root user, named volumes, healthchecks) look from the inside.

In this step, get ready to:
  • Test the full request path through all three services.
  • Inspect the network topology and DNS resolution between containers.
  • Explore container internals to verify production patterns.
Test the full request path

A multi-container stack has multiple points of failure between the user and the database. Testing the full path confirms that Nginx proxies correctly, the API handles requests, and PostgreSQL stores data.

  • Hit the health endpoint through Nginx by running this command:
curl http://localhost:8080/health

What does this command do?

This sends a GET request to port 8080, which Nginx listens on. Nginx forwards the request to the API on port 5000, and the API checks its connection to PostgreSQL before responding.

You should see a JSON response like {"database":"connected","status":"healthy"}.

  • Create a note by POSTing data through Nginx:
curl -X POST http://localhost:8080/notes -H "Content-Type: application/json" -d "{\"content\": \"My first note\"}"

You should see a response with the note ID and content, like {"content":"My first note","id":1}.

  • Retrieve your notes by running:
curl http://localhost:8080/notes

You should see a JSON array containing the note you just created.

  • Now test data persistence by restarting the API container:
docker compose restart api
  • After the restart completes, fetch notes again to confirm they survived:
curl http://localhost:8080/notes

Why does data persist after a restart?

The pgdata named volume in your compose.yaml stores PostgreSQL's data outside the container's filesystem. Restarting or even recreating the api container does not touch the database volume.

This is why named volumes are critical in production. Without one, every container restart would wipe your database clean.

Not seeing your notes after restart?

The API container may still be starting up. Wait a few seconds and try the curl command again. You can check its status with docker compose ps.

If you see a connection refused error, the depends_on condition only applies on first startup. After a restart, the API reconnects to the already-healthy database.

ask me for help debugging

Inspect the network topology

Your compose.yaml defines two networks: frontend and backend. Understanding which containers live on which network is essential for debugging connectivity issues later.

  • List all Docker networks on your system by running:
docker network ls

You should see your project's two networks listed with names like docker-debug-lab_frontend and docker-debug-lab_backend.

  • Inspect the backend network to see which containers are attached and their IP addresses:
docker network inspect docker-debug-lab_backend

What should I see?

The output shows a Containers section listing the api and db containers with their assigned IP addresses. The nginx container is NOT on this network. It connects to the API through the frontend network instead.

This isolation means nginx can never reach db directly. All database traffic must go through the API. This is the same tier-separation pattern used in production deployments.

  • Confirm DNS resolution works between services by running:
docker compose exec api getent hosts db

What does this command do?

This runs getent hosts db inside the running api container. It queries the container's DNS resolver for the hostname db. Docker Compose automatically registers service names as DNS entries, so containers can reach each other by name instead of IP address.

You should see the IP address that matches what docker network inspect showed for the db container.

  • View interleaved logs from all services by running:
docker compose logs --tail=20

You should see the last 20 log lines from each service, prefixed with the service name. This is the first command to reach for when something goes wrong in a multi-container stack.

Network name looks different?

Docker Compose prefixes network names with the project directory name. If your folder is named differently than docker-debug-lab, your network names will use that folder name as the prefix. Adjust the docker network inspect command to match what docker network ls shows.

ask me for help

Inspect container internals

Knowing what is running inside a container is a core debugging skill. You can open a shell inside any running container to check processes, file permissions, and environment.

  • Open a shell inside the running API container:
docker compose exec api sh
  • Check which user the process is running as:
whoami

You should see appuser printed. This confirms your Dockerfile's USER appuser directive is working. Running as non-root is a production security best practice.

  • Check the running processes inside the container:
ps aux

You should see gunicorn master and worker processes. This is your application server running with the 2 workers you configured in the Dockerfile.

  • Exit the container shell by running:
exit
  • Inspect the database container's full configuration to see volume mounts, healthcheck config, and network settings:
docker inspect docker-debug-lab-db-1

What should I look for?

The output is a large JSON document. Look for these three sections:

  • Mounts shows the pgdata named volume mapped to /var/lib/postgresql/data inside the container.
  • Healthcheck shows the pg_isready command and your interval, timeout, and retries configuration.
  • NetworkSettings shows the container is attached only to the backend network with its assigned IP.

Filter docker inspect output

You can use Go template formatting to extract specific fields. For example, to see just the healthcheck status:

docker inspect --format='{{json .State.Health}}' docker-debug-lab-db-1

Container name not found?

Docker Compose names containers using the pattern <project>-<service>-<replica>. Run docker compose ps to see the exact container names in your project, then use that name in the docker inspect command.

ask me for help

You now have a complete mental map of your stack. You know how requests flow, how containers find each other via DNS, and what is running inside each container. Next up, you will systematically break this stack in three ways and use these same inspection tools to debug each failure.

Break and Fix the Stack

Your stack is running smoothly with all three services connected and healthy.

Now you will deliberately break it in three ways that mirror the most common production Docker failures. For each scenario, you will follow the same structured debugging workflow that SREs use on-call: check status, read logs, inspect internals, identify root cause, fix, and verify.

In this step, get ready to:
  • Diagnose and fix a network misconfiguration that isolates the API from the database.
  • Diagnose and fix a missing environment variable that crashes PostgreSQL on startup.
  • Diagnose and fix a wrong volume mount path that breaks Nginx.
Scenario 1: Break the network configuration

In production, network misconfigurations are one of the most common causes of service outages. A container that loses access to its database network will fail silently until the next request hits it.

What are we simulating?

You will remove the api service from the backend network. This isolates it from the PostgreSQL database, which only lives on the backend network. The API will be unable to resolve the hostname db and every database connection will fail.

  • In compose.yaml, find the api service's networks section. It currently looks like this:
    networks:
      - frontend
      - backend
  • Remove the - backend line so it only has frontend:
    networks:
      - frontend
  • Save compose.yaml and apply the broken configuration by running:
docker compose up -d

The stack will come up, but the API will fail its health check because it cannot reach the database.

  • Start the debugging workflow by checking container status:
docker compose ps
  • Read the API logs to see what went wrong:
docker compose logs api
  • Inspect the backend network to confirm the api container is missing from it:
docker network inspect docker-debug-lab_backend

What do these diagnostics reveal?

  • docker compose ps shows the api container as unhealthy or restarting.
  • docker compose logs api shows connection refused or hostname resolution errors when trying to reach db.
  • docker network inspect confirms the api container is NOT attached to the backend network where the db container lives.
  • Fix the issue by restoring - backend to the api service's networks section in compose.yaml. The section should look like this again:
    networks:
      - frontend
      - backend
  • Save compose.yaml and apply the fix by running:
docker compose up -d
  • Verify the fix by checking container status:
docker compose ps

All three services should show as healthy or running again.

API still showing unhealthy?

Make sure your compose.yaml has both - frontend and - backend under the api service's networks key, indented correctly. YAML is whitespace-sensitive.

If the container keeps restarting, wait 10-15 seconds for the db healthcheck to pass before the api can start.

help me debug why my api container can't connect to the database after restoring the network

Scenario 2: Remove a critical environment variable

Missing or misconfigured environment variables are another top cause of container crashes. The PostgreSQL Docker image requires POSTGRES_PASSWORD to be set, and it will refuse to start without it.

  • Open the .env file in the docker-debug-lab folder. It currently looks like this:
POSTGRES_USER=labuser
POSTGRES_PASSWORD=labpass
POSTGRES_DB=debuglab
POSTGRES_HOST=db
  • Delete the entire POSTGRES_PASSWORD=labpass line so the file looks like this:
POSTGRES_USER=labuser
POSTGRES_DB=debuglab
POSTGRES_HOST=db
  • Save .env and force-recreate the containers so they pick up the new environment by running:
docker compose up -d --force-recreate

The db container will exit immediately because PostgreSQL requires a password to initialize.

  • Check container status to observe the failure:
docker compose ps
  • Read the database logs to find the root cause:
docker compose logs db

What do the logs show?

You will see an error message saying the database is uninitialized and the superuser password is not specified. PostgreSQL's Docker image enforces this requirement. Without POSTGRES_PASSWORD, the container refuses to start.

docker compose ps shows the db container has exited. Since the api depends on db being healthy, the api also fails to start.

  • Fix the issue by restoring the POSTGRES_PASSWORD=labpass line in .env. The file should look like this:
POSTGRES_USER=labuser
POSTGRES_PASSWORD=labpass
POSTGRES_DB=debuglab
POSTGRES_HOST=db
  • Save .env. Because the database failed on first initialization, the volume contains incomplete data. Tear everything down including volumes, then bring it back up by running:
docker compose down -v
docker compose up -d

Why is the -v flag needed?

The -v flag removes named volumes. When PostgreSQL fails during its first initialization, it leaves behind a partially-initialized data directory on the pgdata volume. On the next start, PostgreSQL sees this partial state and refuses to reinitialize. Deleting the volume forces a clean start.

  • Verify the fix by checking that all containers are healthy:
docker compose ps

All three services should show as running and healthy.

✔️ Awesome, I've got everything!

Great. Double check you have saved your .env file with the restored POSTGRES_PASSWORD line.

ⓧ I'd like to double check the full code

POSTGRES_USER=labuser
POSTGRES_PASSWORD=labpass
POSTGRES_DB=debuglab
POSTGRES_HOST=db

Database still not starting?

Make sure you ran docker compose down -v before bringing the stack back up. Without the -v flag, the corrupted volume persists and PostgreSQL will keep failing.

Check that POSTGRES_PASSWORD=labpass is on its own line in .env with no leading spaces or extra characters.

help me debug why my PostgreSQL container won't start after I restored the password

Scenario 3: Misconfigure the Nginx volume mount

Volume mount mistakes are subtle. The container starts, but the mounted file lands at the wrong path inside the container, causing configuration conflicts that only surface in the logs.

What are we simulating?

The Nginx Alpine image ships with a main /etc/nginx/nginx.conf that includes all .conf files from /etc/nginx/conf.d/ inside its http {} block. If you mount your custom config (which contains a top-level events {} block) to /etc/nginx/conf.d/default.conf, Nginx will try to parse the events directive inside the http context, which is invalid.

  • In compose.yaml, find the nginx service's volumes section. It currently looks like this:
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
  • Change the mount destination to /etc/nginx/conf.d/default.conf so it reads:
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/conf.d/default.conf:ro
  • Save compose.yaml and apply the broken configuration by running:
docker compose up -d

Nginx will either exit immediately or return errors on every request.

  • Check container status:
docker compose ps
  • Read the nginx logs to see the configuration error:
docker compose logs nginx

What do the logs show?

You will see an error like "events" directive is not allowed here. This happens because the nginx:alpine main config file includes conf.d/*.conf inside its http {} block. Your custom file has a top-level events {} block, which is only valid at the main config level.

This is why mounting a full nginx.conf as a conf.d include fails. The correct path is /etc/nginx/nginx.conf, which replaces the main config entirely.

  • Fix the issue by restoring the correct mount path in compose.yaml. Change the nginx volumes section back to:
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
  • Save compose.yaml and rebuild by running:
docker compose up -d
  • Verify the entire stack is healthy by running:
docker compose ps
  • Confirm end-to-end connectivity by hitting the health endpoint through Nginx:
curl http://localhost:8080/health

You should see a JSON response with "status": "healthy" and "database": "connected".

✔️ Awesome, I've got everything!

Great. Double check you have saved compose.yaml with the correct mount path restored.

ⓧ I'd like to double check the full code

services:
  api:
    build:
      context: .
    networks:
      - frontend
      - backend
    depends_on:
      db:
        condition: service_healthy
    env_file:
      - .env
    expose:
      - "5000"
    restart: unless-stopped

  db:
    image: postgres:17
    networks:
      - backend
    volumes:
      - pgdata:/var/lib/postgresql/data
    env_file:
      - .env
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
      interval: 5s
      timeout: 5s
      retries: 5
      start_period: 10s
    expose:
      - "5432"
    restart: unless-stopped

  nginx:
    image: nginx:alpine
    networks:
      - frontend
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
    ports:
      - "8080:80"
    depends_on:
      api:
        condition: service_started
    restart: unless-stopped

networks:
  frontend:
  backend:

volumes:
  pgdata:

Nginx still returning errors?

Make sure the volume mount path is exactly ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro with no trailing spaces. The :ro suffix means read-only and should be at the end.

If the container was cached with the old config, try docker compose up -d --force-recreate to ensure Nginx picks up the corrected mount.

help me figure out why Nginx is still failing after I fixed the volume mount path

The debugging workflow you just practiced

Each scenario followed the same structured approach. This is the workflow you can reuse on any container failure:

  • Check status with docker compose ps to identify which service is unhealthy.
  • Read logs with docker compose logs <service> to find the error message.
  • Inspect internals with docker network inspect or docker inspect to confirm your hypothesis.
  • Identify root cause by connecting the error message to the configuration change.
  • Fix the configuration and re-apply with docker compose up -d.
  • Verify with docker compose ps and a live request to confirm the fix worked.

Secret mission

Simulate a Crash Loop

In production, containers don't just fail once. They crash, restart, crash again, and eventually exhaust their restart budget. Can you simulate an OOM crash loop using memory limits and restart policies, then diagnose it the way an SRE would on-call?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Docker containers (api, db, nginx services).
  • Docker volume (pgdata for PostgreSQL data persistence).
  • Docker networks (frontend, backend).
  • Locally built Docker image (debug-lab-api).
  • Pulled Docker images (postgres:17, nginx:alpine).
  • Project folder (docker-debug-lab on your Desktop).

Keep everything running

No action needed. Choose this if you're still actively experimenting with the stack or want to try more debugging scenarios.

  • Your containers, volumes, networks, and images remain on your machine.
  • The api container is currently stopped after the crash loop from the Secret Mission. Run docker compose up -d --build from the docker-debug-lab folder to bring it back.
  • Everything is local. There is no billing or metered usage.

Pause - I'll come back to this later

Stop all running containers to free up system memory, but keep your files, images, and volumes intact so you can resume later.

  • Open your terminal in the docker-debug-lab folder.
  • Stop all services by running:
docker compose stop
  • Your containers are now stopped but not removed. Your pgdata volume and images stay on disk.
  • When you're ready to come back, run docker compose up -d from the same folder to restart everything.

Delete - I don't want to use this again

Remove all project containers, volumes, networks, and locally built images. This frees up disk space completely.

  • Open your terminal in the docker-debug-lab folder.
  • Remove all containers, the named volume, project networks, and locally built images by running:
docker compose down -v --rmi local
  • Remove the pulled images (postgres:17 and nginx:alpine) by running:
docker rmi postgres:17 nginx:alpine
  • Navigate to your Desktop and delete the project folder by running:
cd ~/Desktop
rm -rf docker-debug-lab

On Windows without WSL?

If you're running commands in PowerShell directly (not inside WSL), use this instead to delete the project folder:

cd ~\Desktop
Remove-Item -Recurse -Force docker-debug-lab
  • Confirm everything is gone by running:
docker images
docker volume ls
  • You should no longer see debug-lab-api, postgres:17, or nginx:alpine in the images list. The pgdata volume should not appear in the volume list.

Nice Work!

Nice Work!

Nice work! You just built, broke, and fixed a production-grade multi-container application stack using Docker and Docker Compose.

You've learned how to:

  • Build a production-quality multi-stage Dockerfile with layer caching, virtual environment isolation, and non-root execution.
  • Orchestrate a 3-service stack with Docker Compose using healthcheck-gated startup, custom network isolation, and named volumes for data persistence.
  • Debug and fix real-world container failures (network misconfiguration, missing environment variables, and volume mount issues) using the same structured debugging workflow SREs use on-call.
  • (Secret Mission) Simulated and diagnosed an OOM crash loop using memory limits, restart policies, and docker inspect to confirm the kill reason.

Ready to quiz yourself?