Build an Operable Service

Ship a FastAPI service with externalized config, honest health signals, and traces.

Introduction

30 Second Summary

Every service eventually breaks. The question is whether it tells you before your users do.

In this project, you will build a containerized FastAPI service that reads all configuration from the environment, reports its own health through two separate endpoints, and exports every request as a distributed trace you can follow in a local dashboard.

What You'll Build

You will start the entire stack with one command, kill the database, and watch one health endpoint stay green while the other turns red. That single result proves the service can distinguish "I am alive" from "I can serve traffic."

By the end of this project, you'll have:

  • A running service started with docker compose up --build that connects to PostgreSQL through a single environment variable. Swap the variable, swap the database. No code change needed.
  • Two health endpoints that tell separate truths. Kill the database and /healthz/live still returns 200 while /healthz/ready returns 503. The gap between those two responses is the design heart of this project.
  • End-to-end distributed traces flowing to Jaeger where you can follow a single request through the HTTP layer, your application logic, and the database call.
  • Three MADR v4 decision records and a C4 container diagram drafted by AI, then corrected by you. The correction is the design lesson.
  • Secret Mission: Ship a five-slide readout, a teach-back page, and an after-action review, then prove the whole thing works from a clean clone with a measured wall-clock time.

Are there any prerequisites?

This project has a day-zero setup gate in Step 0 that confirms your environment before the clock starts. You need a GitHub account, an editor with Dev Containers support (VS Code or Cursor), Docker Desktop running, and Git installed.

Familiarity with Python and HTTP APIs is assumed. The devcontainer in the starter repo makes the build environment identical regardless of your host OS.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

Write the Constraints and Records

Every service starts with decisions, not code. The AI will draft three decision records and a container diagram from the constraints file already in your repo.

Your job is to review and correct those drafts. An unread rationale is a default, not a decision. The container view comes before the build so the running pieces and how they talk are agreed before the first line of implementation.

In this step, get ready to:
  • Confirm the devcontainer tools and pre-staged stack.
  • Draft and review three MADR v4 decision records with the AI.
  • Draft a C4 container diagram and close Issue 1.
Confirm your tools and review the repo

The devcontainer standardises the environment so it is identical regardless of host OS. Before writing anything, confirm the pinned tools work inside it.

  • In your terminal inside the devcontainer, confirm Docker Compose is available by running:
  • docker compose version
docker compose version
  • Confirm Python is available by running:
python --version
  • Confirm Git is available by running:
git --version

You should see 3.12 for Python and version output for both Docker Compose and Git.

Seeing command not found?

Make sure you are running the commands inside the devcontainer, not your host terminal. Look for the green "Dev Container" badge in the bottom-left corner of VS Code.

If the devcontainer failed to build, reopen the folder and select Reopen in Container from the command palette.

help me get the devcontainer running

  • Verify the pre-staged stack starts by running:
docker compose up --build

What does this command do?

This builds the app container from the Dockerfile and starts all three services defined in compose.yaml: the app, PostgreSQL (postgres:17), and Jaeger (jaegertracing/jaeger:2.19.0).

Jaeger v1 all-in-one reached end of life on 31 December 2025. The v2 image is the current supported line, built on the OpenTelemetry Collector framework.

You should see log lines from all three containers appearing in the terminal. The stack is healthy once you see the Jaeger UI is ready on port 16686.

  • Press Ctrl+C to stop the stack.

Containers failing to start?

If you see port-in-use errors, check that nothing else is running on ports 8000, 5432, or 16686. Run docker compose down and try again.

If the postgres container fails its healthcheck, ensure Docker Desktop has at least 4 GB of memory allocated.

help me get docker compose up working

  • Review the repo structure by running:
ls

You should see compose.yaml, Dockerfile, main.py (skeleton), AGENTS.md, .factor-key.sealed, .devcontainer/, and .github/ISSUE_TEMPLATE/.

  • Open AGENTS.md in the editor and read through it.

What is AGENTS.md?

This file codifies the five rules from the Twelve-Factor App (Adam Wiggins, Heroku, published January 2012) as standing constraints for any AI agent working in this repo.

The five factors in scope are III Config, IV Backing Services, IX Disposability, X Dev/prod parity, and XI Logs. Config in the environment is why one build runs anywhere. The backing service attached by URL is why readiness can be honest.

Draft and review three decision records

MADR is Markdown Architectural Decision Records, a lean format for recording design choices. Version 4 was released September 2024.

  • Create the directory for your decision records by running:
mkdir -p docs/decisions
  • Confirm the directory exists by running:
ls docs/

You should see decisions in the output.

  • Direct your AI coding assistant to draft three MADR v4 decision records. Give it the following prompt (adapt to your tool):
Read AGENTS.md. Draft three MADR v4 decision records and save them to docs/decisions/:

1. 0001-postgres-over-sqlite.md
2. 0002-opentelemetry-over-vendor-agent.md
3. 0003-separate-liveness-readiness.md

Each record MUST include:
- Context and Problem Statement
- Decision Drivers (reference the Twelve-Factor rule it settles)
- Considered Options (at least two)
- Decision Outcome with justification
- Consequences (Good and Bad)
- Pros and Cons of the Options
- More Information (include a reversal trigger: when would we revisit this decision?)

Each must name the alternative it beat and the tradeoff accepted.

What is a reversal trigger?

A reversal trigger is the condition under which you would reopen the decision. For example, the Postgres ADR might say: "Revisit if the service runs in an environment where installing a database server is impractical and the data fits in a single file."

Recording it under "More Information" forces you to think about when this choice stops being the right one.

  • Wait for the AI to generate all three files.
  • Open each file in the editor and review the AI's output.
  • In each record, correct at least one of the following: a factual claim that is wrong or overstated, a missing tradeoff the AI ignored, or a weak reversal trigger that would never actually fire.
  • Save all three files.

Why must I correct the AI's drafts?

AI agents tend to produce plausible-sounding rationale that agrees with the chosen option. A document you accepted without editing is a default, not a decision.

Common AI misses to watch for: SQLite's concurrency limits understated, OpenTelemetry's library size overstated, or a reversal trigger so broad it would never actually cause a revisit.

  • Confirm all three files exist by running:
ls docs/decisions/

You should see 0001-postgres-over-sqlite.md, 0002-opentelemetry-over-vendor-agent.md, and 0003-separate-liveness-readiness.md.

AI created files in the wrong location?

Some AI assistants create files relative to the workspace root while others use the current terminal directory. If the files ended up in a different folder, move them with mv.

help me move my ADR files to the right directory

Draft the container diagram and close Issue 1

The C4 model draws software at four zoom levels. This project uses level 2, the container view: the running pieces and how they talk.

  • Direct your AI to draft a C4 container-level Mermaid diagram. Give it the following prompt:
Read AGENTS.md. Draft a C4 container-level Mermaid diagram and save it to docs/container-view.md.

The diagram uses the C4Container diagram type and must show:
- Three containers: app (FastAPI on port 8000), postgres (port 5432), jaeger (ports 4317/4318/16686)
- Connections: app to postgres via DATABASE_URL, app to jaeger via OTLP HTTP on port 4318
- External actor: On-call engineer accessing Jaeger UI on port 16686 and app health endpoints on port 8000

Use Mermaid's C4Container syntax with Container, ContainerDb, Person_Ext, and Rel elements.

What is a C4 container diagram?

A container diagram shows the high-level shape of a system: the processes that run (web app, database, message queue) and how they communicate. It answers the question "what runs and what talks to what" without going into internal code structure.

Mermaid supports C4 diagrams natively using the C4Container diagram type. Elements include Container, ContainerDb, Person_Ext, and Rel for relationships.

  • Open docs/container-view.md and review the AI's output.
  • Correct any naming inconsistencies (for example, the AI might label the database as "PostgreSQL" in one place and "DB" in another, or use the wrong port for OTLP).
  • Confirm the diagram shows DATABASE_URL as the connection label from app to postgres, and OTLP HTTP on port 4318 from app to jaeger.
  • Save docs/container-view.md.
  • Confirm the file exists by running:
ls docs/

You should see both decisions and container-view.md listed.

Now commit this work and close Issue 1 through a pull request.

  • Create the scaffold branch by running:
git checkout -b scaffold
  • Stage and commit the documentation by running:
git add docs/
git commit -m "Add decision records and container view"
  • Push the branch by running:
git push -u origin scaffold
  • Open your repo on GitHub in the browser.
  • You should see a banner offering to create a pull request from the scaffold branch. Click Compare & pull request.
  • In the PR description, type Closes #1 so the issue closes automatically when the PR merges.
  • Click Create pull request.
  • Click Merge pull request, then Confirm merge.

Why Closes #1 in the PR body?

GitHub links issues to pull requests through keywords in the PR description. Writing Closes #1 tells GitHub to move Issue 1 to "Done" on your Project board when the PR merges.

This is the branch-per-issue workflow: one issue, one branch, one PR, one merge. Six issues will mean six clean PRs by the end of this project.

  • Switch back to main locally by running:
git checkout main
git pull

Check your Project board on GitHub. Issue 1 should now show as closed, with one item in the Done column.

Before moving on, confirm the stack still starts after merging the docs.

  • What do you expect to happen when you run docker compose up --build now? You only added markdown files, so predict whether the three containers will still start cleanly.
  • Run the stack to confirm by running:
docker compose up --build

You should see the same three containers start successfully. The Jaeger UI is available at http://localhost:16686.

  • Press Ctrl+C to stop the stack.

What did this step prove?

You now have three corrected decision records explaining why Postgres over SQLite, why OpenTelemetry over a vendor agent, and why separate liveness and readiness endpoints. Each names the alternative, the tradeoff, and the reversal trigger.

The C4 container diagram shows exactly what will run and how it communicates. The stack still starts cleanly because documentation does not affect build or runtime.

Your design is recorded and your first issue is closed. Next up, you will move config out of the code so the service reads everything from environment variables.

Move Config Out of the Code

The skeleton main.py in your repo has no implementation yet. In a typical codebase, the database connection string would be hardcoded directly in the source. That locks the build to a single environment and means changing the database requires editing code and redeploying.

Moving the connection string to an environment variable settles Factor III (config in the environment, so one build runs anywhere) and Factor IV (backing service attached by URL, so swapping databases means changing the variable, not the code). Adding structured JSON logging to stdout settles Factor XI (logs as an event stream the platform routes, not a file the app manages).

In this step, get ready to:
  • Implement the inventory service with config read entirely from environment variables.
  • Rebuild and verify the service responds with data from Postgres.
  • Close issue 2 using the branch-per-issue workflow.
Direct the AI to implement the service

Your AI coding assistant will generate the implementation across four files. The key constraint is that no host, password, or port may appear anywhere in the Python source.

  • In your AI assistant, direct it to implement the following in main.py:
  • DATABASE_URL read from os.environ with no default (crash if missing).
  • A psycopg connection using that URL.
  • An items table created on startup (id serial, name text, created_at timestamptz).
  • POST /items and GET /items endpoints.
  • Structured JSON logging to stdout: one JSON object per line, no file.
  • Also direct the AI to update requirements.txt, compose.yaml, and the Dockerfile to support the implementation.

Review the AI's output against the expected patterns below. Your AI may format things slightly differently, but these elements must be present.

  • Verify requirements.txt contains these pinned dependencies:
fastapi==0.139.2
uvicorn==0.34.0
psycopg[binary,pool]==3.3.4
opentelemetry-distro==0.52b0
opentelemetry-exporter-otlp==1.31.0

Why pin the OpenTelemetry packages now?

The Dockerfile runs opentelemetry-bootstrap -a install which auto-detects FastAPI and psycopg, then installs their instrumentors. Pinning the base packages now means tracing works the moment you wire it in Step 4, with no extra install step.

  • Verify the Dockerfile matches this structure:
FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt && \
    opentelemetry-bootstrap -a install

COPY . .

CMD ["opentelemetry-instrument", \
     "--traces_exporter", "otlp", \
     "--metrics_exporter", "none", \
     "--logs_exporter", "none", \
     "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

What does this Dockerfile do?

The opentelemetry-instrument wrapper in the CMD line auto-instruments the app at startup. It reads OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_SERVICE_NAME from environment variables, which keeps Factor III intact. The app process itself does not import or configure tracing.

  • Verify the app service in compose.yaml passes config through environment variables:
services:
  app:
    build: .
    ports:
      - "8000:8000"
    environment:
      - DATABASE_URL=postgresql://inventory:inventory@postgres:5432/inventory
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4318
      - OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
      - OTEL_SERVICE_NAME=inventory-service
    depends_on:
      postgres:
        condition: service_healthy
      jaeger:
        condition: service_started
    restart: unless-stopped

Why does the connection string live here, not in the code?

Factor III says config belongs in the environment. The Python source reads os.environ["DATABASE_URL"] and crashes if it is missing. That means the app has zero knowledge of hostnames or passwords. Swapping to a different Postgres instance means editing this one line in the compose file, not touching the code.

  • Verify main.py starts with the config and logging setup:
import os
import logging
import json
import sys
from contextlib import asynccontextmanager

import psycopg
from psycopg.rows import dict_row
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel

# Read config from environment - crash if missing (Factor III)
DATABASE_URL = os.environ["DATABASE_URL"]

# Structured JSON logging to stdout (Factor XI)
logger = logging.getLogger("inventory-service")
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(logging.Formatter(json.dumps({
    "time": "%(asctime)s",
    "level": "%(levelname)s",
    "message": "%(message)s"
})))
logger.addHandler(handler)
logger.setLevel(logging.INFO)

What does this code do?

os.environ["DATABASE_URL"] reads the variable and raises a KeyError if it is not set. There is no fallback. That is intentional. A missing variable means a misconfigured environment, which must fail loudly rather than silently connect to a default.

The logger writes one JSON object per line to sys.stdout. Docker captures stdout automatically. The platform (Docker, Kubernetes, a managed service) routes the stream wherever it needs to go. The app never opens a log file.

  • Verify main.py has the connection helper and table creation on startup:
def get_connection():
    # Connect using the environment-provided URL (Factor IV)
    return psycopg.connect(DATABASE_URL, row_factory=dict_row)


@asynccontextmanager
async def lifespan(app: FastAPI):
    # Create table on startup (idempotent)
    with get_connection() as conn:
        conn.execute("""
            CREATE TABLE IF NOT EXISTS items (
                id SERIAL PRIMARY KEY,
                name TEXT NOT NULL,
                created_at TIMESTAMPTZ DEFAULT NOW()
            )
        """)
        conn.commit()
    logger.info("database_initialized")
    yield


app = FastAPI(lifespan=lifespan)


class ItemCreate(BaseModel):
    name: str

What does this code do?

get_connection() builds a connection from DATABASE_URL. It never constructs a URL from parts. That is Factor IV: the backing service is attached by URL, making it swappable without code changes.

The lifespan context manager runs once when the app starts. CREATE TABLE IF NOT EXISTS means the table is created on first boot and left alone on subsequent restarts.

  • Verify main.py has the API endpoints:
@app.get("/")
def root():
    return {"service": "inventory-service", "status": "running"}


@app.post("/items", status_code=201)
def create_item(item: ItemCreate):
    with get_connection() as conn:
        result = conn.execute(
            "INSERT INTO items (name) VALUES (%s) RETURNING id, name, created_at",
            (item.name,)
        ).fetchone()
        conn.commit()
    logger.info(f"item_created name={item.name}")
    return result


@app.get("/items")
def list_items():
    with get_connection() as conn:
        items = conn.execute("SELECT id, name, created_at FROM items ORDER BY id").fetchall()
    return items

What does this code do?

POST /items inserts a row and returns the created record with its generated ID and timestamp. GET /items lists everything in the table ordered by ID.

Every endpoint logs its action as a structured JSON line. No endpoint hardcodes a host, port, or password. The connection comes from the environment-provided URL.

  • Save all four files: main.py, requirements.txt, Dockerfile, compose.yaml.

AI generated something different?

The critical invariants are: DATABASE_URL read with os.environ["DATABASE_URL"] (no .get() with a default), logging to sys.stdout as JSON, and the opentelemetry-instrument wrapper in the Dockerfile CMD. If your AI used os.getenv with a fallback string, correct it. A default defeats the whole point of Factor III.

help me fix my AI-generated code to match the Factor III pattern

Rebuild and test the stack

The code is ready. Time to see it run against the real Postgres container.

  • Rebuild and start the stack by running:
docker compose up --build

What does this command do?

The --build flag forces Docker to rebuild the app image with your new requirements.txt and main.py. Docker Compose then starts all three containers (app, postgres, jaeger) with the environment variables injected.

You should see log output from all three containers. The app container prints database_initialized once Postgres is healthy and the table is created.

  • Open a second terminal inside the devcontainer. Create an item by running:
curl -X POST http://localhost:8000/items \
  -H "Content-Type: application/json" \
  -d '{"name": "oxygen-sensor"}'

You should see a JSON response with an id, name, and created_at timestamp.

  • List all items by running:
curl http://localhost:8000/items

You should see a JSON array containing the item you just created.

  • Confirm structured logs are flowing by running:
docker compose logs app | grep item_created

You should see a JSON log line containing item_created name=oxygen-sensor. One JSON object per line, written to stdout, routed by Docker. That is Factor XI.

  • Confirm no credentials exist in the Python source by running:
grep -n "inventory" main.py

This should return no matches. The word inventory (the database name and password) appears only in compose.yaml, never in main.py.

App container keeps restarting?

Check docker compose logs app for the error. If you see KeyError: 'DATABASE_URL', the environment variable is not reaching the container. Confirm the environment: block in compose.yaml is indented under the app service, not at the top level.

If you see a psycopg connection error, confirm the depends_on condition waits for service_healthy on the postgres service. Without it, the app may start before Postgres is ready to accept connections.

help me debug my docker compose app container not starting

Prove config is actually read

The service works, but how do you know it is actually reading from the environment rather than falling back to a hardcoded default? You prove it by breaking the variable.

  • Stop the running stack with Ctrl+C in the terminal where docker compose up is running.
  • In compose.yaml, find the DATABASE_URL line under the app service's environment: block.
  • Change the hostname from postgres to does-not-exist:
      - DATABASE_URL=postgresql://inventory:inventory@does-not-exist:5432/inventory

Before you restart, predict what will happen. Will the app start normally, crash immediately, or hang?

  • Restart the stack by running:
docker compose up --build

The app container crashes on startup because it cannot resolve does-not-exist as a hostname. This proves DATABASE_URL is genuinely read from the environment and used for the connection. There is no silent fallback.

  • Revert compose.yaml back to the correct hostname postgres:
      - DATABASE_URL=postgresql://inventory:inventory@postgres:5432/inventory
  • Restart one more time to confirm recovery:
docker compose up --build

The stack comes back healthy. The app container starts, connects to Postgres, and prints database_initialized.

What did that prove?

A wrong DATABASE_URL crashes the app. A correct one lets it start. That is Factor III working as designed. One build, configurable entirely from the outside. No code change required to point it at a different database.

Close the issue
  • Stop the stack with Ctrl+C.
  • Create the branch and commit by running:
git checkout -b config
git add .
git commit -m "feat: implement service with externalized config and structured logging"
  • Push the branch by running:
git push -u origin config
  • Open your repo on GitHub.
  • Click Compare & pull request for the config branch.
  • In the PR description, type Closes #2 to link the PR to issue 2.
  • Click Create pull request, then Merge pull request, then Confirm merge.

Issue 2 closes automatically. Your Project board now shows 2 items in the Done column.

Issue didn't close automatically?

The PR description must contain Closes #2 (capital C, with the hash). If you merged without it, open issue 2 manually and click Close issue.

help me link a pull request to close a GitHub issue

Three Twelve-Factor principles settled in one step. Your service reads config from the environment, connects to Postgres as an attached resource, and logs structured JSON to stdout. Next up, you will split the health signal into two endpoints that tell different truths.

Split Liveness from Readiness

A health check that answers OK while its database is dead is the same defect as an oxygen compressor nobody monitored. It reports healthy while its dependency is gone, and the on-call engineer trusts a green tick that means nothing.

This step builds two separate endpoints whose separation is the design heart of the project. ADR 0003 recorded the decision. The dead-database test at the end proves it.

In this step, get ready to:
  • Add a liveness endpoint that proves the process is running.
  • Add a readiness endpoint that proves the service can serve traffic.
  • Run the dead-database proof to confirm the two signals are genuinely separate.
Add the liveness endpoint

The liveness endpoint answers one question: is the process running? An orchestrator uses this signal to decide whether to kill and replace the container, or leave it alone.

  • In main.py, find the root() function.
  • Add the following endpoint directly below it, before create_item:
@app.get("/healthz/live")
def liveness():
    # Unconditionally return 200 - proves the process is alive
    return {"status": "ok"}

What does this code do?

This endpoint always returns HTTP 200 with {"status":"ok"}. It does not check any dependency.

That is intentional. If the process can respond at all, it is alive. A failing liveness check means the process is stuck or crashed and should be replaced.

  • Save main.py.
  • Rebuild the stack by running:
docker compose up --build -d
  • Test the new endpoint by running:
curl http://localhost:8000/healthz/live

You should see {"status":"ok"} with no delay.

Seeing connection refused?

The container may still be starting. Wait a few seconds and try the curl command again. If it persists, check docker compose logs app for a Python syntax error.

help me debug why my app container is not responding

Add the readiness endpoint

Readiness answers a different question: can this process serve traffic right now? A load balancer uses this to decide whether to send requests, or drain.

Conflating liveness and readiness means a slow database causes a restart storm instead of a traffic drain.

  • In main.py, add the following endpoint directly below the liveness() function you just added:
@app.get("/healthz/ready")
def readiness():
    # Actually test the database connection with a 2-second timeout
    try:
        with psycopg.connect(DATABASE_URL, connect_timeout=2) as conn:
            conn.execute("SELECT 1")
        return {"status": "ok", "checks": {"postgres": "reachable"}}
    except Exception:
        raise HTTPException(
            status_code=503,
            detail={"status": "unavailable", "checks": {"postgres": "unreachable"}}
        )

What does this code do?

  • psycopg.connect(DATABASE_URL, connect_timeout=2) opens a fresh connection to Postgres with a hard 2-second deadline.
  • SELECT 1 is the cheapest possible query. If it succeeds, the database is reachable and accepting queries.
  • On failure, the endpoint raises HTTP 503 with a body explaining which check failed. The load balancer reads 503 as "stop sending traffic here."
  • Save main.py.
  • Rebuild the stack by running:
docker compose up --build -d
  • Test the readiness endpoint by running:
curl http://localhost:8000/healthz/ready

You should see {"status":"ok","checks":{"postgres":"reachable"}} because Postgres is healthy right now.

Seeing a 500 error instead of 200?

A 500 usually means a Python exception before the endpoint logic runs. Check that HTTPException is still in your import line: from fastapi import FastAPI, HTTPException.

help me debug my readiness endpoint

Run the dead-database proof

Both endpoints respond 200 right now because everything is healthy. The real test is what happens when Postgres goes away.

  • Kill the Postgres container by running:
docker compose stop postgres

Before you run the next command, predict: will /healthz/live return 200 or fail?

  • Hit the liveness endpoint by running:
curl http://localhost:8000/healthz/live

You still see {"status":"ok"}. The process is alive. It just cannot serve traffic.

Now predict: what will /healthz/ready return with Postgres stopped?

  • Hit the readiness endpoint by running:
curl http://localhost:8000/healthz/ready

You see {"detail":{"status":"unavailable","checks":{"postgres":"unreachable"}}} with HTTP 503.

Why this gap is the whole point

200 on live, 503 on ready, same dead database. The two signals are genuinely separate, not one check wearing two names.

A single /health endpoint would have reported failure for both. The orchestrator would kill the container (restart storm) when the correct action is to stop routing traffic until Postgres recovers.

  • Bring Postgres back by running:
docker compose start postgres
  • Wait a few seconds, then confirm readiness recovers by running:
curl http://localhost:8000/healthz/ready

You should see {"status":"ok","checks":{"postgres":"reachable"}} again. The service self-healed without a restart.

  • Create the branch, commit, and push by running:
git checkout -b health
git add .
git commit -m "Add liveness and readiness health endpoints"
git push -u origin health
  • Open a pull request on GitHub.
  • In the PR description, type Closes #3 so merging closes the issue automatically.
  • Merge the PR.
  • Switch back to main locally by running:
git checkout main
git pull

Issue 3 is closed. Your board now shows 3 done.

Readiness still returning 503 after restarting Postgres?

Postgres needs a few seconds to become ready after docker compose start postgres. Wait 5 seconds and try again. You can also check Postgres status with docker compose ps and look for "healthy" in the status column.

help me debug why readiness is not recovering

✔️ Awesome, I've got everything!

Great. Double check you have saved main.py and your PR is merged.

ⓧ I'd like to double check the full code

Here is the complete main.py at the end of this step. Scroll through and confirm your file matches.

Your file has these sections in order:

  • Imports and DATABASE_URL from environment.
  • Logger setup (structured JSON to stdout).
  • get_connection() helper.
  • Lifespan context manager creating the items table.
  • The root() endpoint at /.
  • The liveness() endpoint at /healthz/live.
  • The readiness() endpoint at /healthz/ready.
  • The create_item() and list_items() endpoints.

Your service now announces its own health honestly. Next up, you will wire OpenTelemetry so every request leaves a trace an on-call engineer can follow.

Wire Telemetry and Logs

Your service can now tell an orchestrator whether it is alive and whether it can serve traffic. But when a request fails at 2 AM, those two signals only say something is wrong.

Without tracing, the on-call engineer's only diagnostic tool is grep. OpenTelemetry auto-instruments every inbound request and outbound database call as spans, exported to Jaeger via OTLP (the wire format traces travel over). In this step you confirm the telemetry wiring already staged in your repo, rebuild the stack, and prove traces flow end-to-end.

In this step, get ready to:
  • Confirm the OpenTelemetry auto-instrumentation configuration across three files.
  • Rebuild the stack and verify traces appear in Jaeger.
  • Run the full proof sequence and close issue 4.
Confirm the OpenTelemetry auto-instrumentation

ADR 0002 recorded the decision: OpenTelemetry over a vendor-specific agent. The implementation lives across three files that are already in your repo.

  • Open requirements.txt and confirm these two packages are present:
opentelemetry-distro==0.52b0
opentelemetry-exporter-otlp==1.31.0

What do these packages do?

opentelemetry-distro bundles the OpenTelemetry API, SDK, and the opentelemetry-bootstrap CLI tool that auto-detects your libraries.

opentelemetry-exporter-otlp adds the exporter that sends trace data to Jaeger over HTTP/protobuf on port 4318.

  • Open Dockerfile and confirm the two telemetry lines:
RUN pip install --no-cache-dir -r requirements.txt && \
    opentelemetry-bootstrap -a install
CMD ["opentelemetry-instrument", \
     "--traces_exporter", "otlp", \
     "--metrics_exporter", "none", \
     "--logs_exporter", "none", \
     "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

What does each piece do?

  • opentelemetry-bootstrap -a install detects FastAPI and psycopg at build time and installs their instrumentor packages automatically.
  • opentelemetry-instrument wraps uvicorn at runtime, injecting trace hooks into every request and database call without touching your application code.
  • --traces_exporter otlp sends spans to the endpoint defined in the environment. --metrics_exporter none and --logs_exporter none keep the output focused on traces only.
  • Open compose.yaml and confirm the three OTEL environment variables under the app service:
environment:
  - DATABASE_URL=postgresql://inventory:inventory@postgres:5432/inventory
  - OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4318
  - OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
  - OTEL_SERVICE_NAME=inventory-service

Why environment variables instead of code?

This is Factor III (config in the environment) at work again. The same built image can export traces to a cloud-hosted collector in production just by changing OTEL_EXPORTER_OTLP_ENDPOINT. No rebuild needed.

Rebuild and verify traces in Jaeger

The configuration is confirmed. Now rebuild the stack so the instrumentation activates and generate some traffic to produce traces.

  • Rebuild and restart the stack by running:
docker compose up --build -d
  • Wait a few seconds for all three containers to start, then create an item by running:
curl -X POST http://localhost:8000/items \
  -H "Content-Type: application/json" \
  -d '{"name": "telemetry-test"}'

You should see a JSON response with an id, name, and created_at field.

  • List items to generate a second trace by running:
curl http://localhost:8000/items

You should see a JSON array containing the item you just created.

Before you open Jaeger, predict: how many spans do you expect in a single trace for one request?

  • Open Jaeger in your browser at http://localhost:16686.
  • In the Service dropdown, select inventory-service.
  • Click Find Traces.
  • Click on any trace to expand it.

You should see at least 3 spans in a parent-child chain sharing one trace ID: the HTTP request span, a FastAPI handler span, and a psycopg database call span. Three in a parent-child chain means the path through the service is visible rather than guessed.

Don't see inventory-service in the dropdown?

  • Make sure your curl requests returned successful responses. If POST /items returned a connection error, the app container may not be ready yet. Wait 10 seconds and try again.
  • Check that the Jaeger container is running by running docker compose ps and confirming jaeger shows as running.
  • Verify OTEL_EXPORTER_OTLP_ENDPOINT in compose.yaml points to http://jaeger:4318 (not localhost, because the app container resolves the Jaeger container by service name).

Help me debug why traces aren't showing in Jaeger.

Run the full proof sequence and close issue 4

With traces flowing, you can now run the complete proof sequence that demonstrates the service is genuinely operable. This sequence covers liveness, tracing, and the dead-database test from the previous step.

  • Verify liveness responds quickly by running:
curl -w "\nTime: %{time_total}s\n" http://localhost:8000/healthz/live

You should see {"status":"ok"} with a total time well under 200ms.

  • Confirm you already verified proof (b) in the previous substep: one trace with 3+ spans sharing a trace ID in a parent-child chain in Jaeger.
  • Now run proof (c). Stop the postgres container by running:
docker compose stop postgres
  • Test liveness while the database is down:
curl http://localhost:8000/healthz/live

You should see {"status":"ok"} with HTTP 200. The process is alive.

  • Test readiness while the database is down:
curl -s -o - -w "\nHTTP %{http_code}\n" http://localhost:8000/healthz/ready

You should see HTTP 503 with {"detail":{"status":"unavailable","checks":{"postgres":"unreachable"}}}. That gap (200 on live, 503 on ready with the same dead database) is the entire point of separate signals.

  • Bring postgres back by running:
docker compose start postgres
  • Confirm readiness recovers by running:
curl http://localhost:8000/healthz/ready

You should see {"status":"ok","checks":{"postgres":"reachable"}} with HTTP 200 again.

What did the full proof sequence just demonstrate?

  • Proof (a): The liveness endpoint responds in under 200ms. Fast, unconditional, safe for orchestrator kill decisions.
  • Proof (b): A trace carries 3+ spans in a parent-child chain. The on-call engineer sees the full request path, not a guess.
  • Proof (c): Liveness and readiness are genuinely separate signals. A dead database drains traffic without triggering a restart storm.
  • Create the telemetry branch by running:
git checkout -b telemetry
  • Stage, commit, and push by running:
git add -A
git commit -m "Wire OpenTelemetry auto-instrumentation to Jaeger"
git push -u origin telemetry
  • Open your repo on GitHub.
  • Click Compare & pull request.
  • In the PR description, type Closes #4.
  • Click Create pull request, then Merge pull request, then Confirm merge.

Issue 4 closes automatically. Your board now shows 4 done.

  • Switch back to main locally and pull the merge by running:
git checkout main
git pull

PR says there are merge conflicts?

  • Make sure you merged the health branch PR from the previous step before creating this one. Each branch should be based on the latest main.
  • If you forgot to pull main before branching, run git pull origin main on your telemetry branch, resolve conflicts, commit, and push again.

Help me resolve merge conflicts on my telemetry branch.

Your service now has full observability: honest health signals and end-to-end traces. Next up, you will score the AI against the sealed key and write the docs that make this service operable by anyone on the team.

Write the Docs and the Teach-Back

Your service runs, traces flow to Jaeger, and the health endpoints tell genuinely separate truths. But a service is not operable if nobody else can run it.

In this step, you score the AI against a sealed answer key to measure whether it can actually audit the compliance it claims. An audit that agrees with everything means the output was accepted without review. That is the failure the score exists to catch. Then you write the teach-back page that lets a teammate start the whole stack with one command.

In this step, get ready to:
  • Score the AI's audit against the sealed factor key.
  • Write a teach-back page that lets a teammate run the stack with one command.
  • Tag the release and close the repo with five issues done.
Direct the AI to audit the repo

The five Twelve-Factor App rules in scope (III Config, IV Backing Services, IX Disposability, X Dev/prod parity, XI Logs) plus the readiness contract form the audit checklist. The AI reports which file satisfies or violates each rule.

  • Open your AI coding assistant (the same one you used for decision records and implementation).
  • Direct it to audit the repo against the five Twelve-Factor rules in scope plus the readiness contract (the promise the service makes about when it is safe to send traffic).
  • Tell it to report which file satisfies or violates each rule, with specific line references.
  • Save the AI's report to docs/factor-audit.md.

What should the audit cover?

The AI should check each rule against your actual codebase:

  • Factor III (Config): Is DATABASE_URL read from the environment with no default?
  • Factor IV (Backing Services): Is Postgres reached only through the URL variable?
  • Factor IX (Disposability): Can any container be killed and replaced without losing state?
  • Factor X (Dev/prod parity): Does the compose stack mirror the deployed shape?
  • Factor XI (Logs): Does the app write structured JSON to stdout with no file?
  • Readiness contract: Do /healthz/live and /healthz/ready tell separate truths?
  • Confirm the file exists by running:
ls docs/factor-audit.md

You should see the file listed in the output.

File not found?

Make sure your AI saved the file to the docs/ directory, not the repo root. If the docs/ directory does not exist yet, create it first with mkdir -p docs and re-save.

help me find where my AI saved the audit file

Score the AI against the sealed key

The sealed key holds the ground truth for what passes and what does not. Your job is to hold the AI's report against that truth and count where it got things right, wrong, or missed entirely.

  • Open .factor-key.sealed in your editor.
  • Open docs/factor-audit.md side by side.

Before you compare them line by line, predict: do you think the AI got every rule right, or did it miss something?

  • Compare the AI's audit against the sealed key. Record three counts:
  • Confirmed means the AI claimed passing and the key agrees.
  • Contradicted means the AI claimed passing but the key disagrees.
  • Missed means the key holds a factor the AI never reported on.

What does the AI typically miss?

Expect the AI to miss the liveness/readiness split being genuinely separate rather than one check wearing two names. AI training data overwhelmingly shows a single /health endpoint, so agents tend to report "health check present, factor satisfied" without verifying that liveness and readiness are genuinely different signals.

Catching that miss is a real design review. It proves you read the audit rather than accepting it.

  • Calculate accuracy using this formula: Accuracy = Confirmed / (Confirmed + Contradicted + Missed).
  • Create docs/ai-score.md and record the three counts plus the accuracy number.
  • Note the specific miss (or misses) the AI made, so the record is honest about what the AI got wrong.
  • Confirm both docs exist by running:
ls docs/factor-audit.md docs/ai-score.md

You should see both files listed.

Accuracy is 100%?

If your accuracy came out to 1.0 (100%), re-read the sealed key carefully. An audit that agrees with everything means the output was accepted without review. Check whether the key distinguishes between "a health endpoint exists" and "liveness and readiness are genuinely separate signals." The AI almost always conflates these.

help me understand what the AI missed in its factor audit

Write the teach-back and tag the release

The teach-back is the final proof of operability. If a teammate who has never seen the repo cannot run the whole stack from this page alone, the service is not operable yet.

  • Create docs/teach-back.md with the following sections:
  • A one-command start instruction: docker compose up --build. If it takes two commands and a paragraph, it is not finished.
  • An explanation of what each container does (app, postgres, jaeger).
  • A defence of each of the three design decisions, referencing the ADRs in docs/decisions/.
  • The global problem (what the project is about), what this build does about it, and what it does not.

Why carry the stake forward?

The teach-back is not just technical documentation. It carries the honesty boundary: what this service proves (config separation, honest readiness, tracing on one machine) and what it does not prove (behavior under load, scaling, real deployment). A teammate reading this page should know both.

  • Confirm all three docs exist by running:
ls docs/factor-audit.md docs/ai-score.md docs/teach-back.md

You should see all three files listed.

  • Create the docs branch, commit, and push by running:
git checkout -b docs
git add .
git commit -m "Add factor audit, AI score, and teach-back"
git push -u origin docs
  • Open a pull request on GitHub with Closes #5 in the PR body.
  • Merge the PR. Issue 5 closes automatically.
  • Switch back to main and pull the merge by running:
git checkout main
git pull
  • Tag the release by running:
git tag v1.0.0
git push origin v1.0.0

Before you check the board, predict: how many issues should show as done?

  • Open your GitHub Project board. You should see five issues closed and five PRs merged.

Issue 5 still open?

Check that your PR body contained exactly Closes #5 (capital C, correct issue number). GitHub only auto-closes issues when the keyword and number match exactly. If the PR is already merged, close the issue manually from the issue page.

help me figure out why my GitHub issue didn't auto-close

Your repo is tagged at v1.0.0 with five issues closed, five PRs merged, and the board nearly clear. Next up in the Secret Mission, you prove this whole thing works from a clean clone and write the readout that summarizes what it all means.

Secret mission

Demo, Readout, AAR, and Clean-Clone Proof

Ship the readout, teach-back, AAR, and honesty note, then prove from a clean clone on a second directory. The build is not done until someone else can run it and the builder can say what it proved and what it did not.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally using Docker containers and a free-tier GitHub repo, so there are no ongoing costs.

Resources you used:

  • Three Docker containers (app, postgres, jaeger) running locally.
  • A Postgres Docker volume (pgdata) persisting data between restarts.
  • A private GitHub repo and Project board (free tier, no cost).
  • Local project directories (the original clone and, if you completed the Secret Mission, the clean-clone proof directory).

Keep everything running

No action needed. Choose this if you are still actively building or experimenting with the service.

  • Leave docker compose up running to continue sending requests and viewing traces in Jaeger.
  • Your Postgres volume retains all items you created between restarts.
  • Your GitHub repo and Project board remain accessible at no cost on the free tier.

Pause - I'll come back to this later

Shut down running containers to free up memory and CPU, but keep your volumes and files intact so you can resume later.

  • Stop all containers by running:
docker compose stop

This halts the containers without removing the Postgres volume. When you are ready to resume, run docker compose up from the same project directory to pick up where you left off.

Delete - I don't want to use this again

Remove all project resources and start fresh. This is irreversible for your local data.

  • Stop and remove all containers AND the Postgres volume by running:
docker compose down -v
  • Delete the local project directory. On macOS or Linux:
cd ~/Desktop
rm -rf operable-service
  • On Windows:
cd ~\Desktop
rmdir /s /q operable-service
  • If you completed the Secret Mission, also remove the second clean-clone directory you created for the feasibility proof.
  • To delete the GitHub repo, go to your repo page on GitHub, click Settings, scroll to the bottom under Danger Zone, and click Delete this repository. Follow the confirmation prompts. This also removes the linked Project board.

Nice Work!

Nice Work!

You just built and shipped an operable service that reads every setting from the environment, states its health honestly, and leaves a trace on every request. The repo is closed, the AI is scored, and a teammate can clone it and run the whole thing with one command.

You've learned how to:

  • Build a FastAPI service that reads all config from environment variables, connects to Postgres as an attached resource, and writes structured JSON logs to stdout.
  • Implement separate liveness and readiness health endpoints that tell different truths, proving with a dead-database test that the two signals are genuinely independent.
  • Wire OpenTelemetry auto-instrumentation exporting traces to Jaeger v2, close a repo with six issues, six branches, six PRs merged, a release tagged, and an AI accuracy score recorded against a sealed key.
  • Secret Mission: Ship the readout, AAR, and honesty note, then prove the whole thing from a clean clone with a recorded wall-clock time. The skill that transfers: making a system state its health honestly and proving that statement can fail.

Ready to quiz yourself?