Operate a Production Reliability Lab

Monitor and recover a FastAPI incident with Prometheus and Grafana.

Introduction

30 Second Summary

When an app starts rejecting customer requests, knowing that something broke is only the beginning. Without clear evidence, every minute of investigation becomes guesswork.

In Part 3 of App Dev, you will operate a FastAPI inventory service as its on-call engineer inside a local Docker Compose reliability lab. Prometheus metrics will feed a Grafana dashboard where structured logging and correlation IDs support investigation and recovery.

What You'll Build

Your local control room starts with healthy inventory traffic, shows a deliberate validation-error spike, and settles back to normal after you stop the malformed client traffic.

By the end of this project, you'll have:

  • Live readiness and metrics endpoints where you can open /ready to confirm database readiness or open /metrics to inspect application signals.
  • An operational dashboard where request rate, error rate, p95 latency, and duplicate-event evidence respond to inventory traffic.
  • A repeatable incident investigation that generates HTTP 422 responses, traces one correlation ID to a structured log, identifies the malformed payload, and records the recovery in an on-call runbook.
  • Secret Mission: Add a Prometheus alert that enters firing state during the error spike and returns to inactive after recovery.

Are there any prerequisites?

You need the completed Part 2 reliable-inventory-sync-api repository with its Dockerfile and four passing tests. The optional Part 1 negative-inventory Secret Mission is not required.

Your Windows setup should already include Visual Studio Code, Git, Docker Desktop, Docker Compose, and the project's Python virtual environment.

Before We Start

Operating the inherited reliable-inventory-sync-api as its on-call engineer starts with one customer-impact signal: an elevated HTTP error rate. Prometheus metrics, readiness status, correlation IDs, and structured API logs support your root-cause explanation.

Verify the Reliability Lab Baseline

Reliable incident evidence depends on a known-good starting point. An inherited failure could make a later observability change look responsible for an older problem.

You will inspect the Part 2 repository in Visual Studio Code. Git and pytest will prove that its existing code still passes the inherited test suite.

You will also verify Docker Desktop plus Docker Compose. A temporary container will provide the final health check.

In this step, get ready to:
  • Confirm the inherited repository state.
  • Preserve the existing test baseline.
  • Verify the released API through a temporary container.
Confirm the inherited repository

The .venv environment keeps the inherited Python dependencies available to this repository. Activating it ensures that the test command uses the project environment.

  • Press the Windows key to open Windows search.
  • Type Visual Studio Code.
  • Press Enter to open Visual Studio Code.
  • Click File in the top menu bar.
  • Click Open Folder.
  • Select the inherited reliable-inventory-sync-api folder.
  • Click Select Folder.
  • Press Ctrl+` to open the integrated terminal.

Visual Studio Code starts its integrated terminal inside the open repository. The prompt should identify PowerShell before you activate the environment.

  • Select PowerShell from the terminal profile dropdown if another shell is active.
  • Activate the inherited .venv environment by running this command:
.\.venv\Scripts\Activate.ps1

What does activation change?

Activation places the virtual environment at the front of the terminal session. The python command now uses the interpreter plus packages prepared for this repository.

  • Confirm that (.venv) appears at the start of the PowerShell prompt.

Virtual environment not activating?

Check that the terminal is inside the reliable-inventory-sync-api folder. Confirm that .venv appears in the Visual Studio Code Explorer.

A PowerShell policy can block local activation scripts. Help me activate this existing virtual environment safely.

A clean repository keeps the baseline trustworthy. Any unexpected edit could affect the test result before the reliability work begins.

  • Inspect the current repository state by running this command:
git status

What does this check prove?

Git reports the current branch plus any local file changes. This lets you separate inherited code from changes introduced during this project.

  • Confirm that the current branch matches the Part 2 branch you intended to use.
  • Confirm that the repository has no unexpected changes.

Repository state looks unfamiliar?

If Git cannot inspect the repository, check that the Explorer shows requirements.txt plus Dockerfile inside reliable-inventory-sync-api.

If files have unexpected edits, compare them with your completed Part 2 work before continuing. Help me interpret my Git status safely.

The inherited tests define the behavior that later instrumentation must preserve. Your required baseline contains four passing tests.

Before you run the suite, predict whether the untouched Part 2 repository still satisfies every inherited test.

  • Run the inherited test suite with the active virtual environment by using this command:
python -m pytest

What does this test run establish?

  • The test suite checks the inherited health endpoint plus inventory behavior.
  • The result creates a reference point for the observability changes added later.

You should see four passing tests. Five passing tests are also valid if you completed the optional negative-inventory extension from Part 1.

That baseline is locked in. The inherited API behavior now has test evidence behind it.

Tests not passing?

Confirm that (.venv) still appears in the prompt. Check that the terminal remains inside reliable-inventory-sync-api.

If a test fails, keep its output visible before changing any files. Help me diagnose the inherited test failure.

Verify Docker and Compose

The API image depends on Docker Engine for containers. Compose becomes the control layer for the reliability lab in the next step.

  • Press the Windows key to open Windows search.
  • Type Docker Desktop.
  • Press Enter to start Docker Desktop.
  • Return to the PowerShell terminal in Visual Studio Code.
  • Verify Docker Engine plus Docker Compose by running these commands:
docker version
docker compose version

What do these checks prove?

  • The first command checks communication with the Docker client plus server.
  • The second command confirms that the Compose command is available.

The Docker output should include client information plus server information. The Compose command should print its version.

Missing Docker server information?

Docker Desktop may still be starting its engine. Wait for startup to complete before rerunning the checks.

If the Compose command does not respond, restart Docker Desktop before trying again. Help me verify Docker and Compose on Windows.

Recheck the released API behavior

Tests prove the source behaves correctly inside Python. A temporary container proves that the existing Part 2 image still serves the released health endpoint.

  • Enter the local Part 2 image name plus tag here: your Part 2 image name and tag.
  • Start the image on the loopback address by running this command:
docker run -d -p 127.0.0.1:8000:8000 [[PART2_IMAGE="your Part 2 image name and tag"]]

What does this command do?

Detached mode returns control to PowerShell while the API continues running. The port mapping makes container port 8000 available through port 8000 on your computer.

Binding the host side to the loopback address keeps this temporary API endpoint local to your machine.

  • Record the container ID printed by Docker here: the container ID printed by docker run.

Before you open the endpoint, predict whether the released image still returns the original health response.

  • Open http://127.0.0.1:8000/health in your browser.

You should see {"status":"ok"}. This proves that the released API responds through the container boundary.

  • Stop the temporary release container by running this command:
docker stop [[CONTAINER_ID="the container ID printed by docker run"]]

What confirms the container stopped?

Docker prints the stopped container identifier. Port 8000 is now free for the Compose service created in the next step.

The complete baseline now checks out. Your tests pass, your container toolchain responds, and the released API serves its health endpoint.

Health endpoint not loading?

Confirm that the image reference includes the Part 2 tag. Check that another local process is not already using port 8000.

If the container exits immediately, inspect the image reference before rerunning it. Help me troubleshoot the temporary API container.

Your inherited release is healthy and ready for controlled experimentation. Next, you will run it with Docker Compose and test what raw container output can actually explain.

Expose the Observability Gap

The health check proves that your inventory API can respond. It does not show traffic volume, failures, latency, or duplicate deliveries.

In this step, you will run the inherited API with Docker Compose. You will generate real inventory traffic before testing how much the raw service output can explain.

In this step, get ready to:
  • Run the inherited API as a Docker Compose service.
  • Submit the same valid inventory event twice.
  • Test whether the service output can answer four operational questions.
Create the smallest Compose service

A Compose file records how a group of containers should run. This first definition contains only the existing API so you can observe its current behavior.

  • Create compose.yaml inside reliable-inventory-sync-api with the definition below by using the New File control in the Visual Studio Code Explorer sidebar:
services:
  api:
    build: .
    ports:
      - "127.0.0.1:8000:8000"

What does this configuration do?

  • The api service represents your inherited inventory application.
  • The build: . setting builds the service from the Dockerfile in reliable-inventory-sync-api.
  • The port mapping publishes the API at 127.0.0.1:8000 on your computer.
  • Save compose.yaml.
  • Build and start the inherited API by running this command in the activated PowerShell terminal:
docker compose up --build -d

What does this command do?

  • The --build option rebuilds the API image from the inherited Dockerfile.
  • The -d option leaves the service running in the background.
  • Check the Compose service state by running this command:
docker compose ps

What should you see?

Good. Your api service should be listed with a running state.

The published ports should connect 127.0.0.1:8000 on your computer to port 8000 in the container.

🙋‍♀️ Service not running?

Confirm Docker Desktop is still running. Check that compose.yaml uses spaces for indentation.

Ask for help with the build output if the service stops unexpectedly. Help me diagnose why my Compose API service is not running.

✔️ Awesome, I've got everything!

Your compose.yaml file is saved. The api service is running on 127.0.0.1:8000.

ⓧ I'd like to double check the full code

Compare your complete compose.yaml file with this version.

services:
  api:
    build: .
    ports:
      - "127.0.0.1:8000:8000"
Produce a visible inventory baseline

The API already protects inventory from duplicate event deliveries. Sending the same event_id twice lets you confirm that behavior before examining the available evidence.

  • Enter http://127.0.0.1:8000/docs in your browser address bar.
  • Expand the POST /inventory/events endpoint.
  • Use the page control that makes the request body editable.
  • Replace the example request body by pasting this inventory event:
{
        "event_id": "event-2001",
        "store_id": "store-202",
        "sku": "oat-milk",
        "quantity_change": 4
    }

What does this event prove?

  • The event_id gives this delivery a stable identity.
  • The quantity_change value asks the API to add four units of oat-milk inventory.
  • Reusing the same event tests whether the API ignores a duplicate delivery.
  • Submit the inventory event from the documentation page.

The response should contain applied: true. The first delivery has changed the stored quantity.

  • Submit the same request again without changing its body.

That replay proves the duplicate protection works. The second response should contain applied: false.

Responses not showing the expected values?

Confirm both requests use event-2001 as the event_id. Check that the first request body uses the numeric value 4 for quantity_change.

If the endpoint page does not respond, confirm the api service remains running. Help me troubleshoot the inventory event responses.

Test the raw service output

Container output can prove that the application wrote messages while handling traffic. An operational signal must also support calculations across many requests.

Before you inspect the output, which of the four operational questions do you expect it to answer?

  • Display the current API service output by running this command:
docker compose logs api

What should you see?

You should see output from the api service for the requests you sent. Treat these lines as the only evidence available for the next check.

🙋‍♀️ No API output?

Return to the documentation page and submit the event once more. Run the log command again after the API responds.

If the service has stopped, inspect its state before continuing. Help me find why my API logs are empty.

  • Calculate the request rate from the displayed output.
  • Calculate the error rate from the displayed output.
  • Calculate the p95 request latency from the displayed output.
  • Determine the duplicate-event count from the displayed output.

You can confirm that the service produced output. The lines do not provide enough consistent measurements to calculate those four KPIs reliably.

This shortfall is intentional. A responding API still leaves an on-call engineer without the evidence needed to measure impact.

You've proved the API works while its output leaves key operational questions unanswered. Next, you'll turn those questions into measurable request evidence.

Instrument the Inventory API

Your inherited API handles inventory events correctly. However, the raw Docker Compose output cannot explain traffic volume, failures, latency, or duplicate deliveries.

This step adds structured logging, correlation IDs, readiness reporting, and application metrics to the FastAPI service. These signals turn each request into evidence you can use during triage and recovery.

In this step, get ready to:
  • Add the Prometheus dependency and a SQLite readiness check.
  • Instrument every request with metrics, structured logs, and a correlation ID.
  • Expose the new signals and confirm all five tests pass.
Add the metrics dependency and readiness check

The Python client for Prometheus creates counters and histograms inside your application. Prometheus can later collect those measurements from an HTTP endpoint.

  • Switch back to requirements.txt in Visual Studio Code.
  • Add the following dependency below pydantic==2.13.5:
prometheus-client==0.26.0

What Does This Dependency Add?

The Prometheus client provides the Counter and Histogram metric types. It also formats the collected measurements for the new /metrics endpoint.

  • Save requirements.txt.
  • Install the updated dependencies into the active virtual environment by running:
python -m pip install -r requirements.txt

The installation finishes without a dependency error. Your virtual environment can now import the Prometheus client.

Dependency Installation Failing?

Confirm that your PowerShell prompt still shows the active .venv environment. A network or package-index error may require you to retry the command once your connection is stable.

If the command still fails, ask for help diagnosing the installation output.

A readiness check answers whether the API can use a dependency required to serve traffic. This project checks SQLite with a small query that always closes its connection.

  • Switch back to database.py.
  • Add the following function below initialize_database() and above _read_quantity():
def database_is_ready() -> bool:
    connection = _connect()
    try:
        connection.execute("SELECT 1").fetchone()
        return True
    except sqlite3.Error:
        return False
    finally:
        connection.close()

How Does the Readiness Check Work?

  • The SELECT 1 query proves that SQLite accepts a query through the configured connection.
  • The except sqlite3.Error branch converts a database failure into a simple false result.
  • The finally block closes the connection after either outcome.
  • Save database.py.
  • Check that the inherited inventory behavior still passes by running:
python -m pytest

You should see every inherited test pass. This confirms that the readiness helper preserves the existing inventory behavior.

Seeing a Database Test Failure?

Check that database_is_ready() sits outside initialize_database(). Confirm that the finally block aligns with try and except.

If the failure remains, ask for help comparing the traceback with your readiness function.

✔️ Awesome, I've got everything!

Your Prometheus dependency is installed. Your database module now includes a readiness check that closes every connection.

ⓧ I'd like to double check the full code

Compare your requirements.txt file with this complete version.

fastapi[standard]==0.141.1
httpx==0.28.1
pydantic==2.13.5
prometheus-client==0.26.0
pytest==9.1.1

Compare your database.py file with this complete version.

import sqlite3

from models import InventoryEvent

DB_PATH = "inventory.db"


def _connect() -> sqlite3.Connection:
    return sqlite3.connect(DB_PATH)


def initialize_database() -> None:
    connection = _connect()
    try:
        connection.execute(
            """
            CREATE TABLE IF NOT EXISTS inventory (
                store_id TEXT NOT NULL,
                sku TEXT NOT NULL,
                quantity INTEGER NOT NULL DEFAULT 0,
                PRIMARY KEY (store_id, sku)
            )
            """
        )
        connection.execute(
            """
            CREATE TABLE IF NOT EXISTS processed_events (
                event_id TEXT PRIMARY KEY
            )
            """
        )
        connection.commit()
    finally:
        connection.close()


def database_is_ready() -> bool:
    connection = _connect()
    try:
        connection.execute("SELECT 1").fetchone()
        return True
    except sqlite3.Error:
        return False
    finally:
        connection.close()


def _read_quantity(
    connection: sqlite3.Connection,
    store_id: str,
    sku: str,
) -> int:
    row = connection.execute(
        "SELECT quantity FROM inventory WHERE store_id = ? AND sku = ?",
        (store_id, sku),
    ).fetchone()
    return int(row[0]) if row else 0


def apply_event(event: InventoryEvent) -> tuple[bool, int]:
    connection = _connect()
    try:
        try:
            connection.execute(
                "INSERT INTO processed_events (event_id) VALUES (?)",
                (event.event_id,),
            )
        except sqlite3.IntegrityError:
            connection.rollback()
            quantity = _read_quantity(connection, event.store_id, event.sku)
            return False, quantity

        current_quantity = _read_quantity(connection, event.store_id, event.sku)
        new_quantity = current_quantity + event.quantity_change

        if current_quantity == 0:
            existing_row = connection.execute(
                "SELECT 1 FROM inventory WHERE store_id = ? AND sku = ?",
                (event.store_id, event.sku),
            ).fetchone()
        else:
            existing_row = True

        if existing_row:
            connection.execute(
                """
                UPDATE inventory
                SET quantity = ?
                WHERE store_id = ? AND sku = ?
                """,
                (new_quantity, event.store_id, event.sku),
            )
        else:
            connection.execute(
                """
                INSERT INTO inventory (store_id, sku, quantity)
                VALUES (?, ?, ?)
                """,
                (event.store_id, event.sku, new_quantity),
            )

        connection.commit()
        return True, new_quantity
    except Exception:
        connection.rollback()
        raise
    finally:
        connection.close()


def get_quantity(store_id: str, sku: str) -> int:
    connection = _connect()
    try:
        return _read_quantity(connection, store_id, sku)
    finally:
        connection.close()
Instrument every request

Request middleware surrounds every route call. It can measure elapsed time, classify the response status, attach one correlation ID, and emit one consistent log record.

Why Use Bounded Metric Labels?

The request counters use only status_code as a label. Store IDs, SKUs, event IDs, and correlation IDs stay in logs because their many possible values would create excessive metric series.

  • Switch back to main.py.
  • Replace the import section above initialize_database() with this complete import group:
import json
import logging
from time import perf_counter
from uuid import uuid4

from fastapi import FastAPI, HTTPException, Request, Response
from prometheus_client import (
    CONTENT_TYPE_LATEST,
    Counter,
    Histogram,
    generate_latest,
)

from database import (
    apply_event,
    database_is_ready,
    get_quantity,
    initialize_database,
)
from models import InventoryEvent, InventoryResult, InventorySnapshot

What Do These Imports Support?

  • The timing and UUID imports measure each request and create its correlation ID.
  • The logging and JSON imports produce machine-readable request records.
  • The Prometheus imports define metrics and render their current values.
  • Locate the app = FastAPI(...) declaration in main.py.
  • Add the following logger configuration immediately below that declaration:
LOGGER = logging.getLogger("inventory_api")
LOGGER.setLevel(logging.INFO)
if not LOGGER.handlers:
    handler = logging.StreamHandler()
    handler.setFormatter(logging.Formatter("%(message)s"))
    LOGGER.addHandler(handler)
LOGGER.propagate = False

What Does the Logger Setup Do?

The named logger writes each prepared JSON string directly to the container output. The handler check prevents duplicate handlers when the application module loads more than once.

  • Save main.py.
  • Confirm that the import and logger setup preserve the test baseline by running:
python -m pytest

You should see the inherited tests pass again. The application now loads the new dependency and logger without changing its existing routes.

  • Add the following metric definitions below LOGGER.propagate = False:
HTTP_REQUESTS = Counter(
    "inventory_http_requests_total",
    "Total HTTP requests handled by the inventory API.",
    ["status_code"],
)
HTTP_ERRORS = Counter(
    "inventory_http_errors_total",
    "Total HTTP responses with status codes of 400 or greater.",
    ["status_code"],
)
HTTP_REQUEST_DURATION = Histogram(
    "inventory_http_request_duration_seconds",
    "Inventory API request duration in seconds.",
)
DUPLICATE_EVENTS = Counter(
    "inventory_duplicate_events_total",
    "Total inventory event deliveries ignored as duplicates.",
)

What Do the Metrics Measure?

  • The two counters track total requests and error responses by bounded status code.
  • The histogram records request durations in seconds for later latency calculations.
  • The duplicate counter records replayed inventory events that the API ignores.
  • Save main.py.
  • Confirm that the registered metrics load without an error by running:
python -m pytest

The inherited tests should remain green. The metric collectors are now registered when the application starts.

The middleware is one function shown in two pieces for readability. Add both pieces before saving main.py.

  • Add the first part of observe_request() below the metric definitions:
@app.middleware("http")
async def observe_request(request: Request, call_next):
    correlation_id = str(uuid4())
    started = perf_counter()

    try:
        response = await call_next(request)
    except Exception:
        duration_seconds = perf_counter() - started
        HTTP_REQUESTS.labels(status_code="500").inc()
        HTTP_ERRORS.labels(status_code="500").inc()
        HTTP_REQUEST_DURATION.observe(duration_seconds)
        LOGGER.exception(
            json.dumps(
                {
                    "event": "http_request",
                    "correlation_id": correlation_id,
                    "status_code": 500,
                    "duration_ms": round(duration_seconds * 1000, 2),
                }
            )
        )
        raise

How Does the Failure Path Work?

The middleware creates the correlation ID before calling the route. If an unexpected exception escapes, it records a request, an error, the duration, and a structured log before re-raising the exception.

  • Continue the same observe_request() function with this remaining code:
    duration_seconds = perf_counter() - started
    status_code = str(response.status_code)
    HTTP_REQUESTS.labels(status_code=status_code).inc()
    HTTP_REQUEST_DURATION.observe(duration_seconds)

    if response.status_code >= 400:
        HTTP_ERRORS.labels(status_code=status_code).inc()

    response.headers["X-Correlation-ID"] = correlation_id
    log_message = json.dumps(
        {
            "event": "http_request",
            "correlation_id": correlation_id,
            "status_code": response.status_code,
            "duration_ms": round(duration_seconds * 1000, 2),
        }
    )
    if response.status_code >= 400:
        LOGGER.warning(log_message)
    else:
        LOGGER.info(log_message)

    return response

How Does the Response Path Work?

Successful route calls update the request count and latency histogram. Responses with status codes of 400 or greater also update the error counter.

The same UUID goes into the X-Correlation-ID response header and the structured request log. That shared value connects a client-visible failure with its server-side evidence.

  • Save main.py.
  • Rebuild the API service with the completed middleware by running:
docker compose up --build -d

The rebuild can take a little while because the image installs the new dependency. When it finishes, the API service runs in the background on the same loopback port.

API Rebuild Failing?

Check the terminal output for a Python indentation error near observe_request(). Both middleware pieces must remain inside the same function.

If the image build cannot install the dependency, confirm that prometheus-client==0.26.0 appears once in requirements.txt.

If the rebuild still fails, ask for help with the Docker Compose output.

  • Open http://127.0.0.1:8000/health in your browser to send one request through the middleware.

Before you inspect the logs, what evidence do you expect that health request to leave behind?

  • Inspect the API service output by running:
docker compose logs api

You should see a JSON request log containing event, correlation_id, status_code, and duration_ms. The health request now leaves structured operational evidence.

Expose and test the signals

The collectors now hold request measurements inside the API process. Two routes make those signals observable to health checks, tests, and the monitoring stack you build next.

  • Locate the existing /health route in main.py.
  • Add the readiness and metrics routes below health():
@app.get("/ready")
def ready() -> dict[str, str]:
    if not database_is_ready():
        raise HTTPException(status_code=503, detail="Database is not ready.")
    return {"status": "ready"}


@app.get("/metrics")
def metrics() -> Response:
    return Response(content=generate_latest(), media_type=CONTENT_TYPE_LATEST)

What Do the New Routes Expose?

The /ready route returns a ready result only when SQLite accepts the check query. A failed check becomes an HTTP 503 response.

The /metrics route returns the current Prometheus exposition. This includes the request counters and latency histogram.

  • Locate applied, quantity = apply_event(event) inside receive_inventory_event().
  • Add the duplicate counter branch immediately below that line:
    if not applied:
        DUPLICATE_EVENTS.inc()

Why Count Duplicate Deliveries?

The application already ignores a replayed event safely. Incrementing DUPLICATE_EVENTS makes that behavior measurable without adding event IDs or SKUs to metric labels.

  • Save main.py.
  • Rebuild the API with the new routes by running:
docker compose up --build -d
  • Open http://127.0.0.1:8000/ready in your browser.

You should see {"status":"ready"}. This proves that the running container can query its SQLite database.

  • Open http://127.0.0.1:8000/metrics in another browser tab.

You should see Prometheus text containing inventory_http_requests_total and inventory_http_request_duration_seconds. The API now exposes the signals that raw container output could not calculate.

Readiness or Metrics Route Missing?

Confirm that both route functions sit below health() and above receive_inventory_event(). Save main.py before rebuilding the image again.

If the routes still fail, ask for help checking the container output and route definitions.

  • Switch back to tests/test_main.py.
  • Add this observability test below the inherited tests:
def test_observability_endpoints(client: TestClient) -> None:
    health_response = client.get("/health")
    ready_response = client.get("/ready")
    metrics_response = client.get("/metrics")

    assert "X-Correlation-ID" in health_response.headers
    assert ready_response.status_code == 200
    assert ready_response.json() == {"status": "ready"}
    assert "inventory_http_requests_total" in metrics_response.text
    assert "inventory_http_request_duration_seconds" in metrics_response.text

What Does the Test Prove?

The test confirms that middleware adds an X-Correlation-ID header to a normal response. It also checks the ready result and the two core metric names.

  • Save tests/test_main.py.

Before you run the complete suite, do you expect the inherited inventory behavior and the new observability checks to pass together?

  • Run the complete test suite with:
python -m pytest

You should see all five tests pass. The result proves that readiness, metrics, and correlation headers work without breaking the inherited API behavior.

Observability Test Failing?

A missing header usually means observe_request() does not assign response.headers["X-Correlation-ID"] before returning the response.

A missing metric name usually means the collector name differs from the test or the /metrics route does not call generate_latest().

If the failure remains, ask for help tracing the failing assertion.

✔️ Awesome, I've got everything!

Great work. Save every edited file and confirm that the five-test suite passes before continuing.

ⓧ I'd like to double check the full code

Compare your main.py file with this complete version.

import json
import logging
from time import perf_counter
from uuid import uuid4

from fastapi import FastAPI, HTTPException, Request, Response
from prometheus_client import (
    CONTENT_TYPE_LATEST,
    Counter,
    Histogram,
    generate_latest,
)

from database import (
    apply_event,
    database_is_ready,
    get_quantity,
    initialize_database,
)
from models import InventoryEvent, InventoryResult, InventorySnapshot

initialize_database()

app = FastAPI(title="Reliable Inventory Sync API", version="1.0.0")

LOGGER = logging.getLogger("inventory_api")
LOGGER.setLevel(logging.INFO)
if not LOGGER.handlers:
    handler = logging.StreamHandler()
    handler.setFormatter(logging.Formatter("%(message)s"))
    LOGGER.addHandler(handler)
LOGGER.propagate = False

HTTP_REQUESTS = Counter(
    "inventory_http_requests_total",
    "Total HTTP requests handled by the inventory API.",
    ["status_code"],
)
HTTP_ERRORS = Counter(
    "inventory_http_errors_total",
    "Total HTTP responses with status codes of 400 or greater.",
    ["status_code"],
)
HTTP_REQUEST_DURATION = Histogram(
    "inventory_http_request_duration_seconds",
    "Inventory API request duration in seconds.",
)
DUPLICATE_EVENTS = Counter(
    "inventory_duplicate_events_total",
    "Total inventory event deliveries ignored as duplicates.",
)


@app.middleware("http")
async def observe_request(request: Request, call_next):
    correlation_id = str(uuid4())
    started = perf_counter()

    try:
        response = await call_next(request)
    except Exception:
        duration_seconds = perf_counter() - started
        HTTP_REQUESTS.labels(status_code="500").inc()
        HTTP_ERRORS.labels(status_code="500").inc()
        HTTP_REQUEST_DURATION.observe(duration_seconds)
        LOGGER.exception(
            json.dumps(
                {
                    "event": "http_request",
                    "correlation_id": correlation_id,
                    "status_code": 500,
                    "duration_ms": round(duration_seconds * 1000, 2),
                }
            )
        )
        raise

    duration_seconds = perf_counter() - started
    status_code = str(response.status_code)
    HTTP_REQUESTS.labels(status_code=status_code).inc()
    HTTP_REQUEST_DURATION.observe(duration_seconds)

    if response.status_code >= 400:
        HTTP_ERRORS.labels(status_code=status_code).inc()

    response.headers["X-Correlation-ID"] = correlation_id
    log_message = json.dumps(
        {
            "event": "http_request",
            "correlation_id": correlation_id,
            "status_code": response.status_code,
            "duration_ms": round(duration_seconds * 1000, 2),
        }
    )
    if response.status_code >= 400:
        LOGGER.warning(log_message)
    else:
        LOGGER.info(log_message)

    return response


@app.get("/health")
def health() -> dict[str, str]:
    return {"status": "ok"}


@app.get("/ready")
def ready() -> dict[str, str]:
    if not database_is_ready():
        raise HTTPException(status_code=503, detail="Database is not ready.")
    return {"status": "ready"}


@app.get("/metrics")
def metrics() -> Response:
    return Response(content=generate_latest(), media_type=CONTENT_TYPE_LATEST)


@app.post("/inventory/events", response_model=InventoryResult)
def receive_inventory_event(event: InventoryEvent) -> InventoryResult:
    applied, quantity = apply_event(event)
    if not applied:
        DUPLICATE_EVENTS.inc()
    return InventoryResult(
        event_id=event.event_id,
        store_id=event.store_id,
        sku=event.sku,
        applied=applied,
        quantity=quantity,
    )


@app.get(
    "/inventory/{store_id}/{sku}",
    response_model=InventorySnapshot,
)
def read_inventory(store_id: str, sku: str) -> InventorySnapshot:
    return InventorySnapshot(
        store_id=store_id,
        sku=sku,
        quantity=get_quantity(store_id, sku),
    )

Compare your tests/test_main.py file with this complete version.

import pytest
from fastapi.testclient import TestClient

import database
from main import app


@pytest.fixture
def client(tmp_path, monkeypatch) -> TestClient:
    test_database = tmp_path / "inventory.db"
    monkeypatch.setattr(database, "DB_PATH", str(test_database))
    database.initialize_database()
    return TestClient(app)


def test_health(client: TestClient) -> None:
    response = client.get("/health")

    assert response.status_code == 200
    assert response.json() == {"status": "ok"}


def test_event_updates_inventory(client: TestClient) -> None:
    response = client.post(
        "/inventory/events",
        json={
            "event_id": "event-1001",
            "store_id": "store-101",
            "sku": "coffee-beans",
            "quantity_change": 5,
        },
    )

    assert response.status_code == 200
    assert response.json()["applied"] is True
    assert response.json()["quantity"] == 5

    snapshot = client.get("/inventory/store-101/coffee-beans")
    assert snapshot.json()["quantity"] == 5


def test_duplicate_event_is_ignored(client: TestClient) -> None:
    event = {
        "event_id": "event-2001",
        "store_id": "store-202",
        "sku": "oat-milk",
        "quantity_change": 4,
    }

    first_response = client.post("/inventory/events", json=event)
    second_response = client.post("/inventory/events", json=event)

    assert first_response.json()["applied"] is True
    assert second_response.json()["applied"] is False
    assert second_response.json()["quantity"] == 4


def test_invalid_quantity_is_rejected(client: TestClient) -> None:
    response = client.post(
        "/inventory/events",
        json={
            "event_id": "event-3001",
            "store_id": "store-303",
            "sku": "vanilla-syrup",
            "quantity_change": "five",
        },
    )

    assert response.status_code == 422


def test_observability_endpoints(client: TestClient) -> None:
    health_response = client.get("/health")
    ready_response = client.get("/ready")
    metrics_response = client.get("/metrics")

    assert "X-Correlation-ID" in health_response.headers
    assert ready_response.status_code == 200
    assert ready_response.json() == {"status": "ready"}
    assert "inventory_http_requests_total" in metrics_response.text
    assert "inventory_http_request_duration_seconds" in metrics_response.text

That visibility gap is closed. Next, you will connect these application signals to Prometheus and turn them into a Grafana control room.

Build the Reliability Control Room

Your API now emits structured logs, correlation IDs, readiness status, and application metrics. Those signals give you the evidence needed to understand its behavior.

Raw metric text becomes difficult to interpret once requests start arriving. This step turns those measurements into operational trends that you can monitor from one screen.

Your existing Docker Compose stack will run Prometheus beside the API. Grafana will display the collected measurements as four dashboard panels.

In this step, get ready to:
  • Configure Prometheus to collect metrics from the API.
  • Connect Grafana to the Prometheus service.
  • Build a dashboard for four reliability signals.
Connect Prometheus to the API

Prometheus collects measurements by requesting the API's /metrics endpoint at a regular interval. The Compose service name gives it a stable address for each scrape.

  • Select the reliable-inventory-sync-api folder in the Explorer sidebar in Visual Studio Code.
  • Click the new file icon in the Explorer sidebar.
  • Enter prometheus/prometheus.yml as the file path.
  • Add the scrape configuration by pasting this code:
global:
  scrape_interval: 5s
  evaluation_interval: 5s

scrape_configs:
  - job_name: inventory-api
    static_configs:
      - targets: ["api:8000"]

What Does This Configuration Do?

  • The global interval asks Prometheus to collect new measurements every five seconds.
  • The inventory-api job gives this scrape target a recognizable name.
  • The api:8000 target reaches the API through its Compose service hostname.
  • Save prometheus/prometheus.yml.

✔️ Awesome, I've got everything!

Great. Confirm that prometheus/prometheus.yml is saved inside the prometheus folder.

ⓧ I'd like to double check the full code

Compare your saved prometheus/prometheus.yml with this complete file.

global:
  scrape_interval: 5s
  evaluation_interval: 5s

scrape_configs:
  - job_name: inventory-api
    static_configs:
      - targets: ["api:8000"]

The Compose file now needs two monitoring services. Named volumes preserve Prometheus measurements and the Grafana dashboard across normal container restarts.

  • Switch back to compose.yaml from earlier.
  • Replace its current contents with this complete stack:
services:
  api:
    build: .
    ports:
      - "127.0.0.1:8000:8000"

  prometheus:
    image: prom/prometheus:v3.14.0
    ports:
      - "127.0.0.1:9090:9090"
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus
    depends_on:
      api:
        condition: service_started

  grafana:
    image: grafana/grafana:13.2.2
    ports:
      - "127.0.0.1:3000:3000"
    volumes:
      - grafana-data:/var/lib/grafana
    depends_on:
      prometheus:
        condition: service_started

volumes:
  prometheus-data:
  grafana-data:

How Does the Stack Fit Together?

  • The api service keeps the instrumented application available on loopback port 8000.
  • The prom/prometheus:v3.14.0 image loads your scrape configuration and stores collected measurements in prometheus-data.
  • The grafana/grafana:13.2.2 image stores dashboard data in grafana-data.
  • Each startup dependency establishes the required service order.
  • Save compose.yaml.

✔️ Awesome, I've got everything!

Great. Confirm that compose.yaml is saved before rebuilding the stack.

ⓧ I'd like to double check the full code

Compare your saved compose.yaml with this complete file.

services:
  api:
    build: .
    ports:
      - "127.0.0.1:8000:8000"

  prometheus:
    image: prom/prometheus:v3.14.0
    ports:
      - "127.0.0.1:9090:9090"
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus
    depends_on:
      api:
        condition: service_started

  grafana:
    image: grafana/grafana:13.2.2
    ports:
      - "127.0.0.1:3000:3000"
    volumes:
      - grafana-data:/var/lib/grafana
    depends_on:
      prometheus:
        condition: service_started

volumes:
  prometheus-data:
  grafana-data:

The first start can pause while Docker downloads the two monitoring images. That download activity is expected.

  • Rebuild the API and start the complete stack by running this command:
docker compose up --build -d

What Does This Command Do?

The command rebuilds the instrumented API image. It starts the API, Prometheus, and Grafana as background services.

Before you check, predict the three service names you expect to see.

  • Check the service status by running this command:
docker compose ps

What Should You See?

You should see the api, prometheus, and grafana services running. Their published loopback ports should be 8000, 9090, and 3000.

  • Open http://127.0.0.1:9090/targets in your browser.
  • Confirm that the inventory-api target is up.

Prometheus can now collect your API metrics every five seconds. Your measurements have moved from a single endpoint into a time-series store.

Target Still Down?

Confirm that prometheus/prometheus.yml uses the target api:8000. Check that the api service is running in Docker Desktop.

Verify that the configuration mount in compose.yaml points to ./prometheus/prometheus.yml.

Help me diagnose a Prometheus target that is down.

Connect Grafana to Prometheus

Grafana queries Prometheus to turn stored measurements into charts. Its first sign-in uses a local administrator account that immediately prompts you to replace the default password.

The new password protects only this local Grafana instance. Keep it private because the dashboard account has administrative access.

  • Open http://localhost:3000 in your browser.
  • Enter admin in the username field.
  • Enter admin in the password field.
  • Click Sign in.
  • Click OK on the password-change prompt.
  • Enter a private password in the password-change form.
  • Submit the password-change form.

That first-login task is complete. Your Grafana administrator account now uses a password that only you know.

Grafana and Prometheus run in separate containers. The Compose hostname gives Grafana a route to the Prometheus service inside their shared network.

  • Click Connections in the left-side menu.
  • Click Add new connection.
  • Enter Prometheus in the search bar.
  • Click Prometheus data source.
  • Click Add new data source.
  • Enter http://prometheus:9090 in the Prometheus server URL field.
  • Click Save & test.

You should see Successfully queried the Prometheus API. Your visualization layer can now read the measurements collected from the API.

Why Use the Service Hostname?

The hostname prometheus resolves to the Prometheus container inside the Compose network. The URL http://prometheus:9090 therefore reaches the correct service from Grafana.

🙋‍♀️ Data Source Test Failing?

Confirm that the server URL is exactly http://prometheus:9090. Check that the Prometheus target page still shows inventory-api as up.

Help me troubleshoot the Grafana Prometheus connection.

Build the reliability dashboard

Each dashboard panel answers a different production-support question. The four PromQL expressions cover traffic, failures, latency, and duplicate deliveries.

The first panel establishes the workflow you will repeat. Each later panel changes only its title and query.

  • Click Dashboards in the main menu.
  • Click New.
  • Select New Dashboard.
  • Click the Add new element icon.
  • Select a panel element.
  • Select Configure visualization.
  • Select Prometheus from the Data source list.
  • Switch the query editor to Code.
  • Set the panel title to Request Rate.
  • Add the request-rate query by pasting this expression:
sum(rate(inventory_http_requests_total[1m]))

How Does Request Rate Work?

The request counter increases whenever the API handles a request. The rate() expression calculates its recent per-second increase across the one-minute window.

  • Click Refresh to query the data source.

You should see recent API traffic in the panel preview. Prometheus scrapes also contribute requests to the instrumented API.

Request Panel Showing No Data?

Confirm that the selected data source is Prometheus. Compare the query with the metric name exposed by /metrics.

Help me fix the request-rate panel.

  • Click Back to return to the dashboard.
  • Click the Add new element icon.
  • Select a panel element.
  • Select Configure visualization.
  • Select Prometheus from the Data source list.
  • Switch the query editor to Code.
  • Set the panel title to Error Rate.
  • Add the error-rate query by pasting this expression:
sum(rate(inventory_http_errors_total[1m]))

How Does Error Rate Work?

This expression calculates the recent rate of responses whose status codes are 400 or greater. A healthy baseline stays at zero until failing traffic arrives.

  • Click Refresh to preview the error-rate panel.

You should see the healthy error-rate baseline at zero. This panel becomes the first visible symptom during the controlled incident.

Error Query Rejected?

Confirm that the expression uses inventory_http_errors_total with a one-minute range. Check the matching metric name on /metrics.

Help me fix the error-rate panel.

  • Click Back to return to the dashboard.
  • Click the Add new element icon.
  • Select a panel element.
  • Select Configure visualization.
  • Select Prometheus from the Data source list.
  • Switch the query editor to Code.
  • Set the panel title to P95 Request Latency.
  • Add the p95 latency query by pasting this expression:
histogram_quantile(0.95, sum by (le) (rate(inventory_http_request_duration_seconds_bucket[1m])))

How Does P95 Latency Work?

The histogram stores request durations in bounded buckets. histogram_quantile() estimates the duration that covers 95 percent of recent requests.

  • Click Refresh to preview the latency panel.

You should see a latency value based on the API requests collected during the latest one-minute window.

Latency Panel Empty?

Confirm that the query ends with the histogram bucket metric inventory_http_request_duration_seconds_bucket. Generate another API request if the selected time range contains no recent traffic.

Help me fix the p95 latency panel.

  • Click Back to return to the dashboard.
  • Click the Add new element icon.
  • Select a panel element.
  • Select Configure visualization.
  • Select Prometheus from the Data source list.
  • Switch the query editor to Code.
  • Set the panel title to Duplicate Events.
  • Add the duplicate-event query by pasting this expression:
increase(inventory_duplicate_events_total[5m])

How Does the Duplicate Panel Work?

The duplicate-event counter increases when apply_event() returns applied as false. The query shows the increase recorded during the latest five-minute window.

  • Click Refresh to preview the duplicate-events panel.

The panel is ready to record duplicate deliveries. Replaying an event with the same event_id gives it a visible value.

Duplicate Panel Showing No Value?

Confirm that the query uses inventory_duplicate_events_total with the five-minute window. The counter needs one replayed event before it increases.

Help me fix the duplicate-events panel.

  • Click Back to return to the dashboard.
  • Click Save.
  • Enter Inventory API Reliability as the dashboard title.
  • Click Save to store the dashboard.
  • Click Exit edit.

Before you refresh, predict which panels should react after one accepted event plus one duplicate delivery.

  • Return to http://127.0.0.1:8000/docs from earlier.
  • Submit one valid inventory event through the /inventory/events route.
  • Confirm that the first response contains applied: true.
  • Replay the same event_id through the same route.
  • Confirm that the replayed response contains applied: false.
  • Return to the Inventory API Reliability dashboard.
  • Click Refresh.

You should see traffic in the request-rate panel. The p95 latency panel should contain a recent duration.

The duplicate-events panel should show an increase. The error-rate panel should remain at its healthy baseline.

Your local control room is working. It now separates normal traffic, slow requests, failed requests, and duplicate deliveries into four visible signals.

Your reliability control room now turns API measurements into operational evidence. Next, you will use those signals to investigate a controlled validation incident.

Investigate and Recover the Incident

Your Grafana dashboard already turns the API's metrics into operational trends. Docker Compose keeps the complete reliability lab running on your machine.

A dashboard proves its value when it guides you from a visible symptom to a defensible root-cause analysis. You will connect the error signal to structured logging through correlation IDs. You will then verify recovery through live evidence.

In this step, get ready to:
  • Generate a controlled validation incident with malformed inventory events.
  • Trace one failed request from its correlation ID to its structured API log.
  • Recover normal traffic and document the incident response.
Trigger the controlled incident

The incident generator uses HTTPX to send 30 requests that violate the inventory event contract. Every request reaches the running API with a text value where an integer is required.

  • Use the new-file control in the Visual Studio Code Explorer sidebar to create incident.py inside reliable-inventory-sync-api.
  • Paste this incident generator into incident.py:
import time

import httpx

URL = "http://127.0.0.1:8000/inventory/events"

for index in range(30):
    response = httpx.post(
        URL,
        json={
            "event_id": f"incident-{index:03d}",
            "store_id": "store-incident",
            "sku": "coffee-beans",
            "quantity_change": "five",
        },
        timeout=5.0,
    )
    correlation_id = response.headers.get("X-Correlation-ID")
    print(
        f"request={index + 1:02d} "
        f"status={response.status_code} "
        f"correlation_id={correlation_id}"
    )
    time.sleep(0.2)

What does this code do?

  • The URL value points every request at the live inventory event endpoint.
  • The loop creates event IDs from incident-000 through incident-029.
  • The text value "five" violates the integer requirement for quantity_change.
  • Each response exposes its status plus its X-Correlation-ID value.
  • The short pause spreads the requests across several Prometheus scrapes.
  • Save incident.py.
  • Check the Explorer sidebar for incident.py inside reliable-inventory-sync-api.

Cannot find the incident script?

Confirm that incident.py sits beside main.py in the repository. A file created inside docs or another folder runs from the wrong location.

Check that Visual Studio Code saved the file with the .py extension.

Help me check my incident script location.

✔️ Awesome, I've got everything!

Great. Confirm that incident.py is saved beside main.py.

ⓧ I'd like to double check the full code

import time

import httpx

URL = "http://127.0.0.1:8000/inventory/events"

for index in range(30):
    response = httpx.post(
        URL,
        json={
            "event_id": f"incident-{index:03d}",
            "store_id": "store-incident",
            "sku": "coffee-beans",
            "quantity_change": "five",
        },
        timeout=5.0,
    )
    correlation_id = response.headers.get("X-Correlation-ID")
    print(
        f"request={index + 1:02d} "
        f"status={response.status_code} "
        f"correlation_id={correlation_id}"
    )
    time.sleep(0.2)

What should I compare?

Check the request URL first. Then compare the loop range plus every payload field. Finish by checking the printed response fields plus the short pause.

Before you run this, predict what the error rate and readiness signal will do:

  • Generate the malformed traffic by running this command in the PowerShell terminal from earlier:
python incident.py

What does this command do?

Python runs the incident generator against the API that is already listening on loopback port 8000. The command returns after all 30 requests finish.

You should see 30 numbered lines. Every line should show status 422 plus a different correlation ID.

You have reproduced a customer-impacting validation incident without damaging inventory data. The printed IDs give you a direct route into the server-side evidence.

  • Record one of the final ten printed correlation IDs here: your printed correlation ID.
  • Return to the Inventory API Reliability dashboard from earlier.
  • Refresh the browser page.
  • Check the error-rate panel for a visible rise.
  • Visit http://127.0.0.1:8000/ready in another browser tab.

You should see the error-rate panel rise after the burst of rejected requests.

The readiness tab should still show {"status":"ready"}. The database remained reachable throughout the validation failures.

Incident results missing?

Confirm that the API container is still running in Docker Desktop. The script needs the inventory endpoint on loopback port 8000.

Check that the PowerShell terminal is using the repository's virtual environment. HTTPX is installed through the existing requirements.txt file.

Help me diagnose the incident generator.

Correlate the metric with the logs

The dashboard reveals the size plus timing of the symptom. The saved correlation ID isolates one rejected request in the API's container output.

Matching a long UUID in container output can feel fiddly. Using one of the final printed IDs keeps the relevant request inside the last 40 log lines.

Before you inspect the logs, predict whether one client ID can isolate a single failed request:

  • Display the API's latest container output by running this command:
docker compose logs --tail 40 api

What does this command do?

Docker Compose reads the latest output from the api service. The tail limit keeps the investigation focused on the incident you just generated.

  • Find your printed correlation ID in the displayed output.
  • Check the matching JSON log for a status_code value of 422.
  • Compare that failed-request log with the healthy readiness response from the previous substep.

The matching log connects one client-visible response to one server-side request record. Its 422 status confirms that the API rejected the payload.

What does the evidence prove?

The rising error counter proves that many requests failed during the same window. The matching log proves that the sampled request received HTTP 422.

The readiness response proves that SQLite still accepted its readiness check. The root cause is malformed client data. API plus database availability remained healthy.

The text value "five" violates the integer contract for quantity_change. FastAPI rejects that payload before inventory storage changes.

🙋‍♀️ Cannot find the correlation ID?

Use an ID from one of the final ten incident requests. Earlier IDs may sit outside the 40-line log window.

Compare the complete UUID. A partial visual match can point to the wrong request.

Help me correlate the failed request.

Recover and document the incident

The returned PowerShell prompt confirms that the malformed traffic has stopped. A valid request now tests the normal path while readiness confirms continued database access.

  • Return to http://127.0.0.1:8000/docs from earlier.
  • Select the POST /inventory/events operation.
  • Enable the request body editor.
  • Replace the request body with this valid inventory event:
{
            "event_id": "event-1001",
            "store_id": "store-101",
            "sku": "coffee-beans",
            "quantity_change": 5
        }

Why is this request valid?

The value for quantity_change is the integer 5. This matches the inventory event contract.

  • Send the request through the interactive API page.
  • Check the response for HTTP status 200.

You should see a successful inventory response. The request may report an existing event as a duplicate if event-1001 was used earlier.

Valid request still rejected?

Check that quantity_change contains the number 5 without quotation marks. A quoted value repeats the malformed payload pattern.

Confirm that every required field remains in the request body.

Help me fix the recovery request.

The error-rate query uses a one-minute window. Give that window time to clear so the dashboard can show the recovered state.

Before you check recovery, predict whether readiness and error rate will recover on the same schedule:

  • Wait one minute for the error-rate window to clear.
  • Refresh the Inventory API Reliability dashboard.
  • Check the error-rate panel for a return to zero.
  • Return to http://127.0.0.1:8000/ready.

You should see the Grafana error rate return to zero after the malformed traffic leaves the one-minute window.

You should still see {"status":"ready"} from the readiness endpoint. The service stayed available before recovery plus after recovery.

The incident report turns these observations into evidence another on-call engineer can follow. Its Markdown structure separates impact, diagnosis, recovery, prevention, plus the reusable runbook.

  • Use the new-folder control in Visual Studio Code's Explorer sidebar to create docs inside reliable-inventory-sync-api.

You should see the new docs folder in the repository tree.

  • Use the new-file control inside docs to create incident-report.md.
  • Paste the incident summary plus evidence template into docs/incident-report.md:
# Inventory API Validation Incident

## Impact

Thirty inventory updates were rejected. The service remained available and its database remained ready.

## Detection

The Grafana error-rate panel rose while the readiness endpoint stayed healthy.

## Timeline

- Incident started:
- Error spike detected:
- Correlation ID investigated:
- Valid traffic restored:
- Error rate returned to zero:

## Evidence

- Dashboard signal:
- HTTP status:
- Correlation ID:
- Matching structured log:
- Readiness result:

Why record the evidence this way?

Impact states what customers experienced. Detection records the first operational signal that revealed the problem.

The timeline preserves the investigation order. The evidence section connects the dashboard symptom to the HTTP response, correlation ID, structured log, plus readiness result.

  • Save docs/incident-report.md.
  • Use Visual Studio Code's Markdown preview control to inspect the report.
  • Check the preview for the Impact, Detection, Timeline, plus Evidence headings.

Markdown headings missing?

Confirm that the file is named incident-report.md. Another extension prevents Visual Studio Code from treating the file as Markdown.

Check that every section heading starts with two hash characters plus a space.

Help me fix the report structure.

  • Return to the editor at the bottom of docs/incident-report.md.
  • Paste the diagnosis, recovery, prevention, plus runbook sections below the evidence template:
## Root cause

The client sent the text value `"five"` for `quantity_change`, but the API contract requires an integer. FastAPI rejected the request with HTTP 422 before inventory storage changed.

## Recovery

The malformed traffic stopped. A valid inventory event succeeded, readiness stayed healthy, and the one-minute error-rate window returned to zero.

## Prevention

Validate the client payload against the API contract before deployment and monitor sustained validation-error rates.

## On-call runbook

1. Check `/ready` to separate availability failure from bad-request traffic.
2. Check request rate, error rate, and p95 latency on the reliability dashboard.
3. Copy a correlation ID from the failing client response.
4. Find the matching structured API log and confirm the status code.
5. Stop or correct the malformed client traffic.
6. Send one valid event and confirm the error-rate window returns to zero.

How does the report support recovery?

The root-cause section identifies the malformed field plus the contract mismatch. The recovery section records the evidence that normal traffic resumed.

The prevention step reduces the chance of another malformed deployment. The runbook gives the next responder a repeatable investigation order.

  • Save docs/incident-report.md.
  • Refresh the Markdown preview.
  • Check the preview for the root cause, recovery, prevention, plus six-step on-call runbook.

Runbook missing from the preview?

Confirm that the new content sits below the final Readiness result: line. Content pasted into another file does not appear in this report's preview.

Check the numbered runbook for all six entries.

Help me complete the incident report.

✔️ Awesome, I've got everything!

Great. Confirm that docs/incident-report.md contains every required section before adding your observation-specific details.

ⓧ I'd like to double check the full code

# Inventory API Validation Incident

## Impact

Thirty inventory updates were rejected. The service remained available and its database remained ready.

## Detection

The Grafana error-rate panel rose while the readiness endpoint stayed healthy.

## Timeline

- Incident started:
- Error spike detected:
- Correlation ID investigated:
- Valid traffic restored:
- Error rate returned to zero:

## Evidence

- Dashboard signal:
- HTTP status:
- Correlation ID:
- Matching structured log:
- Readiness result:

## Root cause

The client sent the text value `"five"` for `quantity_change`, but the API contract requires an integer. FastAPI rejected the request with HTTP 422 before inventory storage changed.

## Recovery

The malformed traffic stopped. A valid inventory event succeeded, readiness stayed healthy, and the one-minute error-rate window returned to zero.

## Prevention

Validate the client payload against the API contract before deployment and monitor sustained validation-error rates.

## On-call runbook

1. Check `/ready` to separate availability failure from bad-request traffic.
2. Check request rate, error rate, and p95 latency on the reliability dashboard.
3. Copy a correlation ID from the failing client response.
4. Find the matching structured API log and confirm the status code.
5. Stop or correct the malformed client traffic.
6. Send one valid event and confirm the error-rate window returns to zero.

What should I compare?

Compare every heading plus the fixed incident analysis. The timeline plus evidence entries stay open for the observations from your own run.

  • Complete the five Timeline entries with the order plus approximate times from your incident run.
  • Save docs/incident-report.md.
  • Check the Markdown preview for five completed timeline entries.
  • Complete the five Evidence entries with the dashboard spike, HTTP 422, your printed correlation ID, matching log, plus readiness response.
  • Save docs/incident-report.md.
  • Check the Markdown preview for completed evidence plus the documented root cause and recovery.

That closes the complete incident loop. You detected the symptom, traced one request, identified malformed client data, verified recovery, plus left a runbook for the next responder.

Secret mission

Add a Recovering Error-Rate Alert

Turn the recovered validation incident into an alert lifecycle you can prove. You will make InventoryApiHighErrorRate fire during malformed traffic. Then you will watch it return to inactive after recovery.

Clean Up Your Resources

Clean Up Your Resources

Every resource in this reliability lab runs locally, so there are no metered cloud resources or ongoing service costs. Choose whether to keep the lab running, pause its services, or remove it entirely.

Resources you used:

  • A local Docker Compose application running the api, Prometheus, and Grafana services.
  • Docker named volumes declared as prometheus-data and grafana-data. These hold metrics and dashboard data.
  • Local container images tagged prom/prometheus:v3.14.0 and grafana/grafana:13.2.2.
  • The local reliable-inventory-sync-api repository. It contains the complete project series history.

Keep everything running

No action is needed. Choose this option if you want the reliability lab ready for another incident exercise or demonstration.

  • Leave the api, Prometheus, and Grafana services running in Docker Desktop.
  • Keep the prometheus-data volume so your collected metrics remain available.
  • Keep the grafana-data volume so your Inventory API Reliability dashboard remains available.
  • Keep the reliable-inventory-sync-api repository as your incident investigation and recovery reference.

Pause - I'll come back to this later

Shut down the running services to free local resources while preserving the project files, Prometheus data, and Grafana dashboard.

  • Stop the Compose application from the reliable-inventory-sync-api repository by running this command:
docker compose down

What does this command do?

The command stops the API, Prometheus, and Grafana containers. It also removes the Compose network.

The named volumes remain intact. Your metrics and dashboard return when you start the lab again.

Your lab is safely paused. Docker Desktop no longer runs its three services, while your monitoring data remains ready for the next session.

Delete - I don't want to use this again

This option permanently removes the local reliability lab and its monitoring data. The repository also contains your Part 1 and Part 2 work, so delete that folder only when you no longer need the series history.

  • Stop the Compose application from the reliable-inventory-sync-api repository by running this command:
docker compose down

What does this command do?

The command stops and removes the three project containers. It leaves the named volumes available for separate deletion.

The named volumes contain your saved Prometheus metrics and Grafana dashboard. Remove both volumes to erase that monitoring data.

  • Switch back to Docker Desktop.
  • Locate the project volume whose name ends with prometheus-data.
  • Delete the confirmed Prometheus project volume using its available delete control.
  • Locate the project volume whose name ends with grafana-data.
  • Delete the confirmed Grafana project volume using its available delete control.
  • Confirm that neither project volume remains in Docker Desktop.

Match the Volume Names

Docker Desktop can add a project prefix to volume names. Match the endings to prometheus-data and grafana-data in compose.yaml before deleting anything.

The Prometheus and Grafana images use local disk space after the containers are removed. Delete both image tags to clear those remaining resources.

  • Locate prom/prometheus:v3.14.0 in the local image list in Docker Desktop.
  • Delete the Prometheus image using its available delete control.
  • Locate grafana/grafana:13.2.2 in the local image list in Docker Desktop.
  • Delete the Grafana image using its available delete control.
  • Confirm that neither image tag remains in the local image list.

The repository holds the API source, monitoring configuration, incident evidence, and recovery runbook. Removing it erases every local stage of this project series.

  • Leave the repository folder and remove it from its parent location by running these commands in PowerShell:
Set-Location ..
Remove-Item -Recurse -Force reliable-inventory-sync-api

What do these commands do?

The first line moves PowerShell to the folder above your repository. The second line permanently removes the entire reliable-inventory-sync-api folder.

Your Compose services, monitoring volumes, vendor images, and local repository are now removed.

Nice Work!

Nice Work!

You did it! Your local reliability lab now proves a complete incident-response lifecycle from detection through recovery.

You've learned how to:

  • Added readiness checks to the FastAPI inventory API. Connected each response to structured JSON logs through correlation IDs. Exposed Prometheus metrics for each operational KPI.
  • Ran the API with Prometheus plus Grafana in a Docker Compose reliability lab. Built an operational dashboard that makes request rate visible. The same dashboard shows error rate. It also shows p95 latency. Duplicate events complete the operational picture.
  • Triggered a controlled validation incident with malformed traffic. Detected the HTTP 422 spike in Grafana. Matched one correlation ID to a structured API log. Documented the root cause in docs/incident-report.md. Captured recovery evidence in the report. Added an on-call runbook for the next investigation.
  • Secret Mission: Added the InventoryApiHighErrorRate alert to Prometheus. Verified the full alert lifecycle from pending through firing. Confirmed the alert returned to inactive after recovery.

Ready to quiz yourself?