Debug a Payment Outage with MCP

Investigate a simulated payment outage with Claude Code and MCP tools.

Introduction

30 Second Summary

When a checkout outage begins, the loudest error message can make one explanation feel certain. A convincing guess can send an investigation in the wrong direction.

In this project, you will connect Claude Code to a local incident lab through MCP. You will reject a false lead with measured evidence before proving that configuration drift caused the failures.

What You'll Build

You will demo Claude investigating a simulated checkout outage until its root-cause report proves why the payment-timeout clue was misleading.

By the end of this project, you'll have:

  • A tool-driven investigation where Claude queries simulated logs, metrics, deployment history, runtime configuration, probes, and traces through a local server.
  • A timestamped evidence ledger that shows how each observation changes H1, H2, or H3.
  • Causal verification that shows checkout-03 fall from 58.3% to 0.0% while checkout-02 remains at 0.0% as the negative control.
  • Secret Mission: Correct a five-minute trace clock skew so the incident timeline reflects the true order of events.

Are there any prerequisites?

You need existing Claude Code access plus Python 3.10 or newer, Git, a code editor, and macOS Terminal. No MCP experience is required because the incident lab runs locally with the tools already on your Mac.

Before We Start

This checkpoint locks in the scope of your investigation before the first clue appears. You are investigating simulated checkout-api incident INC-2048 only through MCP tools until disciplined timestamped evidence verifies the root cause.

Set Up the Local MCP Incident Lab

Your investigation depends on evidence that Claude Code can query directly. A local MCP server provides that access through typed tools while keeping the entire incident lab on your Mac.

This step gives Claude the server, project-scoped connection, and investigation rules it needs. You will also isolate the lab's Python packages so they do not affect your other projects.

In this step, get ready to:
  • Verify that Python 3.10 or newer is available.
  • Create the incident lab files inside an isolated virtual environment.
  • Connect the project-scoped server to Claude Code.
Verify Python 3.10 or newer

The MCP Python SDK requires Python 3.10 or newer. Checking first prevents an older interpreter from causing confusing installation failures.

  • Press Cmd+Space to open Spotlight.
  • Type Terminal into Spotlight.
  • Press Enter to open macOS Terminal.
  • Check your installed Python version by running:
python3 --version

What does this check do?

The command asks the Python 3 executable to report its version. Use the first two numbers to decide which path below matches your Mac.

✔️ I see version 3.10 or higher

Your Python runtime meets the SDK requirement. That clears the first dependency for the local server.

  • Keep this Terminal window open for the project setup.

ⓧ I see an older version

Your current Python runtime is below the required version. Install the current stable macOS release before creating the environment.

  • Visit the official Python downloads page for macOS.
  • Download the current stable macOS installer.
  • Complete the installer using its default options.
  • Quit Terminal after the installation completes.
  • Reopen Terminal through Spotlight.
  • Check the new version by running:
python3 --version

What should I see?

You should now see Python 3.10 or newer. The first two numbers must meet or exceed that minimum.

ⓧ Command not found

Your shell cannot find a Python 3 installation. The official macOS installer adds the runtime needed for this lab.

  • Visit the official Python downloads page for macOS.
  • Download the current stable macOS installer.
  • Complete the installer using its default options.
  • Quit Terminal after the installation completes.
  • Reopen Terminal through Spotlight.
  • Confirm the installation by running:
python3 --version

What should I see?

You should see Python 3.10 or newer. That confirms Terminal can find the installed runtime.

Create the lab files and environment

A project-scoped server keeps its connection details beside the code that uses it. The isolated environment then gives the server a predictable SDK version.

  • Move to your Desktop by running this command:
cd ~/Desktop

What does this command do?

The command changes Terminal's current location to your Desktop. The lab folder will be easy to find there.

  • Create the checkout-incident-lab folder by running these commands:
mkdir checkout-incident-lab
cd checkout-incident-lab
pwd

What do these commands do?

  • The first command creates the checkout-incident-lab folder on your Desktop.
  • The second command moves Terminal into that folder.
  • The final command prints the current path so you can confirm your location.

Folder command failed?

Check whether a folder named checkout-incident-lab already exists on your Desktop. Remove the duplicate folder only if you no longer need its contents.

Ask for help: Help me create the checkout-incident-lab folder on my macOS Desktop and confirm Terminal is inside it.

  • Open the checkout-incident-lab folder from your Desktop in your code editor.
  • Create each project file by comparing your files with the complete references below.

✔️ Awesome, I've got everything!

Save all seven files in the checkout-incident-lab folder. Keep every filename exactly as shown.

ⓧ I'd like to double check the full code

Create each file directly inside checkout-incident-lab. Copy the matching content exactly.

mcp[cli]==2.1.0
.venv/
.incident-state.json
__pycache_/
{
  "mcpServers": {
    "checkout-incident": {
      "type": "stdio",
      "command": "${CLAUDE_PROJECT_DIR:-.}/.venv/bin/mcp",
      "args": [
        "run",
        "${CLAUDE_PROJECT_DIR:-.}/server.py:mcp"
      ]
    }
  }
}
# Checkout Incident Investigation Protocol

Investigate the checkout incident through the `checkout-incident` MCP tools.

## Evidence rules

1. Do not read or search `server.py` or `.incident-state.json` during the investigation. They are lab fixtures, not admissible incident evidence.
2. Record every observation in `evidence-ledger.md` with an evidence ID, timestamp, source, observation, interpretation, and hypothesis impact.
3. Keep observations separate from interpretations.
4. Maintain at least three competing hypotheses until evidence rejects or supports them.
5. Before any remediation, falsify the leading hypothesis and require two independent sources to support the proposed mechanism.
6. If the payment-provider mitigation produces no improvement, explicitly record H1 as rejected through the MCP decision tool before inspecting runtime configuration.
7. Apply the smallest targeted change. Do not change the application release or payment timeout when correcting instance configuration.
8. Verify with the same target probe used before the fix and with checkout-02 as a negative control.
9. Do not claim root cause from a successful fix alone. Connect the trigger, mechanism, symptom, and before-and-after evidence.
10. Leave the trace tools for the Secret Mission after the main incident is resolved.

## Required hypotheses

- H1: The payment provider is slower than the checkout timeout.
- H2: The latest application deployment introduced a checkout regression.
- H3: One checkout instance has configuration drift that exhausts its worker pool.
from __future__ import annotations

import json
import os
from pathlib import Path
from typing import Any

from mcp.server import MCPServer

mcp = MCPServer(
    "Checkout Incident Lab",
    instructions=(
        "Use these tools to investigate a simulated checkout incident. "
        "Treat tool results as evidence and preserve source timestamps."
    ),
)

PROJECT_ROOT = Path(
    os.environ.get("CLAUDE_PROJECT_DIR", Path(__file__).resolve().parent)
)
STATE_FILE = PROJECT_ROOT / ".incident-state.json"

DEFAULT_STATE: dict[str, Any] = {
    "payment_timeout_ms": 2000,
    "payment_mitigation_tested": False,
    "h1_rejected": False,
    "worker_metric_queried": False,
    "runtime_config_inspected": False,
    "h3_supported": False,
    "retry_enabled_checkout_03": True,
    "fix_applied": False,
}


def _save_state(state: dict[str, Any]) -> None:
    STATE_FILE.write_text(json.dumps(state, indent=2), encoding="utf-8")


def _load_state() -> dict[str, Any]:
    if not STATE_FILE.exists():
        state = DEFAULT_STATE.copy()
        _save_state(state)
        return state
    try:
        return json.loads(STATE_FILE.read_text(encoding="utf-8"))
    except (json.JSONDecodeError, OSError):
        state = DEFAULT_STATE.copy()
        _save_state(state)
        return state


def _probe(instance: str, requests: int, state: dict[str, Any]) -> dict[str, Any]:
    allowed = {"fleet", "checkout-01", "checkout-02", "checkout-03"}
    if instance not in allowed:
        return {"status": "error", "message": f"Unknown target: {instance}"}
    if requests <= 0:
        return {"status": "error", "message": "requests must be greater than zero"}

    faulted = state["retry_enabled_checkout_03"] and instance in {
        "fleet",
        "checkout-03",
    }
    failures = round(requests * 7 / 12) if faulted else 0
    successes = requests - failures
    return {
        "status": "ok",
        "observed_at": "2026-08-14T10:08:00Z",
        "target": instance,
        "requests": requests,
        "successes": successes,
        "failures": failures,
        "failure_rate_pct": round(failures * 100 / requests, 1),
    }


@mcp.tool()
def get_incident_brief() -> dict[str, Any]:
    """Return the initial alert and customer-visible symptom. Use this first."""
    return {
        "incident_id": "INC-2048",
        "reported_at": "2026-08-14T10:04:30Z",
        "service": "checkout-api",
        "symptom": "Customers intermittently receive HTTP 500 during payment.",
        "alert": "PAYMENT_TIMEOUT errors above threshold after a recent release.",
        "known_scope": "Not yet established",
    }


@mcp.tool()
def search_logs(instance: str = "all") -> dict[str, Any]:
    """Search centralized checkout logs. Use for error text and request timing, not for causal conclusions."""
    records = [
        {
            "timestamp": "2026-08-14T10:04:11Z",
            "instance": "checkout-03",
            "level": "WARN",
            "message": "PAYMENT_TIMEOUT after 2000ms",
            "request_id": "req-781",
        },
        {
            "timestamp": "2026-08-14T10:04:12Z",
            "instance": "checkout-03",
            "level": "ERROR",
            "message": "checkout failed: upstream payment canceled",
            "request_id": "req-781",
        },
        {
            "timestamp": "2026-08-14T10:04:14Z",
            "instance": "checkout-02",
            "level": "INFO",
            "message": "checkout completed in 241ms",
            "request_id": "req-782",
        },
        {
            "timestamp": "2026-08-14T10:04:15Z",
            "instance": "checkout-01",
            "level": "INFO",
            "message": "checkout completed in 229ms",
            "request_id": "req-783",
        },
    ]
    if instance != "all":
        records = [record for record in records if record["instance"] == instance]
    return {
        "source": "central-log-index",
        "coverage": "Application-side checkout logs; provider timing is not included.",
        "records": records,
    }


@mcp.tool()
def query_metrics(metric: str = "overview") -> dict[str, Any]:
    """Query service and provider metrics. Use to compare scope across instances and systems."""
    state = _load_state()
    failure_03 = 58.3 if state["retry_enabled_checkout_03"] else 0.0
    worker_03 = 100 if state["retry_enabled_checkout_03"] else 43
    metrics: dict[str, Any] = {
        "checkout_failure_rate_pct": {
            "checkout-01": 0.0,
            "checkout-02": 0.0,
            "checkout-03": failure_03,
        },
        "worker_pool_utilization_pct": {
            "checkout-01": 41,
            "checkout-02": 44,
            "checkout-03": worker_03,
        },
        "payment_provider_p95_ms": {
            "09:59Z": 176,
            "10:04Z": 184,
            "10:09Z": 181,
        },
    }
    if metric in {"overview", "worker_pool_utilization_pct"}:
        state["worker_metric_queried"] = True
        _save_state(state)
    if metric == "overview":
        return {
            "source": "metrics-platform",
            "window": "2026-08-14T09:59:00Z/2026-08-14T10:09:00Z",
            "metrics": metrics,
        }
    if metric not in metrics:
        return {
            "status": "error",
            "available_metrics": sorted(metrics),
        }
    return {
        "source": "metrics-platform",
        "window": "2026-08-14T09:59:00Z/2026-08-14T10:09:00Z",
        "metric": metric,
        "values": metrics[metric],
    }


@mcp.tool()
def get_deployments() -> dict[str, Any]:
    """Return application deployment history. Use to test whether a code release aligns with incident timing and scope."""
    return {
        "source": "deployment-registry",
        "deployments": [
            {
                "release": "checkout-api-2026.08.14.3",
                "started_at": "2026-08-14T09:28:00Z",
                "completed_at": "2026-08-14T09:32:00Z",
                "instances": ["checkout-01", "checkout-02", "checkout-03"],
                "status": "successful",
            }
        ],
        "note": "No application deployment occurred after 09:32Z.",
    }


@mcp.tool()
def run_checkout_probe(instance: str = "fleet", requests: int = 12) -> dict[str, Any]:
    """Run a deterministic synthetic checkout probe against the fleet or one instance. Use the same request count for comparisons."""
    return _probe(instance, requests, _load_state())


@mcp.tool()
def test_payment_provider_mitigation(timeout_ms: int = 5000) -> dict[str, Any]:
    """Test the payment-latency hypothesis by increasing only the simulated provider timeout and measuring the same probe before and after."""
    state = _load_state()
    before = _probe("fleet", 12, state)
    state["payment_timeout_ms"] = timeout_ms
    state["payment_mitigation_tested"] = True
    _save_state(state)
    after = _probe("fleet", 12, state)
    return {
        "hypothesis": "H1",
        "prediction": "A larger payment timeout should reduce checkout failures if provider latency is causal.",
        "change": {"payment_timeout_ms": {"from": 2000, "to": timeout_ms}},
        "before": before,
        "after": after,
        "delta_percentage_points": round(
            after["failure_rate_pct"] - before["failure_rate_pct"], 1
        ),
        "result": "NO IMPROVEMENT. Explicitly reject H1 before inspecting runtime configuration.",
    }


@mcp.tool()
def record_hypothesis_decision(
    hypothesis_id: str, decision: str, evidence: str
) -> dict[str, Any]:
    """Record a supported, challenged, or rejected hypothesis after citing evidence. Required gates prevent premature remediation."""
    state = _load_state()
    normalized_id = hypothesis_id.upper()
    normalized_decision = decision.lower()

    if normalized_id == "H1" and normalized_decision == "rejected":
        if not state["payment_mitigation_tested"]:
            return {
                "status": "blocked",
                "message": "Run the payment-provider mitigation test before rejecting H1.",
            }
        state["h1_rejected"] = True

    if normalized_id == "H3" and normalized_decision == "supported":
        if not (
            state["worker_metric_queried"]
            and state["runtime_config_inspected"]
        ):
            return {
                "status": "blocked",
                "message": "Two-source agreement missing: query worker-pool metrics and inspect runtime configuration.",
            }
        state["h3_supported"] = True

    _save_state(state)
    return {
        "status": "recorded",
        "hypothesis_id": normalized_id,
        "decision": normalized_decision,
        "evidence": evidence,
    }


@mcp.tool()
def inspect_runtime_config(instance: str) -> dict[str, Any]:
    """Read effective runtime configuration for one checkout instance. Available only after H1 is explicitly rejected."""
    state = _load_state()
    if not state["h1_rejected"]:
        return {
            "status": "blocked",
            "message": "Evidence gate closed: explicitly reject H1 after measuring the failed mitigation.",
        }

    configs = {
        "checkout-01": {
            "payment_retry_limit": 0,
            "worker_pool_size": 8,
            "rollout_id": "cfg-8841",
            "last_changed_at": "2026-08-14T10:03:40Z",
            "cancellation_log_label": "PAYMENT_TIMEOUT",
        },
        "checkout-02": {
            "payment_retry_limit": 0,
            "worker_pool_size": 8,
            "rollout_id": "cfg-8841",
            "last_changed_at": "2026-08-14T10:03:40Z",
            "cancellation_log_label": "PAYMENT_TIMEOUT",
        },
        "checkout-03": {
            "payment_retry_limit": (
                3 if state["retry_enabled_checkout_03"] else 0
            ),
            "worker_pool_size": 8,
            "rollout_id": "cfg-8841",
            "last_changed_at": "2026-08-14T10:03:40Z",
            "cancellation_log_label": "PAYMENT_TIMEOUT",
        },
    }
    if instance not in configs:
        return {"status": "error", "available_instances": sorted(configs)}

    state["runtime_config_inspected"] = True
    _save_state(state)
    return {
        "source": "runtime-config-service",
        "instance": instance,
        "effective_config": configs[instance],
    }


@mcp.tool()
def apply_runtime_config_fix(instance: str = "checkout-03") -> dict[str, Any]:
    """Set the divergent retry limit to the fleet value. Requires H3 support from two independent evidence sources."""
    state = _load_state()
    if not state["h3_supported"]:
        return {
            "status": "blocked",
            "message": "Record H3 as supported with two-source evidence before remediation.",
        }
    if instance != "checkout-03":
        return {
            "status": "blocked",
            "message": "The approved minimal change targets checkout-03 only.",
        }

    previous = 3 if state["retry_enabled_checkout_03"] else 0
    state["retry_enabled_checkout_03"] = False
    state["fix_applied"] = True
    _save_state(state)
    return {
        "status": "applied",
        "source": "runtime-config-service",
        "instance": instance,
        "change": {"payment_retry_limit": {"from": previous, "to": 0}},
        "unchanged": ["application_release", "payment_timeout_ms", "worker_pool_size"],
    }


@mcp.tool()
def query_trace_timeline(normalize_timestamps: bool = False) -> dict[str, Any]:
    """Secret Mission only. Return cross-system spans with raw or clock-normalized timestamps."""
    config_timestamp = (
        "2026-08-14T10:03:40Z"
        if normalize_timestamps
        else "2026-08-14T09:58:40Z"
    )
    return {
        "source": "distributed-tracing",
        "normalized": normalize_timestamps,
        "events": [
            {
                "timestamp": config_timestamp,
                "clock": "config-agent",
                "event": "cfg-8841 applied to checkout-03",
            },
            {
                "timestamp": "2026-08-14T10:04:11Z",
                "clock": "checkout-platform",
                "event": "first failed checkout on checkout-03",
            },
        ],
    }


@mcp.tool()
def get_clock_health() -> dict[str, Any]:
    """Secret Mission only. Return measured offsets for clocks that emitted incident evidence."""
    return {
        "source": "time-sync-monitor",
        "measured_at": "2026-08-14T10:05:00Z",
        "clocks": {
            "checkout-platform": {"offset_seconds": 0, "status": "healthy"},
            "config-agent": {"offset_seconds": -300, "status": "degraded"},
        },
        "normalization_rule": "corrected timestamp = raw timestamp - offset_seconds",
    }


@mcp.tool()
def reset_lab() -> dict[str, Any]:
    """Restore the incident to its initial state for a fresh investigation."""
    _save_state(DEFAULT_STATE.copy())
    return {
        "status": "reset",
        "incident_id": "INC-2048",
        "message": "The false lead, evidence gates, configuration drift, and probe failure are restored.",
    }
# Evidence Ledger: INC-2048

## Hypotheses

| ID | Hypothesis | Status | Decisive evidence |
|---|---|---|---|

## Observations

| ID | Timestamp or window | Source | Observation | Interpretation | Hypothesis impact |
|---|---|---|---|---|---|
# Root-Cause Report: INC-2048

## Summary

## Trigger

## Mechanism

## Misleading signals

## Rejected alternatives

## Corrective action

## Verification

How does the lab fit together?

The .mcp.json file defines a project-scoped stdio server named checkout-incident. Its command starts the mcp executable from this project's environment.

The CLAUDE.md file sets the investigation boundaries. It requires competing hypotheses, timestamped evidence, falsification, two-source agreement, a minimal fix, and a negative control.

The server.py fixture exposes twelve deterministic incident tools. Its evidence gates prevent Claude from inspecting configuration or applying a fix too early.

The two Markdown files give your observations and final causal report a stable home. This keeps the investigation auditable from the first clue to the final proof.

  • Save every project file in your code editor.
  • Confirm that the editor sidebar lists requirements.txt, .gitignore, .mcp.json, CLAUDE.md, server.py, evidence-ledger.md, and root-cause.md.

A virtual environment stores this lab's Python packages inside .venv. The pinned SDK version makes the server behavior reproducible.

  • Create the isolated environment by running:
python3 -m venv .venv

What does this command do?

Python creates a self-contained environment in the .venv directory. The folder holds the interpreter and packages used only by this lab.

  • Activate the environment in your current Terminal by running:
source .venv/bin/activate

What changes after activation?

Your Terminal session now directs Python package operations into .venv. You should see the environment name at the start of your prompt.

The SDK installation can take a short moment while Python downloads its dependencies. A quiet pause during this command is normal.

  • Install the pinned MCP Python SDK by running:
python -m pip install 'mcp[cli]==2.1.0'

What does this install provide?

The mcp[cli]==2.1.0 package provides the SDK imported by server.py. Its CLI extra also provides the local mcp command referenced by .mcp.json.

The installation is complete when Terminal returns to the active environment prompt without reporting an installation failure. Your isolated MCP runtime is now ready.

Installation did not complete?

  • Check that your prompt still shows the active .venv environment.
  • Confirm that your Python version is still 3.10 or newer.
  • Check your internet connection if the package download stops.

Ask for help: Help me diagnose why the pinned MCP Python SDK did not install inside my active virtual environment.

Connect Claude Code to the server

Claude Code reads .mcp.json from the current project. The trust prompts protect you from starting an unapproved local process.

The command takes over this Terminal while the interactive session is running. Keep the session open because Claude Code manages the stdio server process for that session.

  • Confirm that Terminal remains inside the checkout-incident-lab folder.
  • Prepare to accept the workspace trust prompt when Claude Code displays it.
  • Prepare to approve the project-scoped .mcp.json server when Claude Code asks.
  • Start Claude Code from the project folder by running:
claude

What happens when Claude starts?

Claude Code loads the investigation protocol from CLAUDE.md. It also discovers the project-scoped server definition in .mcp.json.

Approving the server allows Claude Code to start the local process through the environment's MCP CLI. The server remains attached to this interactive session.

  • Accept the workspace trust prompt.
  • Approve the checkout-incident project server.

Before you check the connection, how many incident tools do you expect Claude Code to report?

  • Open the MCP status view by entering:
/mcp

What should I see?

You should see checkout-incident connected with 12 tools. That confirms Claude Code can reach every incident-response capability exposed by the local server.

Server not connected?

  • Confirm that Claude Code started from inside the checkout-incident-lab folder.
  • Check that .mcp.json is saved at the top level of that folder.
  • Confirm that the .venv directory exists beside server.py.

Ask for help: Help me diagnose why the checkout-incident project-scoped stdio server is not connected in Claude Code.

That is the lab online: Claude Code now has twelve controlled MCP tools plus an evidence-first investigation protocol. Next, you will reproduce the outage and test the persuasive payment-provider hypothesis against a measurable prediction.

Trigger and Reject the False Lead

The local MCP incident lab is connected to Claude Code through its 12 tools. That connection lets Claude investigate the simulated outage through operational evidence.

The first checkout logs point toward a payment-provider timeout. This step tests whether that explanation survives a controlled measurement.

In this step, get ready to:
  • Reproduce the fleet failure through the checkout incident tools.
  • Test the payment-provider hypothesis with a controlled timeout change.
  • Record the failed prediction before widening the investigation.
Reproduce the failure and form H1

A baseline gives every later comparison a fixed starting point. The incident brief provides the symptom while the fleet probe measures its scope.

  • Switch back to the Claude Code session from earlier.
  • Tell Claude to investigate only through the checkout-incident tools.
  • Ask Claude to reset the scenario through reset_lab.
  • Ask Claude to read the incident brief through get_incident_brief.
  • Ask Claude to run run_checkout_probe with instance set to fleet.
  • Require the fleet probe to use 12 requests.
  • Ask Claude to inspect only the checkout logs through search_logs.
  • Ask Claude to add H1 to evidence-ledger.md as The payment provider is slower than the checkout timeout.

Claude reports intermittent HTTP 500 responses from checkout-api. The 12-request fleet probe returns 7 failures.

That result establishes a baseline failure rate of 58.3%. The logs place PAYMENT_TIMEOUT on checkout-03 at 2026-08-14T10:04:11Z.

You now have a reproducible failure plus a specific hypothesis that can make a measurable prediction.

Observation before interpretation

The log label is an observation from the application. H1 is an interpretation of what might have caused that label.

A useful hypothesis predicts what should change when its suspected cause is altered. That prediction makes H1 testable.

Tool calls missing?

  • Return to /mcp in the Claude Code session from earlier.
  • Confirm checkout-incident still shows 12 connected tools.
  • Repeat only the missing tool call after the connection is restored.

Ask for help: My checkout-incident tool call is missing. Can you help me restore the connection?

Test the timeout prediction

A falsification test changes the suspected cause while measuring the same outcome. H1 predicts that a larger payment timeout should reduce checkout failures.

Before you test H1, what change should the fleet failure rate show if the 2000 millisecond timeout is causing the outage?

  • Ask Claude to test H1 through test_payment_provider_mitigation with timeout_ms set to 5000.
  • Ask Claude to compare H1's prediction with the returned before-versus-after measurements.

The before measurement is 58.3%. The after measurement remains 58.3%.

The mitigation produces a 0.0 percentage-point improvement. This unchanged result is the intended shortfall.

Why this rejects H1

H1 predicted fewer failures after the timeout increased from 2000 to 5000 milliseconds. The measured failure rate did not move.

That failed prediction is stronger than the persuasive log label. H1 no longer explains the measured behavior.

Seeing different measurements?

  • Ask Claude to call reset_lab again.
  • Repeat the fleet probe with exactly 12 requests.
  • Repeat test_payment_provider_mitigation with timeout_ms set to 5000.

Ask for help: My mitigation measurements do not match the deterministic baseline. Can you help me check the tool inputs?

Record the rejection and verify the gate

A rejected hypothesis becomes useful evidence when the ledger preserves why it failed. The decision tool then turns that evidence into an explicit investigation gate.

What belongs in the ledger?

  • Each row names its evidence source.
  • Each row preserves the source timestamp.
  • The observation column records returned facts.
  • The interpretation column explains what those facts mean.
  • The hypothesis-impact column states which explanation the evidence supports or challenges.
  • Tell Claude to preserve source timestamps in every new evidence-ledger.md row.
  • Ask Claude to add the incident-brief observation to evidence-ledger.md.
  • Ask Claude to add the checkout-log observation to evidence-ledger.md.
  • Ask Claude to add the baseline fleet-probe measurement to evidence-ledger.md.
  • Ask Claude to add the failed payment-timeout mitigation measurement to evidence-ledger.md.
  • Require every new row to keep observation separate from interpretation.

The ledger now shows why the timeout explanation looked plausible. It also preserves the measurement that disproved it.

  • Ask Claude to call record_hypothesis_decision for H1 as rejected using the no-improvement evidence.
  • Tell Claude to stop without calling inspect_runtime_config.

Before you ask for the checkpoint, which status should H1 hold after its prediction failed?

  • Ask Claude to report a before-versus-after failure-rate comparison.
  • Ask Claude to report H1's recorded decision.
  • Ask Claude to confirm that inspect_runtime_config was not called.

Claude should report 58.3% before the mitigation. Claude should report 58.3% after the mitigation.

The improvement is 0.0 percentage points. H1's recorded status is rejected.

The inspect_runtime_config tool remains unused. The investigation has rejected its first explanation without jumping to a replacement.

Strong work. You have turned a convincing false lead into a measured rejection that the rest of the investigation can build on.

H1 decision blocked?

  • Confirm Claude completed test_payment_provider_mitigation before recording the decision.
  • Ask Claude to cite the matching 58.3% measurements as evidence.
  • Repeat record_hypothesis_decision with H1 set to rejected.

Ask for help: The H1 decision is blocked. Can you help me verify the required mitigation evidence?

H1 is rejected while the runtime-configuration evidence remains untouched. Next, you'll widen the investigation across metrics and deployment history without replacing one guess with another.

Build a Cross-System Evidence Ledger

The failed timeout mitigation gave your Claude Code investigation something valuable: a hypothesis it can reject. However, rejecting H1 does not identify the cause.

A cross-system evidence ledger ties every claim to its provenance. In this step, your MCP investigation will widen from logs to metrics and deployment history without replacing one guess with another.

In this step, get ready to:
  • Maintain three competing hypotheses as separate explanations.
  • Correlate evidence from logs, metrics, and deployment history.
  • Update evidence-ledger.md with source-attributed observations.
Maintain three competing hypotheses

Competing hypotheses stop one plausible explanation from controlling the investigation. Each hypothesis needs its own status so new evidence can challenge one without silently promoting another.

  • Return to the Claude Code session from earlier.
  • Ask Claude to preserve H1 as Rejected in evidence-ledger.md.
  • Ask Claude to add H2 for the hypothesis that the latest application deployment introduced a checkout regression.
  • Ask Claude to add H3 for the hypothesis that one checkout instance has configuration drift that exhausts its worker pool.
  • Ask Claude to keep all three hypotheses visible while it widens the investigation.

Good progress. Your ledger now preserves the rejected false lead while giving the deployment and configuration explanations a fair test.

Correlate metrics with deployment history

Logs show the symptom from the application’s point of view. Metrics reveal whether the pattern is fleet-wide or isolated to one instance.

Deployment history tests whether the timing and scope fit H2.

  • Ask Claude to call query_metrics with overview.

You’ll see checkout-03 at 58.3% failures. You’ll see checkout-01 at 0.0%.

You’ll also see checkout-02 at 0.0%. The payment-provider p95 stays between 176 ms and 184 ms.

  • Ask Claude to add E4 for the provider p95 result from metrics-platform.
  • Ask Claude to add E6 for the instance failure scope from metrics-platform.

The provider timing weakens the rejected latency explanation. The instance-local failure pattern challenges a fleet-wide application regression.

  • Ask Claude to call get_deployments.

You’ll see release checkout-api-2026.08.14.3 completed on all three instances at 2026-08-14T09:32:00Z.

The deployment completed 32 minutes before the first failure. Its fleet-wide scope does not match a problem isolated to checkout-03.

  • Ask Claude to add E5 for the deployment timing from deployment-registry.
  • Ask Claude to compare E2 with E4, E5, and E6.

Claude should conclude that timing and scope challenge H2. The instance-local pattern makes H3 the next hypothesis to test.

Missing metrics or deployments?

  • Use /mcp in the Claude Code session to confirm the checkout-incident server remains connected.
  • Retry only the tool call that did not return evidence.

Ask for help: My MCP metrics or deployment query did not return evidence. Help me check the connection without restarting the investigation.

Separate observations from interpretations

An observation records what a source returned. An interpretation explains what that fact means for a hypothesis.

Keeping those fields separate makes every conclusion traceable. It also exposes any claim that rests on inference alone.

  • Ask Claude to audit every row against the required ledger columns.
  • Ask Claude to revise any row that mixes a sourced fact with an inference.
  • Ask Claude to show the revised observations table.

Each row should now identify an evidence ID. Each row should also identify its timestamp or window.

The source should have its own field. The observation should remain separate from interpretation and hypothesis impact.

  • Ask Claude to set H2 to Challenged.
  • Ask Claude to set H3 to Ready to test.
  • Ask Claude to show the revised hypotheses table.

The hypotheses table should keep H1 as Rejected. It should show the deployment evidence challenging H2.

The table should leave H3 ready for a direct test. Supporting it requires evidence from the next stage of the investigation.

  • Stop before Claude inspects runtime configuration.
  • Leave H3 without a Supported decision.

Before you review the finished ledger, which hypothesis should have the strongest next test based on the evidence so far?

  • Ask Claude to show the finished contents of evidence-ledger.md.

You should see timestamped observations from central-log-index. You should also see observations from metrics-platform.

The ledger should include deployment timing from deployment-registry. Every row should separate observation from interpretation.

The hypothesis impact column should show why each fact matters. H2 should be Challenged.

H3 should be Ready to test. No runtime configuration evidence should appear yet.

✔️ My ledger matches

Your cross-system ledger is ready. It preserves provenance while keeping the next hypothesis open for testing.

ⓧ I'd like to double check

Confirm that H1 remains Rejected because its controlled mitigation produced no improvement.

Confirm that E4 records provider p95 between 176 ms and 184 ms.

Confirm that E5 records the fleet-wide deployment completion at 2026-08-14T09:32:00Z.

Confirm that E6 records checkout-03 at 58.3% failures while both peers remain at 0.0%.

Confirm that H2 is Challenged. Confirm that H3 is Ready to test.

Your ledger now distinguishes persuasive signals from sourced evidence. Next, you’ll test H3 against worker-pool metrics. Runtime configuration provides the independent check.

Prove Configuration Drift

The previous investigation rejected payment-provider latency. It also challenged a fleet-wide deployment regression.

Your Claude Code evidence ledger now points toward configuration drift on one instance. That pattern still needs causal support.

You'll use MCP tools to compare matching probes, worker-pool metrics, and effective runtime configuration. The evidence gate opens only when independent sources agree on the same mechanism.

In this step, get ready to:
  • Capture matching pre-fix probes for the target instance and control instance.
  • Correlate worker-pool metrics with effective runtime configuration.
  • Record H3 as supported through two-source agreement.
Capture matching pre-fix probes

Matching inputs make the target and control results comparable. checkout-02 acts as a negative control under the same 12-request workload.

  • Return to the Claude Code session from earlier.
  • Ask Claude to call run_checkout_probe for checkout-03 with 12 requests.

Claude should report 7 failures from 12 requests. The pre-fix failure rate is 58.3%.

  • Ask Claude to repeat run_checkout_probe for checkout-02 with 12 requests.

Claude should report 12 successful requests. The control failure rate is 0.0%.

  • Ask Claude to add both pre-fix probe results to evidence-ledger.md with their timestamps and sources.

The ledger now preserves the failing target beside a healthy instance under the same test conditions.

Why use a negative control?

The target probe shows that checkout-03 is unhealthy. The control probe shows that the same checkout workload still succeeds on checkout-02.

That stable control helps separate an instance-local fault from a fleet-wide provider problem.

Confirm the two-source mechanism

Worker-pool utilization reveals where the service is saturated. Effective runtime configuration reveals the instance-level difference that can produce that saturation.

  • Ask Claude to call query_metrics for worker_pool_utilization_pct.

Claude should report 100% utilization on checkout-03. The peer values should be 41% on checkout-01 and 44% on checkout-02.

  • Ask Claude to call inspect_runtime_config for checkout-01.
  • Ask Claude to call inspect_runtime_config for checkout-02.

Both peers should show a payment_retry_limit of 0. Each peer should also show a worker_pool_size of 8 and a rollout_id of cfg-8841.

  • Ask Claude to call inspect_runtime_config for checkout-03.

The target should show a payment_retry_limit of 3. Its worker_pool_size remains 8 under the same cfg-8841 rollout.

  • Ask Claude to update evidence-ledger.md with source-attributed metric and runtime-configuration rows that keep observations separate from interpretations.

The ledger now contains two independent views of the same instance-local mechanism. The metrics-platform shows saturation while the runtime-config-service shows the divergent retry setting.

Configuration access blocked?

  • Ask Claude to confirm that H1 was explicitly rejected through record_hypothesis_decision after the failed mitigation.
  • Ask Claude to retry inspect_runtime_config after it confirms the rejection is recorded.

Ask for help: Why is inspect_runtime_config still blocked after H1 was rejected?

Record H3 as supported

A supported hypothesis needs a shared mechanism plus an explicit evidence decision. Here, the divergent retry limit explains why only one eight-worker pool reaches full utilization.

  • Ask Claude to explain the retry-driven worker-pool saturation in evidence-ledger.md using the metric and runtime-configuration evidence IDs.

Claude should connect the retry limit of 3 to repeated work on checkout-03. That mechanism explains its 100% worker-pool utilization and instance-local failures.

Before Claude records the decision, do you expect the evidence gate to accept H3 now?

  • Ask Claude to call record_hypothesis_decision with H3 set to supported and cite both evidence IDs.

You should see the decision return a status of recorded. The configuration fix is now permitted and remains unapplied.

Why does the evidence gate open?

The metric evidence establishes worker-pool saturation on the failing instance. The runtime-configuration evidence independently identifies the retry setting that differs from both healthy peers.

Together, those sources support one mechanism without relying on the misleading timeout label.

H3 decision blocked?

  • Ask Claude to confirm that worker_pool_utilization_pct was queried before the decision.
  • Ask Claude to confirm that inspect_runtime_config returned results for all three checkout instances.

Ask for help: Why is the H3 support decision still blocked?

Before the final check, which finding do you think most clearly separates configuration drift from a fleet-wide failure?

  • Ask Claude to summarize the pre-fix probes, worker-pool metrics, runtime configurations, and hypothesis decisions from evidence-ledger.md.

Claude should report checkout-03 at 58.3% failures. Its worker-pool utilization should be 100%.

Claude should report checkout-02 at 0.0% failures. The peer utilization values should remain 41% and 44%.

Runtime configuration should show payment_retry_limit at 3 only on checkout-03. Both peers should show 0 under the same cfg-8841 rollout.

The ledger should still show H1 rejected and H2 challenged. It should now show H3 supported through two-source agreement.

That is the proof threshold met. Next, you'll apply the smallest targeted fix and verify causality against the same control.

Apply the Smallest Fix and Verify Causality

Your MCP investigation has narrowed the incident to configuration drift on checkout-03. Independent metrics plus runtime configuration show retries saturating the worker pool on that instance.

A plausible diagnosis becomes a root cause only when the smallest targeted change removes the failure. A negative control checks that the improvement is specific to the suspected instance.

In this step, get ready to:
  • Apply the approved runtime configuration fix to checkout-03.
  • Repeat the target probe plus the negative-control probe.
  • Complete the evidence ledger plus the root-cause report.
Apply the approved configuration fix

A controlled mitigation changes only the variable supported by the evidence. The apply_runtime_config_fix tool sets payment_retry_limit from 3 to 0 on checkout-03.

Changing runtime configuration can feel risky, even in a simulated incident. The lab constrains this action to one instance plus one setting.

  • Return to the Claude Code session from earlier.
  • Ask Claude to call apply_runtime_config_fix for checkout-03 only.
  • Ask Claude to confirm that payment_retry_limit changed from 3 to 0.
  • Ask Claude to confirm that the application release remains unchanged.
  • Ask Claude to confirm that payment_timeout_ms remains 5000.
  • Ask Claude to confirm that worker_pool_size remains 8.

Claude reports status as applied. The returned change shows payment_retry_limit moving from 3 to 0.

The application release remains unchanged. The payment timeout remains at 5000 milliseconds.

The worker-pool size remains at 8. Your mitigation has changed only the configuration value supported by the evidence.

Fix Not Applied?

  • Confirm that H3 is still recorded as supported in the current lab state.
  • Check that Claude targeted checkout-03 when calling apply_runtime_config_fix.

Ask for help: Why was my checkout-03 configuration fix blocked?

Repeat the target and control checks

Causal verification repeats the same measurements after the change. The target should recover while the unaffected control stays stable.

Before Claude repeats the checks, do you expect the failure to disappear from both instances or only from the instance that changed?

  • Ask Claude to run run_checkout_probe against checkout-03 with 12 requests.
  • Ask Claude to run run_checkout_probe against checkout-02 with 12 requests.
  • Ask Claude to query worker_pool_utilization_pct through query_metrics.

You will see checkout-03 return 12 successes with 0 failures. Its failure rate has changed from 58.3% to 0.0%.

You will see checkout-02 remain at 0.0% failures. This stable result makes it a useful negative control.

The checkout-03 worker-pool utilization falls from 100% to 43%. The resource saturation has cleared with the checkout failure.

Why Does the Negative Control Matter?

The target result shows that the changed instance recovered. The stable control shows that the outcome does not depend on a fleet-wide recovery.

Together, those measurements connect the targeted configuration change to the recovered checkout behavior.

Results Do Not Match?

  • Confirm that each run_checkout_probe call used exactly 12 requests.
  • Confirm that apply_runtime_config_fix returned applied before the probes ran.
  • Check that the lab was not reset after the fix.

Ask for help: Why do my post-fix target or control measurements differ from the expected results?

Complete the incident record

Evidence provenance keeps the final conclusion tied to specific observations. Your two incident files now need to preserve the rejected alternatives plus the measurements that prove recovery.

  • Ask Claude to add the post-fix checkout-03 probe to evidence-ledger.md.
  • Ask Claude to add the checkout-02 negative-control result to evidence-ledger.md.
  • Ask Claude to preserve H1 as rejected in the hypothesis table.
  • Ask Claude to preserve H2 as challenged in the hypothesis table.
  • Ask Claude to preserve H3 as supported in the hypothesis table.
  • Ask Claude to organize root-cause.md into separate sections for the trigger, mechanism, misleading signals, rejected alternatives, corrective action, and verification.
  • Ask Claude to cite the worker-pool recovery in the verification section.

✔️ Awesome, I've got everything!

Great. Double check that Claude saved both incident files before the final review.

ⓧ I'd like to double check the full code

Compare Claude's completed incident files with these reference versions.

# Evidence Ledger: INC-2048

## Hypotheses

| ID | Hypothesis | Status | Decisive evidence |
|---|---|---|---|
| H1 | The payment provider is slower than the checkout timeout. | Rejected | E3, E4 |
| H2 | The latest application deployment introduced a checkout regression. | Challenged | E5, E6 |
| H3 | One checkout instance has configuration drift that exhausts its worker pool. | Supported | E6, E7, E8 |

## Observations

| ID | Timestamp or window | Source | Observation | Interpretation | Hypothesis impact |
|---|---|---|---|---|---|
| E1 | 2026-08-14T10:04:30Z | incident brief | Customers intermittently receive HTTP 500 and the alert names PAYMENT_TIMEOUT. | Establishes symptom, not cause. | Supports investigating H1 but proves none. |
| E2 | 2026-08-14T10:04:11Z | central-log-index | checkout-03 logs PAYMENT_TIMEOUT after 2000 ms. | The label may indicate provider latency or a locally canceled task. | Weak support for H1. |
| E3 | 2026-08-14T10:08:00Z | synthetic probe | Failure rate is 58.3% before and 58.3% after raising the timeout to 5000 ms. Delta is 0.0 percentage points. | H1 predicted improvement and failed that prediction. | Rejects H1. |
| E4 | 09:59Z to 10:09Z | metrics-platform | Payment-provider p95 remains between 176 ms and 184 ms. | Provider latency is far below both timeout settings. | Rejects H1. |
| E5 | 2026-08-14T09:32:00Z | deployment-registry | The application release completed on all three instances 32 minutes before the first failure. | Timing and fleet-wide scope do not match an instance-local regression. | Challenges H2. |
| E6 | 09:59Z to 10:09Z | metrics-platform | checkout-03 fails at 58.3%; checkout-01 and checkout-02 remain at 0.0%. | The fault is instance-specific. | Challenges H2 and supports H3. |
| E7 | 09:59Z to 10:09Z | metrics-platform | Worker-pool utilization is 100% on checkout-03 versus 41% and 44% on peers. | Local resource saturation is the failure mechanism. | Supports H3. |
| E8 | 2026-08-14T10:03:40Z | runtime-config-service | payment_retry_limit is 3 on checkout-03 and 0 on both peers. | Configuration drift explains the instance-local saturation. | Supports H3 through an independent source. |
| E9 | 2026-08-14T10:08:00Z | synthetic probe | After the targeted fix, checkout-03 changes from 58.3% to 0.0% failures. | Removing the divergent retry behavior removes the symptom. | Verifies H3 causally. |
| E10 | 2026-08-14T10:08:00Z | synthetic probe | checkout-02 remains at 0.0% before and after the fix. | The result is not explained by a fleet-wide provider recovery. | Negative control supports H3. |
# Root-Cause Report: INC-2048

## Summary

Checkout failures were caused by configuration drift on checkout-03, not payment-provider latency or the application release.

## Trigger

Configuration rollout cfg-8841 set `payment_retry_limit` to 3 on checkout-03 at 2026-08-14T10:03:40Z while checkout-01 and checkout-02 remained at 0.

## Mechanism

Retries on checkout-03 saturated its eight-worker pool. Canceled payment tasks were then labeled `PAYMENT_TIMEOUT` by a broad exception wrapper, producing a persuasive but misleading upstream-provider signal.

## Misleading signals

- Checkout logs used the label `PAYMENT_TIMEOUT`.
- The alert referred to a recent release.
- The application release was visible in deployment history, but it completed 32 minutes earlier and affected all instances.

## Rejected alternatives

- H1 payment-provider latency: rejected because increasing the timeout from 2000 ms to 5000 ms left the failure rate at 58.3%, a 0.0 percentage-point improvement, while provider p95 remained at or below 184 ms.
- H2 application regression: challenged because the release affected all instances while failures and worker saturation appeared only on checkout-03.

## Corrective action

Set `payment_retry_limit` from 3 to 0 on checkout-03. The application release, payment timeout, and worker-pool size were left unchanged.

## Verification

| Probe | Before | After |
|---|---:|---:|
| checkout-03 failure rate | 58.3% | 0.0% |
| checkout-02 negative control | 0.0% | 0.0% |
| checkout-03 worker-pool utilization | 100% | 43% |

The target recovered while the negative control remained stable, supporting the configuration change as the causal fix.

Before you request the final summary, which measurements would separate a causal fix from a coincidental recovery?

  • Ask Claude to summarize the before-and-after measurements from evidence-ledger.md.
  • Ask Claude to confirm that root-cause.md connects the trigger to the mechanism.
  • Ask Claude to confirm that the report cites the unchanged negative control.

Claude should cite checkout-03 changing from 58.3% failures to 0.0%. It should cite checkout-02 remaining at 0.0%.

Claude should also cite worker-pool utilization falling from 100% to 43%. The report should identify cfg-8841 as the trigger.

You have now turned the supported configuration-drift hypothesis into a verified causal result.

Secret mission

Correct the Clock-Skewed Trace

Your causal fix is solid. The raw trace places the configuration event too early for the verified sequence. Use source clock health to correct the timeline without reopening the diagnosis.

Clean Up Your Resources

Clean Up Your Resources

Your incident lab now contains the verified investigation record. Choose what happens to the local resources next.

This project creates no cloud infrastructure. It adds no ongoing cost beyond your existing Claude Code access.

Resources you used:

  • Active Claude Code session with the local checkout-incident MCP process.
  • Local project folder containing the incident lab.
  • Python virtual environment at .venv/.
  • Project-scoped MCP configuration at .mcp.json.
  • Local MCP server at server.py.
  • Incident state at .incident-state.json.
  • Investigation protocol at CLAUDE.md.
  • Dependency pin at requirements.txt.
  • Ignore rules at .gitignore.
  • Completed investigation record at evidence-ledger.md.
  • Completed causal report at root-cause.md.

Keep everything running

No action is needed. Choose this option for another investigation run.

  • Leave the Claude Code session active to keep the checkout-incident tools ready.
  • Retain the local project folder for future demonstrations.
  • Preserve evidence-ledger.md as the investigation record.
  • Preserve root-cause.md as the causal report.

Pause - I'll come back to this later

Shut down the active process while leaving the local files untouched. The lab remains ready for another session.

  • Ask Claude to call reset_lab if you want a clean incident state for your next session.
  • Confirm the reset response reports status as reset.
  • Exit the Claude Code session.

The local stdio server stops with the session. Your project files remain ready for later.

Delete - I don't want to use this again

Remove the local lab when you no longer need the tools or reports. Deleting the folder is permanent.

Keep the project if you may want to demonstrate the investigation again.

  • Exit the Claude Code session.

The cleanup command checks for .mcp.json plus server.py before removing the current folder. This guard protects unrelated directories.

  • Delete the incident lab folder from macOS Terminal by running this guarded command:
PROJECT_DIR="$(pwd)"
PROJECT_NAME="${PROJECT_DIR##*/}"

if test ! -f "$PROJECT_DIR/.mcp.json" || test ! -f "$PROJECT_DIR/server.py"
then
  printf 'Stop: incident lab files were not found.\n'
else
  cd ..
  rm -rf "$PROJECT_DIR"
  if test ! -e "$PROJECT_DIR"
  then
    printf '%s removed\n' "$PROJECT_NAME"
  else
    printf 'Stop: the project folder could not be removed.\n'
  fi
fi

What does this cleanup command do?

The command captures the current folder path. The guard then checks for the incident lab files.

A matching folder is removed from its parent directory. The command prints its name after deletion succeeds.

That closes the loop. Your local incident lab is gone once you see the captured folder name followed by removed.

Seeing a Safety Stop?

The guard stops deletion when either incident lab file is missing. A second stop means the folder still exists after the removal attempt.

  • Return to the incident lab folder from earlier in macOS Terminal.
  • Run the guarded cleanup command again.

Help me safely identify the incident lab folder before deleting it.

Nice Work!

Nice Work!

Investigation complete! You proved that configuration drift on checkout-03 caused the phantom payment outage.

You've learned how to:

  • Connect Claude Code to a local stdio MCP server that exposes 12 incident-response tools.
  • Use an evidence ledger to reject the payment-provider hypothesis after its controlled mitigation delivered a 0.0 percentage-point improvement.
  • Produce a root-cause report that proves configuration drift through two-source agreement. Verify causality by reducing checkout-03 failures from 58.3% to 0.0% while checkout-02 remained at 0.0%.
  • Secret Mission: Correct a five-minute clock skew by normalizing 2026-08-14T09:58:40Z to 2026-08-14T10:03:40Z. Restore the trace sequence without changing the verified mechanism.

Ready to quiz yourself?