Test Local Failure and Recovery

Test Docker Compose outages and recover a local OrderFlow stack.

Introduction

30 Second Summary

Reliable systems often look fine until one missing part exposes the weak spot. Without a controlled test, an outage leaves you guessing about the system's true condition.

In this project, you will turn your existing OrderFlow stack into a controlled reliability lab. You will compare each dependency outage with a recorded healthy baseline before restoring the stack with Docker Compose.

What You'll Build

You trigger each outage on demand and watch OrderFlow reveal whether client requests fail immediately or queued work waits safely for recovery.

By the end of this project, you'll have:

A repeatable failure matrix that lets you compare every outage against its healthy baseline. The failure-recovery.md report preserves the evidence behind each conclusion.
A bounded gateway test that shows how a Catalog outage reaches the client through Caddy. Catalog logs then reveal whether Redis Open Source acts as an optional cache or a hard dependency. Recovered product data shows whether PostgreSQL remains the source of truth.
An asynchronous recovery demo where one fresh RabbitMQ-backed order waits while the Inventory worker is offline. After recovery, you watch the inventory value change without submitting the order again.
Secret Mission: Apply a temporary restart policy that recovers the Catalog container after an abrupt crash. A forced Compose recreation then restores its inherited configuration.

Are there any prerequisites?

This project assumes you completed the first seven OrderFlow projects through Route Microservices Through a Caddy API Gateway. The inherited persistent-checkout workspace must still contain the working Compose stack.

You also need Docker Engine, Docker Compose, curl, and Visual Studio Code on Linux.

Before We Start

Before any container state changes, a clear failure hypothesis gives every outage a question to answer. This is your moment to predict which OrderFlow dependency will create the most visible client-side failure and why.

Prepare the OrderFlow Stack

Your recorded failure hypothesis only becomes useful when each experiment starts from the same healthy baseline. A broken starting point would make the later outage evidence ambiguous.

Docker Engine runs the inherited containers. Docker Compose controls their shared lifecycle.

The curl check completes the local tool verification. You will also validate the inherited Compose definition before starting OrderFlow.

In this step, get ready to:

Verify the local failure-testing toolchain on Linux.
Validate the inherited Compose configuration.
Confirm the full inherited stack reaches its expected running state.

Verify the confirmed Linux tools

The Docker check confirms that the client can reach the server. The remaining checks prove Compose and curl are available.

Launch your Linux terminal from the application launcher.
Run the three readiness checks with these commands:

docker version
docker compose version
curl --version

What do these checks prove?

The Docker check asks both the local client and the Docker server for information.
The Compose check confirms that its command is available.
The curl check displays information about the installed build.

Docker should print client details followed by server details. Compose should print its version information.

curl should print its build information. That is your first clean checkpoint because the local tools are ready for stack validation.

Tool Check Not Completing?

Confirm the Docker service is running if server details are missing.
Check that your Linux account has permission to use Docker.
Confirm the Compose check is available through the Docker command.

Help me diagnose the failed local tool check.

Validate the inherited Compose configuration

The inherited workspace already holds the configuration that defines OrderFlow. This check reads that definition without changing any containers.

Visual Studio Code keeps the workspace visible while its integrated terminal runs the validation from the correct directory.

Start Visual Studio Code through your Linux application launcher.
Open the inherited persistent-checkout folder using the top menu's folder-opening command.
Open Visual Studio Code's integrated terminal at the Compose project directory inside persistent-checkout.
Validate the inherited Compose configuration by running this command:

docker compose config --quiet

What Does This Check Do?

The validation command checks the Compose configuration without printing the rendered result. A quiet return to the terminal prompt means the configuration passed.

You should return to the prompt without a configuration error. Your inherited Compose definition is now ready to control the stack.

Configuration Check Not Quiet?

Confirm the integrated terminal is using the Compose project directory inside persistent-checkout.
Review the validation output for the configuration location that needs attention.
Keep the inherited configuration unchanged while investigating the reported issue.

Help me understand the Compose validation issue.

Start the full inherited stack

The startup command applies the inherited Compose definition in the background. It keeps the existing images and persistent volumes available for the failure lab.

Start the inherited OrderFlow services in the background by running this command:

docker compose up --detach

What Does This Startup Command Do?

Detached mode returns control of the terminal after starting the services.
Compose uses the inherited service definitions to create or start the required containers.
The existing named volumes remain available to their services.

Some services need a short startup window before their applications are ready. A temporary startup state is normal while the stack settles.

Before you check the stack, predict whether every required service has already settled.

Display the running and stopped Compose services by running this command:

docker compose ps --all

What Should You See?

The status table includes the inherited OrderFlow services. Each required service should settle into its expected running state before the failure experiments begin.

Inspect the status row for every required inherited service.
Wait briefly if any required service still reports a startup state.
Rerun the status command above after the service has had time to settle.

The checkpoint passes when every required inherited service reports its expected running state. You now have a trustworthy starting point for every outage in the lab.

Services Not Settling?

Check the startup output for the service that did not settle.
Confirm the configuration validation completed successfully in the previous substep.
Keep the inherited images and volumes intact while diagnosing the service state.

Help me diagnose the unsettled OrderFlow service.

Your OrderFlow stack now has a verified starting point. Next up, you will record the healthy application responses that every recovery result must match.

Capture a Healthy Baseline

Your inherited OrderFlow stack is running through Docker Compose. The services have settled into their expected running state.

A recorded healthy baseline gives every later failure result a concrete comparison. Your proven Caddy reads provide that evidence without consuming an order.

The Order request stays prepared for the Inventory worker experiment. Keeping it unsent prevents accidental duplicate work before that test begins.

In this step, get ready to:

Discover the literal Compose service keys used by the inherited stack.
Create a failure matrix for the four planned experiments.
Capture bounded baseline evidence with a fresh Order request ready for later.

Discover the inherited service keys

Compose commands target services by their literal configuration keys. Recording those keys now keeps every outage focused on the intended container.

List the inherited Compose service keys by running this command:

docker compose config --services

You should see one service key per line. The output represents the services defined by the inherited Compose configuration.

Service list missing or incomplete?

Confirm that the terminal from earlier is still inside the Compose project directory in persistent-checkout.

Compare the output with the inherited Compose configuration. Help me identify why my Compose service list is missing expected OrderFlow services.

Compare each output line with its service definition in the inherited Compose configuration.
Record the Caddy gateway key here: your Caddy gateway service key.
Record the Catalog API key here: your Catalog API service key.
Record the Redis Open Source key here: your Redis service key.
Record the Order API key here: your Order API service key.
Record the RabbitMQ key here: your RabbitMQ service key.
Record the Inventory API key here: your Inventory API service key.
Record the Inventory worker key here: your Inventory worker service key.

You now have the exact service names that every later stop command targets. That keeps each experiment controlled.

Create the failure matrix

A failure matrix keeps the healthy response beside the outage evidence. Each row follows one scenario from its baseline through recovery.

Switch back to Visual Studio Code from earlier.
Create failure-recovery.md at the top level of persistent-checkout with the new-file icon in the left file sidebar.
Add the service map and failure matrix to failure-recovery.md by pasting this content:

# Failure Recovery Report

## Compose Service Keys

- Caddy gateway: [[CADDY_SERVICE_KEY="your Caddy gateway service key"]]
- Catalog API: [[CATALOG_SERVICE_KEY="your Catalog API service key"]]
- Redis: [[REDIS_SERVICE_KEY="your Redis service key"]]
- Order API: [[ORDER_SERVICE_KEY="your Order API service key"]]
- RabbitMQ: [[RABBITMQ_SERVICE_KEY="your RabbitMQ service key"]]
- Inventory API: [[INVENTORY_API_SERVICE_KEY="your Inventory API service key"]]
- Inventory worker: [[INVENTORY_WORKER_SERVICE_KEY="your Inventory worker service key"]]

## Failure Matrix

| Scenario | Baseline | Failure response | Evidence | Recovery result |
| --- | --- | --- | --- | --- |
| Catalog API stopped | Catalog lookup below. |  |  |  |
| Redis stopped | Catalog lookup below. |  |  |  |
| Inventory worker stopped | Inventory read below. |  |  |  |
| Catalog process killed | Catalog lookup below. |  |  |  |

What does this report capture?

The service map preserves the literal Compose keys used by later commands.
The Baseline column points to the healthy reads that you record next.
The remaining columns leave room for client responses, service evidence, and recovery results.

Save failure-recovery.md.
Check the file for all five matrix columns and all four scenario rows.

You should see a service map above a five-column table. The table should contain one row for each planned failure.

Matrix formatting look uneven?

Check that every table row begins with a pipe character. Check that every row ends with a pipe character.

Make sure the separator row remains directly below the column names. Help me repair the Markdown table in my failure-recovery.md report.

Record bounded baseline requests

The baseline must exercise the same gateway path that already worked in the previous project. Adding response headers makes status evidence visible.

A ten-second limit keeps later dependency failures from holding the terminal indefinitely. The inherited request method and request data remain unchanged.

Find the exact successful Catalog lookup request retained from the Caddy gateway project.
Add -i to the copied Catalog request.
Add --max-time 10 to the copied Catalog request.
Preserve its inherited method, URL, headers, body, and TLS options.
Record the complete bounded command here: your bounded Catalog lookup request.

Before you run it, make your prediction about whether the bounded Catalog response still matches the successful gateway result.

Run the bounded Catalog lookup through Caddy with the recorded command:

[[CATALOG_REQUEST="your bounded Catalog lookup request"]]

Why bound the request?

The -i option includes response headers in the output.
The --max-time 10 option prevents an unavailable dependency from hanging the exercise indefinitely.
The unchanged request details keep this result comparable with the earlier gateway success.

You should see the successful Catalog status and comparison body established by the gateway project. Your first bounded baseline is now reproducible.

Record the Catalog status here: your Catalog baseline status.
Record the Catalog comparison body here: your Catalog baseline response body.

Catalog response differs from the earlier success?

Compare the copied request with the successful inherited command. Confirm that only the two bounded diagnostic options were added.

Confirm that the request still targets the Caddy route from the gateway project. Help me compare my bounded Catalog request with the inherited successful request.

Find the exact successful Inventory read request retained from the Caddy gateway project.
Add -i to the copied Inventory request.
Add --max-time 10 to the copied Inventory request.
Preserve its inherited method, URL, headers, body, and TLS options.
Record the complete bounded command here: your bounded Inventory read request.

Before you run it, make your prediction about whether the current inventory value still matches the earlier successful read.

Run the bounded Inventory read with the recorded command:

[[INVENTORY_REQUEST="your bounded Inventory read request"]]

What does this baseline prove?

This response records the Inventory API state before the worker outage. The selected inventory value becomes the comparison point for delayed processing.

The request remains read-only. It does not consume or duplicate the fresh order reserved for the later experiment.

You should see the successful Inventory status and body established by the earlier project. You now have a pre-order inventory value for the worker test.

Record the Inventory status here: your Inventory baseline status.
Record the Inventory comparison body here: your Inventory baseline response body.
Record the current inventory value here: your current inventory value.

Inventory value missing or unexpected?

Confirm that you copied the same Inventory read used in the previous project. Check that the inherited item reference remains unchanged.

Confirm that the response completed before the ten-second limit. Help me diagnose my bounded Inventory baseline request.

An Order creation request changes local state and can produce downstream inventory work. This step stores one fresh request shape without sending it.

Find the successful Order creation request shape retained from the previous OrderFlow project.
Add -i to the copied Order request.
Add --max-time 10 to the copied Order request.

The prepared copy now has visible response headers and a bounded wait. Its inherited request shape remains available for the worker experiment.

Replace any client-supplied order identifier or idempotency value with a fresh value.
Record the complete prepared command here: your prepared fresh Order request.
Leave the prepared Order request unsent.
Add the baseline evidence and prepared request to failure-recovery.md by pasting this content below the matrix:

## Healthy Baseline

### Catalog Lookup

```bash
[[CATALOG_REQUEST="your bounded Catalog lookup request"]]
```
- Status: [[CATALOG_STATUS="your Catalog baseline status"]]
- Comparison body: [[CATALOG_BODY="your Catalog baseline response body"]]

### Inventory Read

```bash
[[INVENTORY_REQUEST="your bounded Inventory read request"]]
```
- Status: [[INVENTORY_STATUS="your Inventory baseline status"]]
- Comparison body: [[INVENTORY_BODY="your Inventory baseline response body"]]
- Current inventory value: [[INVENTORY_VALUE="your current inventory value"]]

## Prepared Fresh Order Request

```bash
[[ORDER_REQUEST="your prepared fresh Order request"]]
```

Prepared only. Not sent in this step.

How does this complete the baseline?

The Catalog section preserves the bounded request beside its healthy response.
The Inventory section preserves the read request beside its pre-order value.
The final section keeps one fresh Order request ready for the worker outage.

Save failure-recovery.md.
Compare the report with the service keys and baseline results you just recorded.

You should see all seven literal service keys above the matrix. Below it, you should see both successful read baselines and one clearly unsent Order request.

Report values missing after pasting?

Confirm that every recorded value above is filled in before copying the report section. Check that the two bounded requests include both required diagnostic options.

Keep the fresh Order request in the prepared section without running it. Help me complete the missing values in my healthy baseline report.

✔️ Awesome, I've got everything!

Your failure-recovery.md report now contains the service map, failure matrix, healthy reads, and prepared Order request.

ⓧ I'd like to double check the full code

Compare your complete failure-recovery.md file with this reference.

# Failure Recovery Report

## Compose Service Keys

- Caddy gateway: [[CADDY_SERVICE_KEY="your Caddy gateway service key"]]
- Catalog API: [[CATALOG_SERVICE_KEY="your Catalog API service key"]]
- Redis: [[REDIS_SERVICE_KEY="your Redis service key"]]
- Order API: [[ORDER_SERVICE_KEY="your Order API service key"]]
- RabbitMQ: [[RABBITMQ_SERVICE_KEY="your RabbitMQ service key"]]
- Inventory API: [[INVENTORY_API_SERVICE_KEY="your Inventory API service key"]]
- Inventory worker: [[INVENTORY_WORKER_SERVICE_KEY="your Inventory worker service key"]]

## Failure Matrix

| Scenario | Baseline | Failure response | Evidence | Recovery result |
| --- | --- | --- | --- | --- |
| Catalog API stopped | Catalog lookup below. |  |  |  |
| Redis stopped | Catalog lookup below. |  |  |  |
| Inventory worker stopped | Inventory read below. |  |  |  |
| Catalog process killed | Catalog lookup below. |  |  |  |

## Healthy Baseline

### Catalog Lookup

```bash
[[CATALOG_REQUEST="your bounded Catalog lookup request"]]
```
- Status: [[CATALOG_STATUS="your Catalog baseline status"]]
- Comparison body: [[CATALOG_BODY="your Catalog baseline response body"]]

### Inventory Read

```bash
[[INVENTORY_REQUEST="your bounded Inventory read request"]]
```
- Status: [[INVENTORY_STATUS="your Inventory baseline status"]]
- Comparison body: [[INVENTORY_BODY="your Inventory baseline response body"]]
- Current inventory value: [[INVENTORY_VALUE="your current inventory value"]]

## Prepared Fresh Order Request

```bash
[[ORDER_REQUEST="your prepared fresh Order request"]]
```

Prepared only. Not sent in this step.

Before the final check, make your prediction about whether the read-only baseline requests changed any Compose service state.

Confirm the inherited stack remains running by running this command:

docker compose ps --all

You should see every required inherited OrderFlow service in its expected running state. The stack now matches the healthy state recorded before the failure experiments.

Any required service no longer running?

Confirm that the terminal remains inside the same Compose project directory. Compare the current status with the settled service list from the previous step.

Do not begin an outage experiment until the healthy state is restored. Help me diagnose an unexpected service state before the failure lab.

Your healthy comparison point is recorded and the stack remains intact. Next, you will stop only the Catalog API and observe the first client-visible failure.

Break and Recover the Catalog API

Your healthy baseline proves the Catalog lookup works through Caddy. The full OrderFlow stack is running behind that request.

Stopping only the Catalog API exposes how the gateway behaves when its synchronous dependency disappears. The resulting response gives you evidence to diagnose the break from both sides.

In this step, get ready to:

Isolate the Catalog API with a controlled stop.
Correlate the bounded Caddy response with Compose status and logs.
Restore the Catalog API to its recorded healthy baseline.

Stop only the Catalog API

A Docker Compose stop removes one service from the active request path while retaining its container. This isolates the Catalog dependency without disturbing the rest of OrderFlow.

In failure-recovery.md, find the literal service key recorded for the Catalog API.
Use that service key wherever the commands show <catalog-service-key>.
Stop the Catalog API and display its state by running these commands:

docker compose stop <catalog-service-key>
docker compose ps --all <catalog-service-key>

What do these commands do?

The first command stops the selected service without removing its container.
The second command includes stopped containers when it displays the Catalog service state.

You should see the Catalog service reported as stopped or exited. Its container remains available for the recovery test.

Check the state of the full inherited stack by running this command:

docker compose ps --all

What does this status prove?

The full status view separates the controlled Catalog outage from a wider stack failure. It confirms whether Caddy and the remaining OrderFlow services stayed running.

You should see the Catalog service stopped or exited. Caddy and the remaining required services should still report running states.

Catalog not isolated?

Confirm that you replaced <catalog-service-key> with the literal Catalog service key from your report.
Compare the stopped service with the keys recorded in failure-recovery.md before continuing.
Use the full Compose status to confirm that the gateway remains running.

Ask for help with isolating the Catalog service without stopping the rest of OrderFlow.

Reproduce and diagnose the failure

The gateway is still available while its Catalog dependency is offline. Your bounded request now captures the inherited application's actual failure behavior without assuming a particular HTTP status.

Before you send the request, do you expect Caddy to return an immediate response or wait until the request reaches its time limit?

Rerun the exact bounded Catalog lookup retained from Step 2 through Caddy.

You should see a result that differs from the healthy baseline. The difference may appear in the status or body. The request may instead reach the curl time limit.

Record the observed request outcome in the Failure response cell for Catalog API stopped.

Request still matches the baseline?

Check the Catalog service state from the full Compose status output.
Confirm that you reran the same Catalog lookup used for the healthy baseline.
Confirm that the retained request still includes -i plus --max-time 10.

Ask for help with explaining why the bounded Caddy request still matches the baseline.

The client result shows the symptom. Gateway and Catalog logs connect that symptom to upstream behavior or the service shutdown.

Collect the final 50 log lines from the gateway and Catalog services by running this command:

docker compose logs --tail 50 <gateway-service-key> <catalog-service-key>

What do these logs reveal?

The gateway logs can show how Caddy handled the request while its upstream Catalog service was unavailable.
The Catalog logs can provide shutdown evidence from the controlled stop.
The --tail 50 option limits the evidence to the latest 50 lines from each selected service.

Add the relevant gateway or Catalog evidence to the Evidence cell for Catalog API stopped.

You have captured the failure from three angles. Your matrix now connects the client-visible break to container state and service logs.

Logs do not explain the response?

Check that <gateway-service-key> matches the literal Caddy gateway key in failure-recovery.md.
Check that <catalog-service-key> matches the literal Catalog API key in the report.
Keep only log lines that directly support the shutdown or gateway behavior you observed.

Ask for help with correlating the Caddy response with the latest gateway and Catalog logs.

Restore the synchronous path

The stopped Compose service still has an existing container. Starting that service reactivates the Catalog path without rebuilding images or deleting persistent data.

Restart the Catalog service by running this command:

docker compose start <catalog-service-key>

What does this recovery command do?

The command starts the existing Catalog container for the selected Compose service. It preserves the inherited configuration and attached persistent storage.

The terminal should report that the selected Catalog service has started.

Check the restarted Catalog service by running this command:

docker compose ps <catalog-service-key>

What should the service state show?

The status output should report the Catalog service as running. This confirms the container has rejoined the Compose stack before the client check.

If the service still shows a startup state, wait briefly before checking it again.

Before you rerun the Catalog lookup, do you think the recovered response will match every part of the healthy baseline?

Rerun the exact bounded Catalog lookup retained from Step 2 through Caddy.

You should see the same response status and comparison body recorded in the healthy baseline. That match proves the synchronous Catalog path has recovered.

Record the baseline match in the Recovery result cell for Catalog API stopped.

Recovery does not match the baseline?

Wait briefly if the Catalog process has only just entered its running state.
Rerun the same bounded Catalog lookup without changing its inherited method or route.
Compare the new status and body with the exact baseline values in failure-recovery.md.

Ask for help with diagnosing a Catalog service that is running but has not returned to baseline.

The Catalog route is healthy again with its failure evidence safely recorded. Next, you will test whether the Catalog can keep serving authoritative product data when its cache is unavailable.

Test Redis Cache Recovery

Your Catalog API is healthy again after the previous outage. This step tests what clients see when Redis Open Source disappears from the same Caddy-routed lookup.

PostgreSQL owns the Catalog product data. This outage reveals whether Redis behaves as an optional acceleration layer or a hard dependency.

In this step, get ready to:

Run a controlled Redis outage through the bounded Catalog request.
Correlate the client result with Catalog and Redis logs.
Restore Redis to the recorded healthy baseline.

Remove Redis from the request path

Redis can only be classified by observing Catalog while the cache is unavailable. Docker Compose stops the Redis service without removing its container.

Stop the Redis service and display its state by running these commands:

docker compose stop <redis-service-key>
docker compose ps --all <redis-service-key>

What Do These Commands Prove?

The first command stops only the service identified by your literal Redis service key.
The second command includes stopped containers in its status output.
Keeping the container preserves a reversible recovery path for this experiment.

You should see the selected Redis service reported as stopped or exited. The remaining OrderFlow services stay running.

Did the Wrong Service Stop?

Compare the value you used with the literal Redis service key recorded in failure-recovery.md.
Confirm that you ran the commands from the Compose project directory inside persistent-checkout.

Help me verify the Redis service key and Compose status for my OrderFlow stack.

Before you send the request, do you expect Catalog to return its baseline product data or expose a dependency on Redis?

Rerun the bounded Catalog lookup from earlier through Caddy.

You may see product data from a database fallback. You may instead see an error or reach the 10-second curl limit.

Record the Redis Compose state in the Redis stopped row of failure-recovery.md.
Record the bounded Catalog result in the same row. Include the observed response or timeout.

How Should You Classify the Result?

Product data matching the baseline shows that the inherited Catalog path can reach PostgreSQL while Redis is unavailable.

A client error or timeout shows that Redis currently acts as a hard dependency in this stack. Both outcomes are valid findings when supported by evidence.

Correlate the response with service logs

The client result shows the visible symptom. Service logs help connect that symptom to the cache outage.

Display the final 50 log lines from Catalog and Redis by running:

docker compose logs --tail 50 <catalog-service-key> <redis-service-key>

What Does This Log Check Add?

The command limits the output to the final 50 lines from each selected service.
Catalog logs can show how the application reacted while its cache was unavailable.
Redis logs can provide shutdown evidence for the same test window.

Find the cache connection or shutdown evidence that corresponds with the bounded request.
Add the relevant evidence to the Redis stopped row of failure-recovery.md.

You now have the client symptom beside the service-side evidence. That makes your Redis classification defensible.

No Relevant Cache Evidence?

Check that both placeholders were replaced with the literal Catalog and Redis service keys from your report.
Record the available shutdown evidence if Catalog produced no cache connection line.
State clearly in the report when the selected logs contain no matching connection evidence.

Help me correlate my Catalog response with the Redis outage logs.

Restore Redis and verify the data

The controlled stop preserved the Redis container. Starting that existing service restores the cache without deleting PostgreSQL-owned Catalog data.

Start Redis and display its current state by running these commands:

docker compose start <redis-service-key>
docker compose ps <redis-service-key>

What Does This Recovery Do?

The first command starts the existing Redis service container.
The second command displays the current state of that service.
This recovery leaves the inherited Compose volumes untouched.

You should see the selected Redis service reported as running. A startup state means the service still needs a brief moment to settle.

Wait briefly if the selected Redis service still reports a startup state.
Rerun the status check above after the wait.

Before you repeat the Catalog lookup, do you expect the recovered product data to match the Step 2 baseline?

Rerun the bounded Catalog lookup from earlier through Caddy.

You should receive product data matching the Step 2 baseline. This proves that the cache outage did not delete the Catalog data owned by PostgreSQL.

Compare the recovered product data with the Step 2 baseline.
Update the recovery result in the Redis stopped row of failure-recovery.md.

That completes the cache recovery test. Redis is running again with the Catalog response restored to its recorded baseline.

Catalog Still Differs From the Baseline?

Wait briefly if Redis has only just entered its running state.
Rerun the same bounded Catalog lookup without changing its URL or options.
Confirm that the Catalog service also remains running in the Compose stack.

Help me diagnose why Catalog has not returned to its baseline after Redis restarted.

Your cache is restored with its failure behavior documented. Next, you will test whether queued inventory work catches up after its worker returns.

Recover Inventory Worker Backlog

Your Catalog outage is recorded. Redis is running again with product data matching the healthy baseline.

A successful Order API response confirms that OrderFlow accepted a request. Downstream inventory processing is a separate asynchronous event backed by RabbitMQ.

In this step, get ready to:

Record the pre-order value for one in-stock item.
Submit one bounded Order request while the Inventory worker is offline.
Prove that the recovered worker applies the queued inventory change.

Select one item for the outage

A reliable asynchronous test needs a value from before the new order. Docker Compose then lets you isolate the Inventory worker while the broker and APIs keep running.

Rerun the bounded Inventory read request from Step 2.
Choose one in-stock item from the response.
Record the selected item in the Inventory worker stopped row of failure-recovery.md.
Record its current inventory as the pre-order value in the same row.
Create a controlled worker outage by running these commands:

docker compose stop <inventory-worker-service-key>
docker compose ps --all <inventory-worker-service-key>
docker compose ps --all

What do these commands prove?

The first command stops the Inventory worker without removing its container.
The second command shows the worker's current Compose state.
The final command shows whether RabbitMQ and the OrderFlow APIs remain running.

You should see the Inventory worker stopped or exited. RabbitMQ and the APIs should still report running states.

Record the stopped worker state in failure-recovery.md.

Worker still running?

Confirm that you replaced <inventory-worker-service-key> with the literal Inventory worker service key from your report.
Check the complete Compose status for a similarly named service that may be the actual worker.

Ask for help with the service state by selecting help me confirm why my Inventory worker did not stop.

Expose delayed inventory processing

The prepared Order request creates the work that the stopped consumer cannot process. The bounded Inventory read through Caddy reveals whether that work has reached the inventory state.

Send the order once

An automatic retry could submit the same work more than once. Keep the single-attempt request so any later inventory change has one clear cause.

Before you send the order, do you expect the Order response and the next Inventory read to show the same stage of work? Your next two checks test that prediction.

Switch back to the prepared Order creation request from Step 2.
Confirm that --max-time 10 remains in the request.
Confirm that the request contains no automatic retry flags.
Use the fresh required client-supplied identifier already prepared.
Send the prepared Order request exactly once.
Save the Order response in the Inventory worker stopped row of failure-recovery.md.
Rerun the bounded Inventory read for the selected item.
Compare the returned inventory with the pre-order value.

The Order request returns its response. The Inventory read still shows the pre-order value while the worker is offline.

That gap is the intended result. OrderFlow has accepted work that its asynchronous consumer has not processed yet.

Record the unchanged inventory result in failure-recovery.md.

What does the unchanged value prove?

The Order API response covers synchronous acceptance. The unchanged Inventory read shows that downstream processing waits for the consumer.

RabbitMQ keeps the work available for delivery when the Inventory worker returns. The recovery check tests that queue-backed handoff.

Inventory changed before recovery?

Confirm that the Inventory worker still shows a stopped state.
Check that the Inventory read targets the same item recorded before the order.
Check that you submitted the prepared Order request only once.

Investigate the unexpected result by selecting help me determine why inventory changed while the worker was stopped.

Observe the queued work catch up

The queued order now provides a recovery test. Restarting the same worker should process that existing work without another Order request.

Start the Inventory worker and check its state by running:

docker compose start <inventory-worker-service-key>
docker compose ps <inventory-worker-service-key>

What does this recovery command do?

The first command starts the existing Inventory worker container.
The second command checks whether that service reports a running state.

Wait until the Inventory worker reports a running state. Its application may need a short startup window before it consumes the queued order.

Before the next read, do you expect the queued order to change inventory without another POST? The next check tests that prediction.

Keep the prepared Order request untouched.
Rerun the bounded Inventory read for the selected item.
Repeat the same Inventory read until the returned value changes from the pre-order baseline.

You should eventually see the selected item's inventory change from its recorded pre-order value. The queued order has now reached downstream processing.

Strong work. You have proved that Order acceptance can precede inventory processing.

Collect the last 50 Inventory worker and RabbitMQ log lines by running:

docker compose logs --tail 50 <inventory-worker-service-key> <rabbitmq-service-key>

What do these logs add?

The logs connect the changed Inventory response to activity from the recovered worker and RabbitMQ. This gives the matrix service-side evidence alongside the client-side result.

Add the relevant Inventory worker processing evidence to the Inventory worker stopped row.
Add the relevant RabbitMQ recovery evidence to the same row.
Record that inventory changed after worker recovery without reposting the order.

Before the final status check, do you expect any outage target to remain stopped? Use the result to test whether the full stack has recovered.

Confirm the final OrderFlow service states by running:

docker compose ps --all

What does the final status prove?

You should see Catalog, Redis, RabbitMQ, the APIs, and the Inventory worker running. The controlled outage preserved the existing containers and persistent volumes.

Secret mission

Recover Automatically From a Catalog Crash

A temporary `on-failure:3` policy turns an abrupt Catalog crash into an automatic recovery test. You will prove Docker restores the service without a manual start. A final Compose recreation removes the temporary runtime policy.

Clean Up Your Resources

This reliability lab adds no paid resources or ongoing cloud costs. Choose whether to keep the local stack running, pause it for later, or remove its runtime and stored data.

Resources you used:

Inherited OrderFlow Docker Compose containers in persistent-checkout, including Caddy, Catalog, Redis Open Source, RabbitMQ, the APIs, and the Inventory worker.
The local Compose project network used by the OrderFlow services.
Docker named volumes containing persisted PostgreSQL database data and RabbitMQ data.

Every option preserves the source files and failure-recovery.md report in the open persistent-checkout workspace.

Keep everything running

No cleanup action is needed. Choose this if you want the recovered OrderFlow stack ready for another local reliability session.

Switch back to the terminal in persistent-checkout.
Confirm the full stack state by running this command:

docker compose ps --all

What does this command do?

This lists the running and stopped containers in the inherited Compose project. It gives you one final status check without changing the stack.

You should see Catalog, Redis, RabbitMQ, the APIs, and the Inventory worker running. Your recovered lab is ready for another session.

Pause - I'll come back to this later

Shut down the running services to free local memory. The stopped containers and persisted data remain available for your next session.

Switch back to the terminal in persistent-checkout.
Stop the inherited stack by running this command:

docker compose stop

What does this command do?

This stops the Compose services while preserving their containers. Your PostgreSQL and RabbitMQ data remain in their named volumes.

That is your pause point. The OrderFlow services are stopped without discarding the state preserved during the failure experiments.

Resume the stopped stack later by running this command from persistent-checkout.

docker compose start

How does the stack resume?

This starts the existing Compose containers again. It reuses the preserved runtime configuration and named-volume data.

You should see the stopped services start without rebuilding images or recreating the project.

Delete - I don't want to use this again

Deleting the local runtime permanently removes the Compose containers and stored OrderFlow data. Your source files and failure-recovery.md report remain in persistent-checkout as evidence of the lab.

Switch back to the terminal in persistent-checkout.
Remove the local Compose runtime and its named volumes by running this command:

docker compose down --volumes

What gets deleted?

This removes the Compose containers and project networks. It also removes the named volumes containing persisted OrderFlow database and RabbitMQ data.

You should see Compose report the local containers and network as removed. The stored PostgreSQL and RabbitMQ state is now permanently deleted.

Confirm that no OrderFlow Compose containers remain by running this command:

docker compose ps --all

What confirms cleanup?

This checks the same Compose project after deletion. An empty service listing confirms that its containers are gone.

You should see no Compose containers for the OrderFlow project. The local runtime cleanup is complete.

Nice Work!

Excellent work! Your OrderFlow reliability lab now captures controlled failure evidence through to a recovered healthy baseline.

You've learned how to:

Build a repeatable failure and recovery matrix in failure-recovery.md that follows each experiment from a healthy baseline through recovery. Correlate bounded Caddy responses with Docker Compose status plus service logs.
Reproduce a client-visible Catalog outage through the gateway. Classify the Redis Open Source outage from observed evidence. Verify that PostgreSQL-owned catalog data survives cache recovery.
Submit exactly one fresh order while the Inventory worker is offline. Prove that RabbitMQ-backed work completes after worker recovery. Confirm the selected item's inventory changes without reposting the order.
Secret Mission: Demonstrate automatic Catalog recovery after an abrupt crash with a temporary Docker restart policy. Recreate the service from Docker Compose to restore its inherited configuration.

Ready to quiz yourself?