Build a Serverless Order Pipeline

Build a local order pipeline with idempotency, retries, DLQs, and evidence.

Build a Serverless Event-Driven Order Pipeline

30 Second Summary

A customer can place one order yet trigger the same business effect twice when software fails at the wrong moment. That risk often stays hidden until retries begin.

In this project, you will build a local order pipeline that simulates event-driven architecture with Python and SQLite. You will prove that at-least-once delivery cannot duplicate a completed order.

What You'll Build

You will demo a local order pipeline that survives duplicate delivery while its reports prove every valid event creates one business effect.

By the end of this project, you'll have:

A command-line order demo that accepts a request and shows its stable event ID, schema version, and durable queue state.
A side-by-side failure demo showing how unsafe processing creates two effects while idempotency preserves one effect per event.
Saved reliability evidence that traces retries, dead-letter queue isolation, controlled replay, and restart safety.
Secret Mission: Inject a deterministic 100-message delivery storm and prove every reliability invariant from machine-readable output.

Are there any prerequisites?

You need intermediate experience with Python, SQL transactions, message queues, and automated testing.

Everything runs locally, with no AWS account or paid service required.

Before We Start

Before any hands-on work, commit to the reliability claim this project must prove. This local correctness simulator must show that a crash between the business write and queue acknowledgment cannot duplicate an order's business effect. AWS infrastructure stays outside its scope.

Create the Workspace and Event Contracts

Reliable recovery depends on an event contract with stable identity. Ambiguous events make durable queue state impossible to interpret after a crash.

This step prepares the complete simulator in Visual Studio Code with Python, SQLite, and pytest. You will finish by proving that a validated event reaches durable storage with a predictable message ID.

In this step, get ready to:

Verify Python 3.11+ and prepare the local workspace.
Load the complete simulator source with pytest pinned to 9.2.0.
Trace the event contract and confirm its durable queue state.

Verify Python and create the workspace

The project needs Python 3.11+ so its runtime matches the simulator's supported environment. Check the installed version before creating the virtual environment.

Use your system search to open Terminal on macOS or PowerShell on Windows.

macOS

Check the installed Python version by running this command:

python3 --version

What does this command check?

The command prints the Python version available through the macOS python3 executable. Use the result to choose the matching outcome below.

Windows

Check the installed Python version by running this command:

python --version

What does this command check?

The command prints the Python version available through the Windows python executable. Use the result to choose the matching outcome below.

✔️ I see version 3.11 or higher

That is one dependency settled. Your Python runtime can support the complete simulator.

ⓧ I see an older version

Your installed version is below the project's minimum. Update Python before creating the virtual environment so the environment uses the correct interpreter.

Download Python 3.11 or newer from the official Python downloads page.
Run the downloaded installer.
Close the current terminal after installation finishes.
Reopen Terminal on macOS or PowerShell on Windows.
Repeat the version check above.

ⓧ Command not found

Python is not available through your terminal yet. Install a supported interpreter before continuing.

Download Python 3.11 or newer from the official Python downloads page.
Run the downloaded installer.
Close the current terminal after installation finishes.
Reopen Terminal on macOS or PowerShell on Windows.
Repeat the version check above.

The simulator belongs in a named folder on your Desktop. The final command below opens that folder as a VS Code workspace.

Create the serverless-order-pipeline folder on your Desktop by running these commands:

cd ~/Desktop
mkdir serverless-order-pipeline
cd serverless-order-pipeline
code .

What does this workspace command do?

The first command places the project on your Desktop so you can find it later.
The next two commands create the serverless-order-pipeline folder before making it the active terminal location.
The final command opens the current folder as a VS Code workspace.

Confirm that VS Code opens with serverless-order-pipeline shown in the Explorer sidebar.

The Microsoft Python extension connects VS Code to your interpreter. It also enables Python language features and pytest integration.

Select the Extensions icon in the VS Code Activity Bar.
Search for Python in the Extensions sidebar.
Select the Python extension published by Microsoft.
Click Install if the extension is not already installed.

You should see the extension marked as installed. VS Code can now discover the virtual environment you create inside the workspace.

macOS

Create and activate .venv by running these commands in the terminal:

python3 -m venv .venv
source .venv/bin/activate
python --version

What does this environment setup do?

The first command creates an isolated Python environment inside .venv. Activation directs later Python commands to that environment.

The final command confirms which Python version the activated environment uses. You should see version 3.11 or higher.

Windows

Create and activate .venv by running these commands in PowerShell:

python -m venv .venv
.venv\Scripts\Activate.ps1
python --version

What does this environment setup do?

The first command creates an isolated Python environment inside .venv. Activation directs later Python commands to that environment.

The final command confirms which Python version the activated environment uses. You should see version 3.11 or higher.

Interpreter selection can be fiddly because VS Code may keep using a system interpreter after discovering .venv. Select the workspace interpreter explicitly so tests and terminal commands use the same environment.

Select the interpreter name in the VS Code status bar.
Choose the interpreter inside .venv from the list.
Use Python: Select Interpreter from the Command Palette if the status bar does not show an interpreter.

The VS Code status bar should now identify .venv as the selected interpreter.

Is the virtual environment missing?

Confirm the .venv folder appears under serverless-order-pipeline in the Explorer sidebar.
Confirm the terminal still shows the Python version you verified after activation.
Reload the interpreter list with Python: Select Interpreter after installing the Microsoft Python extension.

Still stuck? Help me connect VS Code to the .venv interpreter.

Load the complete simulator scaffold

The project uses a prebuilt scaffold so this setup step can establish every later failure path without building features early. Each empty file needs a matching source block from the Complete Code reference.

macOS

Create the project directories and empty files by running these commands:

mkdir -p pipeline data tests reports .vscode
touch main.py pipeline/__init__.py pipeline/core.py data/.gitkeep reports/.gitkeep tests/test_pipeline.py .vscode/settings.json .gitignore requirements.txt README.md

What does this scaffold command create?

The first command creates five directories for source code, persistent data, tests, reports, and workspace settings. The second command creates the matching empty files inside those directories.

Return to the Explorer sidebar in VS Code.
Confirm that all five directories appear under serverless-order-pipeline.

Windows

Create the project directories by running this command in PowerShell:

New-Item -Path "pipeline", "data", "tests", "reports", ".vscode" -ItemType Directory

What does this directory command create?

The command creates five directories for source code, persistent data, tests, reports, and workspace settings.

Create the empty project files by running this command:

New-Item -Path "main.py", "pipeline/__init__.py", "pipeline/core.py", "data/.gitkeep", "reports/.gitkeep", "tests/test_pipeline.py", ".vscode/settings.json", ".gitignore", "requirements.txt", "README.md" -ItemType File -Force

What does this file command create?

The command creates every file used by the simulator. The directories from the previous command provide the required destinations.

Return to the Explorer sidebar in VS Code.
Confirm that all five directories appear under serverless-order-pipeline.

Use the project Complete Code reference to populate every matching file in the Explorer sidebar.
Save all open files in VS Code.

What has the scaffold loaded?

The source now contains the command-line demos, queue lifecycle, safe consumer, unsafe consumer, dead-letter recovery, reporting, and deterministic storm workload. The test suite contains eight regression tests for the same lifecycle.

The two .gitkeep files preserve the otherwise empty data and reports directories. Generated databases remain excluded through .gitignore.

Install the pinned dependency and print its installed version by running:

python -m pip install -r requirements.txt
pytest --version

What does this dependency setup do?

The first command installs the dependency recorded in requirements.txt into the active virtual environment. The second command confirms that pytest 9.2.0 is available.

You should see pytest 9.2.0 in the final line. That confirms the dependency is installed inside the environment selected for this workspace.

✔️ Awesome, I've got everything!

Your workspace now contains the complete simulator scaffold. Double-check that every file is saved before running the contract demo.

ⓧ I'd like to double check the full code

Compare each file character for character with its matching block in the Complete Code reference. Use this checklist to confirm the cumulative scaffold.

Confirm requirements.txt pins pytest==9.2.0.
Confirm .vscode/settings.json enables pytest for the tests directory.
Confirm .gitignore excludes .venv, Python caches, and generated SQLite databases.
Confirm pipeline/__init__.py exports ApprovalRequired, ManualClock, Pipeline, ValidationError, make_event, and run_storm.
Confirm pipeline/core.py contains the event contract, SQLite schema, queue lifecycle, consumers, replay gate, reports, and storm workload.
Confirm main.py exposes the demo, report, storm, and replay commands.
Confirm tests/test_pipeline.py contains eight deterministic tests.
Confirm README.md contains the Mermaid architecture, setup guide, correctness model, AWS comparison, troubleshooting, and portfolio talking points.
Confirm data/.gitkeep and reports/.gitkeep are empty.

Did the dependency installation fail?

Confirm the terminal prompt shows that .venv is active.
Confirm requirements.txt contains the exact pytest pin from the Complete Code reference.
Confirm VS Code still shows the .venv interpreter in the status bar.

Need help? Help me diagnose the pytest installation in .venv.

Trace and verify the event contract

The producer accepts API-shaped input only after validating its event identity and schema compatibility. Successful events become durable rows in the messages table.

Select pipeline/core.py in the Explorer sidebar.
Find the make_event function.
Follow the event into validate_event.
Trace the validated event through api_gateway_producer and produce.
Locate the messages table inside SCHEMA.
Locate inject_raw beside the validated producer path.

How does the contract protect the queue?

The make_event function creates a stable event_id with a supported schema_version.
The validate_event function rejects missing fields, incompatible schemas, and invalid payload values.
The api_gateway_producer function returns a client error when validation fails.
The produce method inserts validated events into durable queue storage.
The inject_raw method bypasses producer validation only so later tests can place corrupted or incompatible data directly into the queue.

Before you run the contract demo, do you expect the validated event to be rejected or accepted into the queue?

Run the contract demo from the VS Code terminal with this command:

python main.py demo contract

What does the contract demo prove?

The demo creates one valid event before sending it through the API-shaped producer. The producer validates the event before inserting it into the queue.

The demo then reads the durable message back from SQLite. Stable IDs make this result reproducible on every run.

You should see "statusCode": 202, "state": "READY", and "message_id": "msg-0001" in the JSON output.

Your first durable event is accepted and waiting in the queue. The contract now gives every later retry a stable identity to protect.

Did the contract demo fail?

Confirm the terminal is running inside the serverless-order-pipeline folder that contains main.py.
Confirm pipeline/core.py and pipeline/__init__.py match their Complete Code blocks.
Confirm every edited file is saved in VS Code.

Still blocked? Help me debug the contract demo output.

The workspace and durable contract are ready. Next, you will follow one message through receive, processing, acknowledgment, and the business-effect ledger.

Process and Acknowledge Orders

The producer now validates an order event before placing it in the durable queue. This step follows that event through its consumer lifecycle.

Before failures make delivery diverge from processing, you need a visible success path. A local SQLite transaction provides that baseline.

In this step, get ready to:

Trace how receive() leases one ready message.
Trace how process_safe() commits one business effect with its acknowledgment.
Run the normal demo to verify the complete lifecycle.

Trace the message lease

The receive() method models a visibility timeout with a logical clock. The lease temporarily hides one message while a consumer works.

In the VS Code Explorer sidebar, select pipeline/core.py.
Find the receive() method inside the Pipeline class.
Follow the expired update to see eligible INFLIGHT messages return to READY.
Follow the query that selects the oldest eligible READY message.
Inspect the single update that assigns the lease fields.
Locate the call that increments deliveries.
Follow the query that reloads the updated message row.
Locate the call that records the RECEIVED attempt.

How Does the Lease Work?

An expired INFLIGHT message becomes READY when its visible_at value reaches ManualClock.now().
The oldest eligible READY row is selected first.
The lease update changes the state to INFLIGHT.
The same update increments receive_count.
The new visible_at value equals the logical time plus visibility_timeout.
The deliveries metric counts receive attempts.
The RECEIVED attempt creates lifecycle evidence.

Trace the safe transaction

The safe path must coordinate business state with message state. The process_safe() method uses one write transaction for both decisions.

Find the _tx() context manager near the start of the Pipeline class.
Trace self.conn.execute("BEGIN IMMEDIATE") at the start of the context manager.
Follow the exception path that calls self.conn.rollback().
Follow the success path that calls self.conn.commit().

What Defines the Transaction Boundary?

The BEGIN IMMEDIATE statement starts a write transaction immediately.
The rollback() path removes pending changes after an exception.
The commit() path makes every pending change durable.

Return to the process_safe() method.
Locate message = self.receive().
Locate event = self._decode(message) inside the transaction.
Trace the lookup for an existing event_id in effects.
Follow the ledger insertion inside the not duplicate branch.
Follow the update that changes the message state to ACKED.
Locate the call that increments acknowledgments.
Compare the APPLIED outcome with the DUPLICATE_ACK outcome.

How Does Safe Processing Stay Idempotent?

The idempotency decision uses event_id as the primary key for the effects ledger. A completed event already has a row with that identity.

The _decode() method parses the message body.
The same method validates the event contract.
A new event creates one effects row.
An existing event increments duplicate_effects_prevented.
The message becomes ACKED within the same transaction.
The lifecycle record stores either APPLIED or DUPLICATE_ACK.

Verify the normal delivery lifecycle

The normal demo resets the default database before creating one valid order event. It sends that event through the safe consumer.

In the VS Code menu bar, select Terminal.
Select New Terminal.
Before you run the demo, predict whether one delivery creates one ledger effect or multiple ledger effects.
Run the normal delivery demo from the repository root by entering this command:

python main.py demo normal

What Does This Demo Do?

The reset_db() function removes earlier demo state.
The make_event() function creates the order event.
The produce() method writes the event to the durable queue.
The process_safe() method runs the safe consumer.
The summary() method prints the final JSON metrics.

You'll see "effects": 1 in the JSON output. You'll also see "acknowledgments": 1.

The output includes "queue_states": {"ACKED": 1}. You've now closed the first complete order lifecycle in your simulator.

Normal Demo Output Looks Different?

Confirm the terminal is running from the serverless-order-pipeline folder.
Confirm the VS Code status bar still shows the .venv interpreter.
If Python reports an import problem, run the command from the folder containing main.py.

Help me diagnose my normal demo output.

In the VS Code Explorer sidebar, expand data/.
Confirm pipeline.db appears inside data/.

The data/pipeline.db database now stores one ACKED message. It also stores one effects row for evt-demo-001.

The lifecycle rows record ENQUEUED, RECEIVED, and APPLIED evidence. The metrics row records the acknowledgment.

Why Is the Local Transaction Stronger?

The simulator places the effects insert inside one SQLite database transaction. The ACKED update shares that transaction.

A real AWS pipeline crosses service boundaries. Amazon DynamoDB would store the effect. Amazon SQS would acknowledge processing through DeleteMessage.

Those operations do not share the simulator's local transaction. The simulator therefore demonstrates correctness under a stronger atomic boundary.

Your consumer now carries an order from READY to ACKED with one durable effect. Next, you'll crash the unsafe consumer after its effect commit. Redelivery will expose the duplicate business effect.

Reproduce Duplicate Delivery

Your normal demo proved that one order can move from READY to ACKED with one ledger effect. That success hides a failure window between a durable business write and queue acknowledgment.

At-least-once delivery means a queue can present the same message again when a worker never acknowledges it. This step exposes that risk by crashing the unsafe consumer after its business write commits.

In this step, get ready to:

Trigger the unsafe consumer's crash after its business commit.
Trace the 5 logical-second visibility timeout through redelivery.
Capture deterministic evidence of the duplicate business effect.

Challenge the unsafe consumer

The duplicate demo drives the existing unsafe path through a deterministic crash. Every run begins from the same database state.

Your Demo Database Resets

The reset looks destructive. It only replaces data/pipeline.db with a known demo state.

Your source files and tests remain unchanged.

Predict whether one event can create more than one durable effect when the worker never reaches acknowledgment.
Run the duplicate demo from the repository root with this command:

python main.py demo duplicate

What Did the Failure Reveal?

The output "unsafe_effects": 2 shows that one event created two durable unsafe ledger rows.
The output "delivery_attempts": 2 shows that the same queued event reached the consumer twice.
The output "crashes": 1 confirms that the first delivery stopped before acknowledgment.

Confirm the JSON output contains "unsafe_effects": 2.
Confirm the output contains "delivery_attempts": 2.
Confirm the output contains "crashes": 1.

Metrics Do Not Match?

Make sure the terminal is currently inside the serverless-order-pipeline folder.
Confirm that the selected virtual environment remains active in the terminal.
Check that main.py and pipeline/core.py still match the provided simulator source.

Get targeted help here: Help me diagnose why my duplicate demo metrics differ from the expected values.

You have reproduced the core risk: one stable event now owns two durable unsafe ledger rows. The duplicate came from redelivery after the missed acknowledgment.

Trace the split transaction

The counters show the symptom. The existing process_unsafe method reveals the unsafe SQLite transaction boundary that caused it.

In the file tree on the left side of VS Code, select pipeline/core.py.
Scroll to def process_unsafe.
Read from message = self.receive() through the end of the first with self._tx(): block.
Continue to the if crash_after_effect: branch.
Locate the later transaction that updates the message state to ACKED.

Where Is the Unsafe Boundary?

The call to self.receive() leases the message as INFLIGHT before business processing begins.
The first with self._tx(): block inserts the row into unsafe_effects.
The _tx context manager commits that insert when the block exits normally.
The crash branch records CRASHED in a separate transaction.
The branch returns before the later acknowledgment transaction can update the message to ACKED.
The committed business effect survives while the queue message remains INFLIGHT.

Follow the logical redelivery

The simulator uses ManualClock to advance directly to the visibility deadline. This makes the retry repeatable without waiting for wall-clock time.

Select main.py from the file tree.
Scroll to elif name == "duplicate":.
Trace the first call to pipeline.process_unsafe(crash_after_effect=True).
Trace the next line that calls clock.advance(5).
Trace the second call to pipeline.process_unsafe().

How Does Redelivery Happen?

The first process_unsafe call commits an unsafe effect before taking the crash branch.
The call to clock.advance(5) moves the logical clock to the message's visible_at deadline.
The next receive call changes the expired message from INFLIGHT to READY.
That same call leases the message again and increments receive_count.
The second unsafe processing attempt commits another ledger row before reaching acknowledgment.

Predict which three counters should remain identical when the deterministic failure runs again.
Recreate the final duplicate-delivery evidence by running the demo again:

python main.py demo duplicate

Why Is the Result Reproducible?

The call to reset_db clears the previous demo database before each run.
The demo always creates the event evt-demo-001.
The manual clock always advances by exactly 5 logical seconds.
The completed run leaves evt-demo-001 acknowledged after its second receive.
The database retains two unsafe_effects rows for that single event.

Verify that "unsafe_effects": 2 appears in the final output.
Verify that "delivery_attempts": 2 appears in the final output.
Verify that "crashes": 1 appears in the final output.

Output Changed Between Runs?

Confirm that you reran the exact duplicate demo command from the repository root.
Check that you only read the source files during this step.
Restore the provided source if either process_unsafe or the duplicate demo branch was edited accidentally.

Get targeted help here: Help me find why the duplicate demo changed between deterministic runs.

That repeatable result is your failure evidence. It isolates the split between the durable business-effect commit and the later acknowledgment transaction.

Your failure evidence now isolates the split transaction. Next, you'll make redelivery safe through atomic idempotency.

Add Idempotency and Recovery Controls

You now have deterministic proof that a committed order effect can survive a worker crash while its acknowledgment disappears. Redelivery then applies that effect a second time.

The safe path closes this gap with idempotency inside one SQLite transaction. It also bounds failed deliveries before controlled recovery takes over.

In this step, get ready to:

Trace the atomic transaction that protects each business effect.
Follow failed deliveries through bounded retries into DLQ isolation.
Inspect approval-gated replay before verifying the reliable lifecycle.

Trace the atomic safe path

An idempotency key gives one logical event a stable identity across repeated deliveries. The safe consumer uses event_id to decide whether the corresponding order effect already exists.

Return to pipeline/core.py in the VS Code editor.
Locate CREATE TABLE IF NOT EXISTS effects.
Confirm that event_id TEXT PRIMARY KEY makes every effect identity unique.
Scroll to process_safe.
Trace the code inside with self._tx(): from _decode(message) through _record.
Identify the line that raises InjectedCrash before the message state changes to ACKED.

How does the safe transaction work?

The _tx context manager starts a BEGIN IMMEDIATE transaction before the consumer changes durable state.
The existing lookup checks whether the ledger already contains the event ID.
A new event creates one effects row. A repeated event increments duplicate_effects_prevented.
An InjectedCrash exits the transaction through its rollback path. The prospective effect insert disappears.
A successful attempt changes the message to ACKED before the transaction commits. The effect decision and acknowledgment become one local atomic result.

Message delivery can repeat safely because each stable event ID can produce only one durable business effect.

Follow bounded retries into the DLQ

A visibility timeout acts as a temporary lease on a received message. A failed message remains INFLIGHT until its logical deadline expires.

Bounded retries prevent one poison message from cycling forever. The final failed attempt moves its recovery evidence into a dead-letter queue.

Locate set_fault in pipeline/core.py.
Confirm that the supported plans are transient plus crash_after_effect.
Follow receive to the update that returns expired INFLIGHT messages to READY.
Locate _handle_failure.
Confirm that max_receives: int = 3 sets the delivery limit.
Inspect the dlq insert to confirm that it preserves the failed message evidence.

What does each recovery control protect?

The faults table persists deterministic failure plans by event ID.
ManualClock advances logical time without sleeps. Each retry therefore occurs at a reproducible deadline.
_handle_failure records the error while leaving the message under its current lease.
The third failed receive copies message_id, event_id, body, receive_count, reason, plus moved_at into dlq.
Permanently invalid events follow the same bounded path. This isolates malformed input from valid queue traffic.

Prove recovery without duplicate effects

DLQ replay is an operator-controlled recovery action. The approval gate stops failed input from returning to the queue without an explicit decision.

Locate replay_dlq in pipeline/core.py.
Find the if not approve: guard.
Confirm that a denied replay increments replays_denied before raising ApprovalRequired.
Trace the approved branch into _enqueue_in_tx.
Verify that source_message_id links the replayed message to its DLQ source.
Confirm that the replayed body retains its original event_id.

Why retain the original event ID?

The original event ID keeps replay inside the same idempotency boundary. A previously completed event reaches DUPLICATE_ACK without creating another effects row.

The replay still creates a fresh queue message with a new message ID. Business identity remains stable across physical deliveries.

Before you run the reliable demo, do you expect its simulated crash to leave a committed effect behind?

Run the reliable scenario from the repository root by entering this command:

python main.py demo reliable

What does the reliable result prove?

The demo resets data/pipeline.db before creating two messages for evt-demo-001.
The first delivery raises InjectedCrash inside the transaction. Its prospective effect is rolled back.
The output contains "effects": 1 because only one durable business effect survives.
The output contains "duplicate_effects_prevented": 1 because the redelivered event finds the existing ledger row.
The output contains "crashes": 1 because the deterministic crash path ran once.
The output contains "acknowledgments": 2 because both physical queue messages finish in the acknowledged state.

Reliable metrics look different?

Confirm that the terminal is running from the serverless-order-pipeline repository root.
Check that main.py routes the reliable demo through pipeline.drain_safe().
Verify that process_safe keeps the effect decision plus acknowledgment inside the same _tx block.

Still stuck? Help me trace why my reliable demo metrics differ from the expected result.

That closes the failure window you reproduced earlier. Your pipeline now permits repeated delivery while preserving one correct order effect.

The recovery controls now have a deterministic success path. Next up, you'll turn these guarantees into repeatable regression evidence.

Test and Generate Reliability Evidence

Your safe consumer now protects the order ledger across crashes, redelivery, and replay. A successful demo alone cannot guard those guarantees against future regressions.

Pytest turns each failure mode into a repeatable assertion. The report generator converts the persisted SQLite state into evidence you can inspect or process with other tools.

In this step, get ready to:

Prove all eight deterministic tests pass.
Trace the controls that remove timing noise from the test suite.
Generate three reliability reports from the persisted simulator state.

Prove the regression baseline

A reliability test suite must reproduce both success and failure paths on demand. These eight tests cover durable contracts, normal delivery, unsafe duplication, safe rollback, visibility-based retry, poison isolation, controlled replay, and the delivery storm.

In VS Code, return to the terminal from earlier that is using .venv.

Before you run the suite, predict whether every lifecycle check can finish without waiting for real time.

Exercise every lifecycle path by running this command:

pytest -q

What Does This Run Prove?

You should see 8 passed. The quiet -q option keeps the result focused on the suite outcome.

The contract test proves a queued message survives a database restart.
The consumer tests reproduce duplication, rollback, visibility expiry, DLQ routing, and replay behavior.
The storm test verifies exact workload counts plus the final reliability invariants.

That gives you a regression baseline for the entire lifecycle. Every failure mode now has an executable proof.

Tests Not Passing?

Confirm the terminal is at the serverless-order-pipeline repository root that contains main.py.
Confirm the selected environment contains the dependency pinned in requirements.txt.
Confirm tests/test_pipeline.py still contains all eight test functions from the project source.

Still stuck? Help me diagnose the failing deterministic test.

Trace the deterministic test design

Deterministic tests replace real waiting and random failures with state you can control. This makes every retry boundary reproducible across machines.

In the VS Code file sidebar, select tests/test_pipeline.py.
Locate new_pipeline near the top of the file.
Trace ManualClock() into each new Pipeline instance.
Find pipeline.set_fault("evt-retry", "transient", 1) in the transient retry test.
Find clock.advance(5) in the retry and crash recovery tests.
Locate tmp_path in each database-backed test.
Read the exact assertions in test_duplicate_delivery_storm_writes_machine_readable_evidence.

Why Are These Tests Deterministic?

ManualClock controls logical time. Calling clock.advance(5) reaches the visibility deadline immediately.
Stable event IDs and message IDs make exact state assertions possible.
The faults table persists each remaining failure count inside the test database.
The tmp_path fixture gives each test a unique temporary directory. Every test leaves data/pipeline.db unchanged.
Exact assertions catch changes to counts, queue states, and invariant results.

Capture proof from the current database

The report command reads the safe lifecycle state already stored in data/pipeline.db. Each generated file presents that evidence for a different audience or workflow.

Before you generate the reports, predict whether the command prints one output path or all three output paths.

Ask main.py to summarize the existing database by running this command:

python main.py report

What Does This Command Generate?

You should see paths for reports/reliability.json, reports/reliability.md, and reports/lifecycle.jsonl printed in the terminal.

The values reflect the current contents of data/pipeline.db. A check records the evidence exercised in that persisted lifecycle.

The JSON file stores summary metrics, queue states, and invariant checks for machine processing.
The Markdown file presents the metrics and checks as readable evidence tables.
The JSONL file stores one ordered lifecycle attempt per line.

In the VS Code file sidebar, expand the reports folder.
Select reliability.json.

You should see summary metrics plus the zero_duplicate_business_effects, zero_lost_acknowledged_orders, and replay_requires_explicit_approval checks.

Select reliability.md.

You should see a Reliability Evidence document with separate metrics and checks tables.

Select lifecycle.jsonl.

Each line is one ordered attempt record. Its outcome can be ENQUEUED, RECEIVED, FAILED, CRASHED, DLQ, APPLIED, DUPLICATE_ACK, or UNSAFE_APPLIED.

Reports Missing or Empty?

Confirm the terminal is at the repository root that contains data/pipeline.db.
Collapse the reports folder in the file sidebar. Expand it again to refresh the visible files.
Confirm data/pipeline.db still contains the reliable demo state.

Still stuck? Help me diagnose my missing reliability reports.

That is the evidence layer complete. Your reliability claims now have a passing regression suite plus three inspectable artifacts.

Secret mission

Step 6: Secret Mission - Survive the Duplicate Delivery Storm

Run a deterministic 100-message delivery storm across the completed simulator. Use the generated reports to prove that every reliability invariant survives.

Clean Up Your Resources

Your benchmark evidence now proves that the 100-message workload stayed correct. Every resource is local and free, so choose whether to keep the evidence, pause your virtual environment, or delete the generated runtime artifacts.

Resources you used:

The local .venv environment with pytest==9.2.0 installed.
The SQLite database at data/pipeline.db with the persisted storm workload.
The generated evidence files at reports/reliability.json, reports/reliability.md, and reports/lifecycle.jsonl.

Keep everything running

No action is needed. Choose this option if you want to rerun the simulator or demonstrate its reliability evidence.

Keep .venv so the pinned test dependency remains available.
Keep data/pipeline.db so you can inspect the persisted message lifecycle.
Keep the three report files so you can present the storm metrics and lifecycle journal.

These files consume only local disk space. They create no ongoing process or cloud cost.

Pause - I'll come back to this later

The storm command has already finished, so no simulator process remains active. Pausing returns your terminal to its system interpreter while preserving every project file.

Return the active terminal to its system interpreter by running this command:

deactivate

What does this command do?

The command stops the current shell from using .venv. It does not delete the environment or any project evidence.

Confirm that your terminal prompt no longer begins with .venv.

Your source code, database, tests, and reports remain ready for your next session.

Does deactivate fail?

If your shell cannot find the command, the current terminal is already outside the virtual environment. No further pause action is needed.

Help me check whether my Python virtual environment is active.

Delete - I don't want to use this again

Deleting generated artifacts is permanent, but the commands and file actions below preserve your simulator source and tests. You remove only .venv, the storm database, and the three generated reports.

Choose your operating system below to remove the generated runtime artifacts.

macOS

Return to the integrated terminal at the serverless-order-pipeline repository root.
Remove the generated runtime artifacts by running these commands:

rm -rf .venv
rm -f data/pipeline.db
rm -f reports/reliability.json reports/reliability.md reports/lifecycle.jsonl

What gets removed?

The commands remove the virtual environment, persisted storm database, and generated reliability evidence.
The commands preserve your pipeline source, test suite, documentation, and both .gitkeep files.

Return to the file tree in Visual Studio Code.
Confirm that .venv no longer appears.
Confirm that data and reports contain only their .gitkeep files.

Do the files still appear?

Check that the terminal path ends with serverless-order-pipeline before retrying the cleanup commands. A different terminal location targets the wrong paths.

Help me safely remove these local project artifacts.

Windows

The shared file tree in Visual Studio Code provides a direct way to remove the generated artifacts without changing your source files.

Delete the .venv folder from the file tree.
Delete data/pipeline.db from the file tree.
Select reports/reliability.json, reports/reliability.md, and reports/lifecycle.jsonl in the file tree.
Delete the selected report files.
Confirm that data and reports contain only their .gitkeep files.

Your pipeline implementation, test suite, configuration, and README remain available.

Unable to remove a file?

Close any terminal or editor tab that is using the database before trying again. Another local process can keep the file unavailable for deletion.

Help me identify which local project file is still in use.

Nice Work!

Mission accomplished. You built a deterministic local order pipeline that keeps business effects correct through the full failure lifecycle.

Your reports capture 115 delivery attempts from a 100-message workload. They prove 88 valid business effects with every reliability check passing.

What you learned:

Reproduce the duplicate-effect risk behind at-least-once delivery. Crash the unsafe consumer between its business write and acknowledgment to capture deterministic failure evidence.
Implement atomic idempotency with SQLite. Add bounded retries plus poison-message isolation. Guard dead-letter queue replay with explicit approval.
Run eight deterministic regression tests with pytest. Generate machine-readable reliability evidence in reports/reliability.json, reports/reliability.md, plus reports/lifecycle.jsonl. Use that evidence to separate local guarantees from real AWS service behavior.
Complete the Secret Mission by driving a deterministic 100-message storm through mixed failures. Prove every reliability invariant from the generated reports.

Ready to quiz yourself?