Coordinate a Distributed Ticket API

Introduction

30 Second Summary

A busy ticket release can send the same purchase through more than once. When several servers handle those requests, keeping every buyer response fast without selling a seat twice becomes the real challenge.

In this project, you will extend your existing Spring Boot ticket reservation API into a two-instance system that coordinates admission through Redis while PostgreSQL remains in charge of each purchase. You will prove its behavior through concurrent retries, service outages, database checks, and local load tests.

What You'll Build

During your finished demo, you can send competing confirmations to two API instances, interrupt Redis, restore it, and show from PostgreSQL that each seat still has one owner.

By the end of this project, you'll have:

• Transactional idempotency that returns the same reservation when concurrent clients reuse one Idempotency-Key. A different request using that key receives a conflict without creating another reservation.
• Shared admission control that limits simultaneous confirmation work across two API instances. You can demonstrate why separate local semaphores fail before replacing them with an atomic Redis lease.
• Outage and load evidence from fail-open and fail-closed experiments. You can compare real load-test measurements before using database queries to confirm that duplicate confirmed seats remain at zero.
• Secret Mission: Stop Redis during simultaneous same-seat retries against both API instances. Separate request failures from database correctness violations using an automated outage proof.

Are there any prerequisites?

You need the completed Part 1 repository plus Java 25, Apache Maven, Docker Desktop, and Visual Studio Code on your Mac. Grafana k6 runs from a Docker image, so it needs no host installation.

Before We Start

Part 1 already uses PostgreSQL as the final authority for booking correctness. Part 2 extends that foundation across two application instances.

Distributed coordination introduces a new decision point before each booking transaction. A clear boundary keeps Redis focused on shared admission while PostgreSQL protects every purchase.

In this step, get ready to:

• Open the completed Part 1 repository in Visual Studio Code.
• Confirm the existing Part 1 project structure.
• Define the correctness boundary between Redis and PostgreSQL.

Open the Part 1 repository

The existing ticket-reservation-api folder is the baseline for every change in Part 2. You only need to open it at this point.

• Press Cmd+Space on macOS or the Windows key on Windows to open your search bar.
• Type Visual Studio Code into the search bar.
• Press Enter to open Visual Studio Code.
• Choose the option for opening an existing folder from the welcome screen.
• Select your local ticket-reservation-api folder.
• Confirm that pom.xml appears at the top level of the Explorer sidebar.
• Confirm that the src/ folder appears in the Explorer sidebar.

You should see pom.xml at the top level. You should also see src/ in the file tree.

Good, you now have the exact Part 1 baseline ready for this extension. No Part 2 files or containers have been created.

Can't find the repository?

Use your operating system's file search to locate ticket-reservation-api. Choose the folder whose top level contains pom.xml.

Help me locate the correct Part 1 repository folder.

Define the correctness boundary

Admission control decides whether a request can proceed at a given moment. Purchase authorization remains a PostgreSQL decision.

• Assign shared admission decisions to Redis.
• Assign load-shedding decisions to Redis.
• Assign idempotency decisions to PostgreSQL.
• Assign seat ownership decisions to PostgreSQL.
• Assign reservation validity decisions to PostgreSQL.

Who decides what?

• Redis decides whether a confirmation may enter the booking transaction.
• Redis sheds load when the shared admission limit is full.
• PostgreSQL enforces transactional idempotency inside the confirmation transaction.
• PostgreSQL decides who owns each seat.
• PostgreSQL decides whether each reservation is valid.

An admitted request still has to pass the PostgreSQL transaction. This boundary keeps database correctness intact during coordination failures.

Before you complete the checkpoint, can a Redis admission decision alone prove that a reservation is valid?

You have the Part 1 baseline open with its correctness boundary intact. Next, you'll add the local infrastructure needed to observe distributed behavior.

Extend the Local Stack

Part 1 already protects each purchase with PostgreSQL. This step keeps that correctness boundary intact while preparing the API for distributed experiments.

You will add managed Redis support with bounded timeouts. You will also package the existing API for two local instances through Docker Compose.

In this step, get ready to:

• Prepare Spring Boot for Redis with bounded admission defaults.
• Package the existing API in a Java 25 runtime image.
• Define the two-instance local stack with an optional k6 load service.

Add Redis support and bounded defaults

Spring Boot manages the compatible Spring Data Redis version through its dependency catalog. Adding the starter gives later admission code a configured Redis client without changing the existing JPA stack.

• In the VS Code Explorer sidebar, select pom.xml.
• Find the existing spring-boot-starter-data-jpa dependency.
• Add the Redis starter directly below that dependency by pasting this line:

    <dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-data-redis</artifactId></dependency>

What does this dependency add?

• The starter lets Spring Boot configure a StringRedisTemplate for string-based Redis operations.
• Spring Boot supplies the compatible Spring Data Redis version.
• Lettuce provides the default Redis client connection.

• Save pom.xml.
• Open the integrated terminal from the top menu in VS Code.
• Confirm the managed dependency preserves the Part 1 tests by running:

mvn test

What does this check prove?

Maven resolves the new starter through Spring Boot 4.1.1. The test run confirms that adding Redis support has not changed the existing booking behavior.

You should see the original Part 1 test suite complete successfully. Your Maven build now has Redis support without depending on a running Redis service.

Dependency resolution failing?

Check that the Redis dependency sits inside the existing dependencies element. Confirm that every opening XML tag still has its matching closing tag.

Ask for help with the Maven change:

Redis now sits on a synchronous request path that you will build later. Short connection and operation timeouts keep an unavailable coordination service from holding requests indefinitely.

• In the Explorer sidebar, select src/main/resources/application.properties.
• Add the Redis connection settings and admission defaults below the existing JPA setting by pasting:

spring.data.redis.host=localhost
spring.data.redis.port=6379
spring.data.redis.connect-timeout=200ms
spring.data.redis.timeout=200ms
app.instance-id=local
app.admission.mode=off
app.admission.failure-policy=fail-open
app.admission.limit=8
app.admission.lease-ttl=2s
app.admission.hold-delay=0ms

What do these defaults control?

• The two 200ms values bound Redis connection attempts and operations.
• The off mode keeps admission disabled during the original Part 1 tests.
• The fail-open value records the availability-first outage policy used later.
• The limit and lease settings establish safe local defaults for the shared gate.
• The 0ms delay keeps normal confirmations free from the lab delay used in a later experiment.

• Save src/main/resources/application.properties.
• Confirm the disabled admission mode leaves the original test path unchanged by running:

mvn test

Why does Redis stay optional here?

The application has Redis connection settings available. Admission remains disabled through app.admission.mode=off, so the original PostgreSQL tests do not need to contact Redis.

You should see the Part 1 tests complete successfully again. That result confirms the new defaults preserve the existing correctness path.

Tests trying to use Redis?

Confirm that app.admission.mode=off appears once in application.properties. Check that the file uses the exact property names shown above.

Ask for help tracing the unexpected Redis access:

Package the existing API for Docker

Each application container needs the same packaged JAR. A small Docker image copies that artifact into a Java 25 runtime.

• In the Explorer sidebar, create a file named Dockerfile at the top level of the open repository.
• Paste this container definition into Dockerfile:

FROM eclipse-temurin:25-jre
WORKDIR /app
COPY target/ticket-reservation-api-0.0.1-SNAPSHOT.jar app.jar
ENTRYPOINT ["java", "-jar", "app.jar"]

What does this image contain?

• The base image provides the Java 25 runtime required by the existing build.
• The working directory gives the application a fixed location inside the container.
• The copy instruction places the packaged JAR at app.jar.
• The entry point starts that JAR whenever the container runs.

• Save Dockerfile.
• Create the JAR expected by the container image by running:

mvn package

What does Maven package produce?

Maven runs the preceding lifecycle phases before packaging the application. The Spring Boot plugin produces the executable JAR referenced by Dockerfile.

You should see a successful package run. The Explorer sidebar should now contain target/ticket-reservation-api-0.0.1-SNAPSHOT.jar.

JAR missing after packaging?

Read the first failed Maven test or build task in the terminal. The JAR is created only after the earlier lifecycle phases complete successfully.

Ask for help with the failed package run:

Define the two-instance Compose stack

The distributed lab needs one shared PostgreSQL database plus one shared Redis service. Starting those dependencies first gives you a visible checkpoint before adding the two API instances.

• In the Explorer sidebar, create a file named compose.yaml at the top level of the open repository.
• Define the PostgreSQL and Redis services by pasting:

services:
  postgres:
    image: postgres:18.6-alpine3.24
    environment: {POSTGRES_DB: tickets, POSTGRES_USER: tickets, POSTGRES_PASSWORD: tickets}
    ports: ["5432:5432"]
    volumes: ["postgres-data:/var/lib/postgresql"]
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U tickets -d tickets"]
      interval: 2s
      timeout: 2s
      retries: 20
  redis:
    image: redis:8.10.2-alpine
    ports: ["6379:6379"]
volumes:
  postgres-data:

What does this first stack provide?

• PostgreSQL uses the tickets database for the existing booking data.
• The named volume keeps database data across container restarts.
• The health check lets application services wait until PostgreSQL accepts connections.
• Redis exposes its local port for the shared admission gate you will build later.

• Save compose.yaml.
• Start the two shared data services by running:

docker compose up -d

What does this command start?

Docker Compose reads compose.yaml and starts the PostgreSQL and Redis services in the background. The named PostgreSQL volume is created during this first run.

The first run can take a little longer while Docker downloads the pinned images. You should see both data services reach a running state.

Data services not starting?

Confirm Docker Desktop is running. Check that ports 5432 and 6379 are not already occupied by local services.

Ask for help with the local container startup:

The next layer runs the same application image twice. Both instances point to the shared services while exposing separate host ports for direct testing.

• In compose.yaml, place the following service definitions directly above the top-level volumes section:

  app-1:
    build: .
    environment: &app-environment
      SPRING_DATASOURCE_URL: jdbc:postgresql://postgres:5432/tickets
      SPRING_DATASOURCE_USERNAME: tickets
      SPRING_DATASOURCE_PASSWORD: tickets
      SPRING_DATA_REDIS_HOST: redis
      SPRING_DATA_REDIS_PORT: 6379
      APP_ADMISSION_MODE: ${ADMISSION_MODE:-redis}
      APP_ADMISSION_FAILUREPOLICY: ${ADMISSION_FAILURE_POLICY:-fail-open}
      APP_ADMISSION_LIMIT: ${ADMISSION_LIMIT:-8}
      APP_ADMISSION_LEASETTL: ${ADMISSION_LEASE_TTL:-2s}
      APP_ADMISSION_HOLDDELAY: ${ADMISSION_HOLD_DELAY:-25ms}
      APP_INSTANCEID: app-1
    depends_on:
      postgres: {condition: service_healthy}
    ports: ["8081:8080"]
  app-2:
    build: .
    environment:
      <<: *app-environment
      APP_INSTANCEID: app-2
    depends_on:
      postgres: {condition: service_healthy}
    ports: ["8082:8080"]

How do the two instances differ?

• Both services build the same Dockerfile.
• Both services connect to the shared PostgreSQL and Redis service names.
• The YAML anchor keeps the shared environment settings in one place.
• The instance identifiers distinguish the two processes.
• Host ports 8081 and 8082 let you target each JVM directly.

• Save compose.yaml.

Building two images can take a few minutes on the first run because Docker needs the Java runtime layer. The two services reuse the same application image once that layer is available.

• Build the application image and start both API instances by running:

docker compose up -d --build

What does this build prove?

Compose builds the JAR image from the repository. It starts one copy on port 8081 and another copy on port 8082.

You should see PostgreSQL, Redis, and both application services reach a running state. The same Part 1 API is now available as two local processes.

Application containers stopping?

Confirm that target/ticket-reservation-api-0.0.1-SNAPSHOT.jar exists before the image build. Check that the database service reaches its healthy state before the applications start.

Ask for help with the two-instance build:

The load generator belongs behind a Compose profile. Normal stack startup leaves it inactive until you deliberately run a ticket-drop workload.

• In compose.yaml, place the optional load service directly above the top-level volumes section:

  k6:
    image: grafana/k6:2.3.0
    profiles: [load]
    working_dir: /scripts
    volumes: ["./load:/scripts:ro"]
    depends_on: [app-1, app-2]
    command: ["run", "ticket-drop.js"]

Why use a profile for k6?

The load profile keeps Grafana k6 out of normal startup. The service mounts the future workload from load as read-only data.

This prepares a repeatable load runner without requiring a host installation. The workload file arrives in the measurement step.

• Save compose.yaml.
• Confirm normal startup still excludes the profiled load service by running:

docker compose up -d

What should remain active?

Compose keeps PostgreSQL, Redis, app-1, and app-2 running. The k6 service stays inactive until its load profile is requested.

✔️ Awesome, I've got everything!

Great. Save pom.xml, Dockerfile, compose.yaml, and application.properties before the final verification.

ⓧ I'd like to double check the full code

Compare each file with the complete setup below.

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <parent><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-parent</artifactId><version>4.1.1</version><relativePath/></parent>
  <groupId>com.nextwork</groupId><artifactId>ticket-reservation-api</artifactId><version>0.0.1-SNAPSHOT</version>
  <properties><java.version>25</java.version><flyway.version>13.10.0</flyway.version><postgresql.version>42.7.14</postgresql.version><testcontainers.version>2.0.5</testcontainers.version></properties>
  <dependencies>
    <dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-web</artifactId></dependency>
    <dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-data-jpa</artifactId></dependency>
    <dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-data-redis</artifactId></dependency>
    <dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-flyway</artifactId></dependency>
    <dependency><groupId>org.flywaydb</groupId><artifactId>flyway-database-postgresql</artifactId><version>${flyway.version}</version></dependency>
    <dependency><groupId>org.postgresql</groupId><artifactId>postgresql</artifactId><version>${postgresql.version}</version><scope>runtime</scope></dependency>
    <dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-test</artifactId><scope>test</scope></dependency>
    <dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-testcontainers</artifactId><scope>test</scope></dependency>
    <dependency><groupId>org.testcontainers</groupId><artifactId>testcontainers-postgresql</artifactId><version>${testcontainers.version}</version><scope>test</scope></dependency>
    <dependency><groupId>org.testcontainers</groupId><artifactId>testcontainers-junit-jupiter</artifactId><version>${testcontainers.version}</version><scope>test</scope></dependency>
  </dependencies>
  <build><plugins><plugin><groupId>org.springframework.boot</groupId><artifactId>spring-boot-maven-plugin</artifactId></plugin></plugins></build>
</project>

What to compare in pom.xml

Confirm the existing version pins remain unchanged. The Redis starter should appear between the JPA and Flyway starters.

Dockerfile

FROM eclipse-temurin:25-jre
WORKDIR /app
COPY target/ticket-reservation-api-0.0.1-SNAPSHOT.jar app.jar
ENTRYPOINT ["java", "-jar", "app.jar"]

What to compare in Dockerfile

Confirm the image uses the Java 25 runtime. The copied filename must match the JAR produced by Maven.

compose.yaml

services:
  postgres:
    image: postgres:18.6-alpine3.24
    environment: {POSTGRES_DB: tickets, POSTGRES_USER: tickets, POSTGRES_PASSWORD: tickets}
    ports: ["5432:5432"]
    volumes: ["postgres-data:/var/lib/postgresql"]
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U tickets -d tickets"]
      interval: 2s
      timeout: 2s
      retries: 20
  redis:
    image: redis:8.10.2-alpine
    ports: ["6379:6379"]
  app-1:
    build: .
    environment: &app-environment
      SPRING_DATASOURCE_URL: jdbc:postgresql://postgres:5432/tickets
      SPRING_DATASOURCE_USERNAME: tickets
      SPRING_DATASOURCE_PASSWORD: tickets
      SPRING_DATA_REDIS_HOST: redis
      SPRING_DATA_REDIS_PORT: 6379
      APP_ADMISSION_MODE: ${ADMISSION_MODE:-redis}
      APP_ADMISSION_FAILUREPOLICY: ${ADMISSION_FAILURE_POLICY:-fail-open}
      APP_ADMISSION_LIMIT: ${ADMISSION_LIMIT:-8}
      APP_ADMISSION_LEASETTL: ${ADMISSION_LEASE_TTL:-2s}
      APP_ADMISSION_HOLDDELAY: ${ADMISSION_HOLD_DELAY:-25ms}
      APP_INSTANCEID: app-1
    depends_on:
      postgres: {condition: service_healthy}
    ports: ["8081:8080"]
  app-2:
    build: .
    environment:
      <<: *app-environment
      APP_INSTANCEID: app-2
    depends_on:
      postgres: {condition: service_healthy}
    ports: ["8082:8080"]
  k6:
    image: grafana/k6:2.3.0
    profiles: [load]
    working_dir: /scripts
    volumes: ["./load:/scripts:ro"]
    depends_on: [app-1, app-2]
    command: ["run", "ticket-drop.js"]
volumes:
  postgres-data:

What to compare in compose.yaml

Confirm the two API services share their database and Redis settings. Check that their host ports differ.

Confirm the k6 service has the load profile. The PostgreSQL volume declaration should remain at the top level.

src/main/resources/application.properties

spring.application.name=ticket-reservation-api
spring.datasource.url=jdbc:postgresql://localhost:5432/tickets
spring.datasource.username=tickets
spring.datasource.password=tickets
spring.jpa.hibernate.ddl-auto=validate
spring.data.redis.host=localhost
spring.data.redis.port=6379
spring.data.redis.connect-timeout=200ms
spring.data.redis.timeout=200ms
app.instance-id=local
app.admission.mode=off
app.admission.failure-policy=fail-open
app.admission.limit=8
app.admission.lease-ttl=2s
app.admission.hold-delay=0ms

What to compare in application.properties

Confirm both Redis timeouts are bounded at 200ms. Admission should remain off for the local Part 1 test path.

Before you run the final checks, do you expect the disabled admission mode to preserve every original Part 1 test?

• Run the complete test suite from the repository directory by running:

mvn test

What should the tests confirm?

The original Part 1 tests should complete successfully. This proves that the Redis dependency and configuration have not replaced the PostgreSQL correctness core.

• Build the application JAR after the successful test run by running:

mvn package

What should the package check confirm?

The package run should complete successfully. You should see target/ticket-reservation-api-0.0.1-SNAPSHOT.jar in the repository.

That is the local foundation complete. Your original tests still pass while two containerized API instances can share PostgreSQL and Redis.

Your two-instance stack is ready for a stronger retry contract. Next, you will make confirmation idempotency part of the authoritative PostgreSQL transaction.

Add Transactional Idempotency

Your local stack now runs the Part 1 booking transaction against PostgreSQL. A retry can reach another API instance after the first confirmation succeeds.

Transactional idempotency records the request inside the purchase transaction. The API can replay a completed result without creating another reservation.

In this step, get ready to:

• Add a database record for each confirmation key.
• Claim each key inside the existing booking transaction.
• Prove replay safety through concurrent integration tests.

Create the idempotency schema

A Flyway migration gives every idempotency key one authoritative row. The row stores the request fingerprint plus the committed response.

• Create `V2__add_idempotency_and_load_seats.sql` inside `src/main/resources/db/migration` from the VS Code file sidebar.
• Paste the migration into `V2__add_idempotency_and_load_seats.sql`:

CREATE TABLE idempotency_records (
  idempotency_key VARCHAR(100) PRIMARY KEY,
  request_fingerprint VARCHAR(160) NOT NULL,
  response_status INTEGER,
  reservation_id UUID REFERENCES reservations(id),
  created_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
  completed_at TIMESTAMPTZ,
  CONSTRAINT ck_idempotency_completion CHECK (
    (response_status IS NULL AND reservation_id IS NULL AND completed_at IS NULL)
    OR (response_status IS NOT NULL AND completed_at IS NOT NULL)
  )
);
INSERT INTO seats (id, event_id, label, state)
SELECT md5('load-seat-' || n)::uuid,
       '20000000-0000-0000-0000-000000000001'::uuid,
       'L' || lpad(n::text, 4, '0'),
       'AVAILABLE'
FROM generate_series(1, 1000) AS n;

What does this migration do?

• The `idempotency_key` primary key allows one row for each client key.
• The `request_fingerprint` identifies the hold associated with the key.
• The completion constraint keeps unfinished rows separate from stored outcomes.
• The generated `L0001` through `L1000` seats provide repeatable input for the later load test.

• Save `src/main/resources/db/migration/V2__add_idempotency_and_load_seats.sql`.
• Return to the repository terminal from the previous step.
• Apply the migration through the existing test lifecycle by running:

mvn test

Your existing tests should pass. This confirms that the new schema preserves the Part 1 booking behavior.

Does the migration fail?

Check that the filename starts with `V2__`. Confirm that both underscores appear before `add_idempotency_and_load_seats`.

Compare every table name with the migration above. A mismatched reservation table name prevents the foreign key from being created.

Ask for help with the migration failure.

Spring Data JPA needs an entity that mirrors the new table. The entity also owns the transition from an unfinished claim to a stored result.

• Create the `src/main/java/com/nextwork/ticketing/idempotency` folder from the VS Code file sidebar.

You should see the new `idempotency` package beside the existing booking packages.

• Create `IdempotencyRecord.java` inside `src/main/java/com/nextwork/ticketing/idempotency`.
• Paste the entity into `IdempotencyRecord.java`:

package com.nextwork.ticketing.idempotency;
import java.time.Instant;
import java.util.UUID;
import jakarta.persistence.*;
@Entity
@Table(name = "idempotency_records")
public class IdempotencyRecord {
 @Id @Column(name = "idempotency_key", nullable = false, length = 100) private String key;
 @Column(name = "request_fingerprint", nullable = false, length = 160) private String requestFingerprint;
 @Column(name = "response_status") private Integer responseStatus;
 @Column(name = "reservation_id") private UUID reservationId;
 @Column(name = "created_at", nullable = false) private Instant createdAt;
 @Column(name = "completed_at") private Instant completedAt;
 protected IdempotencyRecord() {}
 public String getRequestFingerprint() { return requestFingerprint; }
 public Integer getResponseStatus() { return responseStatus; }
 public UUID getReservationId() { return reservationId; }
 public boolean isCompleted() { return responseStatus != null; }
 public void complete(int status, UUID completedReservationId, Instant time) {
  responseStatus = status; reservationId = completedReservationId; completedAt = time;
 }
}

What does this entity track?

• The `key` field maps to the table primary key.
• The fingerprint ties that key to one confirmation request.
• The `isCompleted()` method distinguishes a claim from a committed outcome.
• The `complete()` method stores the response status plus the reservation reference.

• Save `src/main/java/com/nextwork/ticketing/idempotency/IdempotencyRecord.java`.
• Verify the entity mapping through the existing test lifecycle by running:

mvn test

Your tests should pass again. Hibernate validation now recognizes the entity as a match for `idempotency_records`.

Does entity validation fail?

Check that `@Table` uses `idempotency_records`. Confirm that each `@Column` value matches the migration exactly.

Ask for help with the mapping error.

The repository claims a key with one database statement. PostgreSQL waits on a concurrent unique-index decision before reporting whether the insert succeeded.

• Create `IdempotencyRecordRepository.java` inside `src/main/java/com/nextwork/ticketing/idempotency`.
• Paste the repository into `IdempotencyRecordRepository.java`:

package com.nextwork.ticketing.idempotency;
import org.springframework.data.jpa.repository.*;
import org.springframework.data.repository.query.Param;
public interface IdempotencyRecordRepository extends JpaRepository<IdempotencyRecord, String> {
 @Modifying
 @Query(value = """
   INSERT INTO idempotency_records (idempotency_key, request_fingerprint)
   VALUES (:key, :fingerprint)
   ON CONFLICT (idempotency_key) DO NOTHING
   """, nativeQuery = true)
 int insertIfAbsent(@Param("key") String key, @Param("fingerprint") String fingerprint);
}

How does the claim work?

`INSERT ... ON CONFLICT DO NOTHING` returns `1` when this transaction claims the key. It returns `0` when the key already belongs to another request.

The unique primary key serializes concurrent claims. The reservation and its idempotency result share one commit boundary.

• Save `src/main/java/com/nextwork/ticketing/idempotency/IdempotencyRecordRepository.java`.
• Confirm that the repository loads through the application context by running:

mvn test

Your original Part 1 tests should remain green. The database layer is ready to claim confirmation keys.

Does the repository query fail?

Check that `@Modifying` sits directly above `@Query`. Confirm that the query parameter names match `key` plus `fingerprint`.

Ask for help with the native query.

Claim and replay confirmation keys

The idempotency claim belongs inside the existing `@Transactional` confirmation method. A failed confirmation rolls back the reservation plus the key claim.

The fingerprint uses `confirm:<holdId>`. Reusing the key for another hold therefore becomes a conflict instead of an incorrect replay.

• Switch back to `src/main/java/com/nextwork/ticketing/booking/ReservationService.java`.
• Replace its import group with the following imports:

import java.time.*;
import java.util.*;
import com.nextwork.ticketing.idempotency.*;
import com.nextwork.ticketing.reservation.*;
import com.nextwork.ticketing.seat.*;
import org.springframework.http.HttpStatus;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.web.server.ResponseStatusException;

Why add the idempotency package?

The service now coordinates the idempotency record with the existing seat plus reservation repositories. `@Transactional` keeps those writes in one PostgreSQL transaction.

• Replace the repository fields plus constructor in `ReservationService.java` with this version:

 private final SeatRepository seats; private final ReservationRepository reservations;
 private final IdempotencyRecordRepository idempotency; private final Clock clock;
 public ReservationService(SeatRepository seats, ReservationRepository reservations, IdempotencyRecordRepository idempotency, Clock clock) {
  this.seats = seats; this.reservations = reservations; this.idempotency = idempotency; this.clock = clock;
 }

What changes in the constructor?

Spring injects `IdempotencyRecordRepository` beside the existing repositories. The service can now claim or read a key without moving seat ownership outside PostgreSQL.

• Save `src/main/java/com/nextwork/ticketing/booking/ReservationService.java`.
• Check that the new dependency compiles by running:

mvn test

Your tests should pass. This proves the new repository is available to `ReservationService`.

Does constructor injection fail?

Check that the constructor parameter uses `IdempotencyRecordRepository`. Confirm that the assignment stores it in the `idempotency` field.

Ask for help with the service dependency.

• Replace the existing confirmation method in `ReservationService.java` with this transactional version:

 @Transactional public ConfirmationResult confirm(UUID holdId, String key) {
  if (key == null || key.isBlank() || key.length() > 100)
   throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "Idempotency-Key must contain 1 to 100 characters");
  Instant now = clock.instant(); String fingerprint = "confirm:" + holdId;
  boolean claimed = idempotency.insertIfAbsent(key, fingerprint) == 1;
  IdempotencyRecord record = idempotency.findById(key).orElseThrow(() -> new IllegalStateException("Idempotency claim disappeared"));
  if (!fingerprint.equals(record.getRequestFingerprint()))
   throw new ResponseStatusException(HttpStatus.CONFLICT, "Idempotency-Key was used for another request");
  if (record.isCompleted()) {
   if (record.getResponseStatus() == 410) return new ConfirmationResult(null, true, true);
   Reservation replay = reservations.findById(record.getReservationId()).orElseThrow(() -> new IllegalStateException("Completed reservation is missing"));
   return new ConfirmationResult(replay, false, true);
  }
  if (!claimed) throw new ResponseStatusException(HttpStatus.CONFLICT, "Idempotency request is still in progress");
  Seat seat = seats.findByHoldIdForUpdate(holdId).orElseThrow(() -> new ResponseStatusException(HttpStatus.NOT_FOUND, "Hold not found"));
  if (seat.isExpired(now)) {
   seat.release(); seats.save(seat); record.complete(410, null, now); idempotency.save(record);
   return new ConfirmationResult(null, true, false);
  }
  Reservation reservation = new Reservation(UUID.randomUUID(), seat.getId(), seat.getHeldBy(), now);
  reservations.save(reservation); seat.confirm(); seats.save(seat);
  record.complete(201, reservation.getId(), now); idempotency.save(record);
  return new ConfirmationResult(reservation, false, false);
 }

How does confirmation become idempotent?

• The method validates that the client key contains between 1 and 100 characters.
• The fingerprint binds the key to one hold confirmation.
• A completed record replays the stored reservation or expired outcome.
• Any runtime failure rolls back the key claim with the booking transaction.

Legacy Part 1 clients may omit the new header. The controller gives those calls a generated fallback key while distributed clients send a stable `Idempotency-Key`.

• Switch back to `src/main/java/com/nextwork/ticketing/booking/TicketController.java`.
• Replace the file with this controller version:

package com.nextwork.ticketing.booking;
import java.util.*;
import com.nextwork.ticketing.booking.ApiModels.*;
import org.springframework.http.*;
import org.springframework.web.bind.annotation.*;
@RestController
@RequestMapping("/api")
public class TicketController {
 private final ReservationService service;
 public TicketController(ReservationService service) { this.service = service; }
 @GetMapping("/events/{eventId}/seats") public List<SeatView> seats(@PathVariable UUID eventId) {
  return service.listSeats(eventId).stream().map(s -> SeatView.from(s, service.now())).toList();
 }
 @PostMapping("/events/{eventId}/holds") public ResponseEntity<HoldResponse> hold(@PathVariable UUID eventId, @RequestBody HoldRequest request) {
  var result = service.createHold(eventId, request.seatId(), request.buyerId());
  return new ResponseEntity<>(new HoldResponse(result.holdId(), result.seatId(), result.buyerId(), result.expiresAt()), HttpStatus.CREATED);
 }
 @PostMapping("/holds/{holdId}/confirm") public ResponseEntity<?> confirm(@PathVariable UUID holdId,
   @RequestHeader(value = "Idempotency-Key", required = false) String suppliedKey) {
  String key = suppliedKey == null || suppliedKey.isBlank() ? "legacy-" + UUID.randomUUID() : suppliedKey;
  var result = service.confirm(holdId, key);
  var builder = ResponseEntity.status(result.expired() ? HttpStatus.GONE : HttpStatus.CREATED)
   .header("Idempotency-Replayed", Boolean.toString(result.replayed()));
  return result.expired() ? builder.body(new StatusResponse("EXPIRED")) : builder.body(ReservationView.from(result.reservation()));
 }
 @DeleteMapping("/holds/{holdId}") public ResponseEntity<Void> release(@PathVariable UUID holdId) { service.release(holdId); return ResponseEntity.noContent().build(); }
 @GetMapping("/reservations/{id}") public ReservationView reservation(@PathVariable UUID id) { return ReservationView.from(service.getReservation(id)); }
}

What changes at the HTTP boundary?

`@RequestHeader` accepts the caller's `Idempotency-Key`. The fallback preserves compatibility for Part 1 callers that omit it.

The `Idempotency-Replayed` response header tells the client whether PostgreSQL returned a stored result. Distributed clients must send their own stable key.

• Save `src/main/java/com/nextwork/ticketing/booking/ReservationService.java`.
• Save `src/main/java/com/nextwork/ticketing/booking/TicketController.java`.
• Verify the updated endpoint contract by running:

mvn test

The original Part 1 tests should pass. Confirmation now requires a key internally while the controller preserves legacy requests.

Does the controller fail to compile?

Check that `service.confirm` receives both `holdId` plus `key`. Confirm that the response header reads `result.replayed()`.

Ask for help with the endpoint update.

Prove replay and rollback safety

A single request cannot prove concurrent correctness. JUnit 5 plus Testcontainers can send simultaneous HTTP requests against a real PostgreSQL container.

• Create `IdempotencyIntegrationTest.java` inside `src/test/java/com/nextwork/ticketing`.
• Paste this test foundation into `IdempotencyIntegrationTest.java`:

package com.nextwork.ticketing;
import static org.junit.jupiter.api.Assertions.*;
import java.net.URI;
import java.net.http.*;
import java.util.*;
import java.util.concurrent.*;
import java.util.regex.*;
import org.junit.jupiter.api.*;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.*;
import org.springframework.boot.test.web.server.LocalServerPort;
import org.springframework.boot.testcontainers.service.connection.ServiceConnection;
import org.springframework.jdbc.core.JdbcTemplate;
import org.testcontainers.containers.PostgreSQLContainer;
import org.testcontainers.junit.jupiter.*;
@Testcontainers
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
class IdempotencyIntegrationTest {
 static final UUID EVENT = UUID.fromString("20000000-0000-0000-0000-000000000001");
 static final UUID A1 = UUID.fromString("10000000-0000-0000-0000-000000000001");
 static final UUID A2 = UUID.fromString("10000000-0000-0000-0000-000000000002");
 @Container @ServiceConnection static final PostgreSQLContainer<?> POSTGRES = new PostgreSQLContainer<>("postgres:18.6-alpine3.24");
 @LocalServerPort int port; @Autowired JdbcTemplate jdbc; final HttpClient client = HttpClient.newHttpClient();
 @BeforeEach void reset() {
  jdbc.update("DELETE FROM idempotency_records"); jdbc.update("DELETE FROM reservations");
  jdbc.update("UPDATE seats SET state='AVAILABLE', hold_id=NULL, held_by=NULL, hold_expires_at=NULL, version=0");
 }
}

What does the test foundation provide?

• Testcontainers starts PostgreSQL with the same image used by the local stack.
• The random web port lets each test call the real controller.
• The reset method restores the seat plus reservation tables before each scenario.

• Save `src/test/java/com/nextwork/ticketing/IdempotencyIntegrationTest.java`.
• Confirm that the empty test class starts its PostgreSQL container by running:

mvn test

Your test suite should pass after starting the PostgreSQL test container. The new class is connected to the migrated schema.

Does the test container fail to start?

Confirm that Docker Desktop is running. Check that the test image matches `postgres:18.6-alpine3.24`.

Ask for help with the container startup.

The first helpers create a hold through HTTP. They also submit a confirmation with a caller-supplied key.

• Insert the `createHold` helper before the final closing brace in `IdempotencyIntegrationTest.java`:

 UUID createHold(UUID seat, String buyer) throws Exception {
  String body = "{\"seatId\":\"%s\",\"buyerId\":\"%s\"}".formatted(seat, buyer);
  HttpRequest request = HttpRequest.newBuilder(URI.create("http://localhost:" + port + "/api/events/" + EVENT + "/holds"))
   .header("Content-Type", "application/json").POST(HttpRequest.BodyPublishers.ofString(body)).build();
  HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
  Matcher matcher = Pattern.compile("\"holdId\":\"([^\"]+)\"").matcher(response.body()); assertTrue(matcher.find());
  return UUID.fromString(matcher.group(1));
 }

What does createHold do?

The helper sends a hold request to the running test server. It extracts the generated hold identifier from the response body.

• Insert the `post` helper below `createHold`:

 HttpResponse<String> post(String path, String key) throws Exception {
  HttpRequest request = HttpRequest.newBuilder(URI.create("http://localhost:" + port + path))
   .header("Content-Type", "application/json").header("Idempotency-Key", key)
   .POST(HttpRequest.BodyPublishers.ofString("{}")).build();
  return client.send(request, HttpResponse.BodyHandlers.ofString());
 }

What does post do?

The helper submits a confirmation to any supplied path. It sends the same key format that distributed clients use.

• Save `src/test/java/com/nextwork/ticketing/IdempotencyIntegrationTest.java`.
• Check that both HTTP helpers compile by running:

mvn test

Your suite should pass. The test class can now create holds plus send keyed confirmations.

Virtual threads let the test release two confirmations at almost the same time. PostgreSQL decides which transaction owns the unique key.

• Insert the `concurrent` helper below `post`:

 List<HttpResponse<String>> concurrent(String path, String key) throws Exception {
  try (ExecutorService executor = Executors.newVirtualThreadPerTaskExecutor()) {
   Future<HttpResponse<String>> first = executor.submit(() -> post(path, key));
   Future<HttpResponse<String>> second = executor.submit(() -> post(path, key));
   return List.of(first.get(), second.get());
  }
 }

How does the race start?

Two virtual-thread tasks submit the same path plus key. Waiting on both futures gives the test both HTTP outcomes.

• Insert the concurrent replay test below `reset`:

 @Test void concurrentSameKeyReplaysOnePurchase() throws Exception {
  UUID hold = createHold(A1, "buyer-a"); String path = "/api/holds/" + hold + "/confirm";
  List<HttpResponse<String>> responses = concurrent(path, "same-key");
  assertEquals(List.of(201, 201), responses.stream().map(HttpResponse::statusCode).sorted().toList());
  assertEquals(1, jdbc.queryForObject("SELECT COUNT(*) FROM reservations WHERE seat_id=?", Integer.class, A1));
  assertEquals(1, jdbc.queryForObject("SELECT COUNT(*) FROM idempotency_records WHERE idempotency_key='same-key'", Integer.class));
 }

What does the replay test prove?

Both callers receive a successful confirmation. Direct database assertions prove that those responses share one reservation plus one idempotency row.

• Save `src/test/java/com/nextwork/ticketing/IdempotencyIntegrationTest.java`.

Before you run the suite, do you expect two successful responses to create one reservation or two reservations?

• Run the concurrent replay test through the full suite by running:

mvn test

The test should pass with two `201` responses. PostgreSQL should contain one reservation plus one completed idempotency row.

Does concurrent replay fail?

Check that `insertIfAbsent` runs inside the same `@Transactional` method as the reservation write. Confirm that completed records return the stored reservation.

Ask for help with the race result.

Two boundary cases complete the proof. A mismatched fingerprint must return `409`. A rolled-back attempt must leave its key available.

• Insert the fingerprint mismatch test below the concurrent replay test:

 @Test void sameKeyWithDifferentFingerprintIsRejected() throws Exception {
  UUID first = createHold(A1, "buyer-a"), second = createHold(A2, "buyer-b");
  assertEquals(201, post("/api/holds/" + first + "/confirm", "shared-key").statusCode());
  assertEquals(409, post("/api/holds/" + second + "/confirm", "shared-key").statusCode());
  assertEquals(1, jdbc.queryForObject("SELECT COUNT(*) FROM reservations", Integer.class));
 }

What does the mismatch test prove?

The same key succeeds for the first hold. Reusing it for a different fingerprint returns `409` without creating another reservation.

• Insert the rollback safety test below the mismatch test:

 @Test void failedTransactionDoesNotConsumeTheKey() throws Exception {
  String key = "rollback-safe";
  assertEquals(404, post("/api/holds/" + UUID.randomUUID() + "/confirm", key).statusCode());
  assertEquals(0, jdbc.queryForObject("SELECT COUNT(*) FROM idempotency_records WHERE idempotency_key=?", Integer.class, key));
  assertEquals(201, post("/api/holds/" + createHold(A1, "buyer-a") + "/confirm", key).statusCode());
 }

What does the rollback test prove?

The missing hold causes the transaction to fail after claiming the key. The following database assertion proves that rollback removed the claim.

A valid confirmation can then reuse `rollback-safe`. Failed work does not poison future retries.

• Save `src/test/java/com/nextwork/ticketing/IdempotencyIntegrationTest.java`.

Before the final run, which outcome should distinguish a safe retry from accidental key reuse?

• Run the complete test suite by running:

mvn test

You should see the full test suite pass. The concurrent scenario produces two `201` responses backed by one reservation plus one idempotency row.

The mismatch scenario returns `409`. The rollback scenario reuses its key successfully after the failed `404` attempt.

Does the final suite fail?

Use the failing test name to isolate the broken branch. A replay failure points to completed-record handling. A rollback failure points to transaction boundaries.

Ask for help with the failed assertion.

✔️ Awesome, I've got everything!

Strong work. Your confirmation transaction now replays retries without duplicating reservations.

ⓧ I'd like to double check the full code

Compare each changed file with the complete versions below:

`src/main/resources/db/migration/V2__add_idempotency_and_load_seats.sql`:

CREATE TABLE idempotency_records (
  idempotency_key VARCHAR(100) PRIMARY KEY,
  request_fingerprint VARCHAR(160) NOT NULL,
  response_status INTEGER,
  reservation_id UUID REFERENCES reservations(id),
  created_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
  completed_at TIMESTAMPTZ,
  CONSTRAINT ck_idempotency_completion CHECK (
    (response_status IS NULL AND reservation_id IS NULL AND completed_at IS NULL)
    OR (response_status IS NOT NULL AND completed_at IS NOT NULL)
  )
);
INSERT INTO seats (id, event_id, label, state)
SELECT md5('load-seat-' || n)::uuid,
       '20000000-0000-0000-0000-000000000001'::uuid,
       'L' || lpad(n::text, 4, '0'),
       'AVAILABLE'
FROM generate_series(1, 1000) AS n;

`src/main/java/com/nextwork/ticketing/idempotency/IdempotencyRecord.java`:

package com.nextwork.ticketing.idempotency;
import java.time.Instant;
import java.util.UUID;
import jakarta.persistence.*;
@Entity
@Table(name = "idempotency_records")
public class IdempotencyRecord {
 @Id @Column(name = "idempotency_key", nullable = false, length = 100) private String key;
 @Column(name = "request_fingerprint", nullable = false, length = 160) private String requestFingerprint;
 @Column(name = "response_status") private Integer responseStatus;
 @Column(name = "reservation_id") private UUID reservationId;
 @Column(name = "created_at", nullable = false) private Instant createdAt;
 @Column(name = "completed_at") private Instant completedAt;
 protected IdempotencyRecord() {}
 public String getRequestFingerprint() { return requestFingerprint; }
 public Integer getResponseStatus() { return responseStatus; }
 public UUID getReservationId() { return reservationId; }
 public boolean isCompleted() { return responseStatus != null; }
 public void complete(int status, UUID completedReservationId, Instant time) {
  responseStatus = status; reservationId = completedReservationId; completedAt = time;
 }
}

`src/main/java/com/nextwork/ticketing/idempotency/IdempotencyRecordRepository.java`:

package com.nextwork.ticketing.idempotency;
import org.springframework.data.jpa.repository.*;
import org.springframework.data.repository.query.Param;
public interface IdempotencyRecordRepository extends JpaRepository<IdempotencyRecord, String> {
 @Modifying
 @Query(value = """
   INSERT INTO idempotency_records (idempotency_key, request_fingerprint)
   VALUES (:key, :fingerprint)
   ON CONFLICT (idempotency_key) DO NOTHING
   """, nativeQuery = true)
 int insertIfAbsent(@Param("key") String key, @Param("fingerprint") String fingerprint);
}

`src/main/java/com/nextwork/ticketing/booking/ReservationService.java`:

package com.nextwork.ticketing.booking;
import java.time.*;
import java.util.*;
import com.nextwork.ticketing.idempotency.*;
import com.nextwork.ticketing.reservation.*;
import com.nextwork.ticketing.seat.*;
import org.springframework.http.HttpStatus;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.web.server.ResponseStatusException;
@Service
public class ReservationService {
 private static final Duration HOLD_DURATION = Duration.ofSeconds(30);
 private final SeatRepository seats; private final ReservationRepository reservations;
 private final IdempotencyRecordRepository idempotency; private final Clock clock;
 public ReservationService(SeatRepository seats, ReservationRepository reservations, IdempotencyRecordRepository idempotency, Clock clock) {
  this.seats = seats; this.reservations = reservations; this.idempotency = idempotency; this.clock = clock;
 }
 @Transactional(readOnly = true) public List<Seat> listSeats(UUID eventId) { return seats.findByEventIdOrderByLabel(eventId); }
 @Transactional public HoldResult createHold(UUID eventId, UUID seatId, String buyerId) {
  Instant now = clock.instant();
  Seat seat = seats.findForUpdate(eventId, seatId).orElseThrow(() -> new ResponseStatusException(HttpStatus.NOT_FOUND, "Seat not found"));
  if (!seat.isAvailableAt(now)) throw new ResponseStatusException(HttpStatus.CONFLICT, "Seat is not available");
  seat.expireIfNeeded(now); UUID holdId = UUID.randomUUID(); Instant expiresAt = now.plus(HOLD_DURATION);
  seat.placeHold(holdId, buyerId, expiresAt); seats.save(seat);
  return new HoldResult(holdId, seatId, buyerId, expiresAt);
 }
 @Transactional public ConfirmationResult confirm(UUID holdId, String key) {
  if (key == null || key.isBlank() || key.length() > 100)
   throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "Idempotency-Key must contain 1 to 100 characters");
  Instant now = clock.instant(); String fingerprint = "confirm:" + holdId;
  boolean claimed = idempotency.insertIfAbsent(key, fingerprint) == 1;
  IdempotencyRecord record = idempotency.findById(key).orElseThrow(() -> new IllegalStateException("Idempotency claim disappeared"));
  if (!fingerprint.equals(record.getRequestFingerprint()))
   throw new ResponseStatusException(HttpStatus.CONFLICT, "Idempotency-Key was used for another request");
  if (record.isCompleted()) {
   if (record.getResponseStatus() == 410) return new ConfirmationResult(null, true, true);
   Reservation replay = reservations.findById(record.getReservationId()).orElseThrow(() -> new IllegalStateException("Completed reservation is missing"));
   return new ConfirmationResult(replay, false, true);
  }
  if (!claimed) throw new ResponseStatusException(HttpStatus.CONFLICT, "Idempotency request is still in progress");
  Seat seat = seats.findByHoldIdForUpdate(holdId).orElseThrow(() -> new ResponseStatusException(HttpStatus.NOT_FOUND, "Hold not found"));
  if (seat.isExpired(now)) {
   seat.release(); seats.save(seat); record.complete(410, null, now); idempotency.save(record);
   return new ConfirmationResult(null, true, false);
  }
  Reservation reservation = new Reservation(UUID.randomUUID(), seat.getId(), seat.getHeldBy(), now);
  reservations.save(reservation); seat.confirm(); seats.save(seat);
  record.complete(201, reservation.getId(), now); idempotency.save(record);
  return new ConfirmationResult(reservation, false, false);
 }
 @Transactional public void release(UUID holdId) {
  Seat seat = seats.findByHoldIdForUpdate(holdId).orElseThrow(() -> new ResponseStatusException(HttpStatus.NOT_FOUND, "Hold not found"));
  seat.release(); seats.save(seat);
 }
 @Transactional(readOnly = true) public Reservation getReservation(UUID id) {
  return reservations.findById(id).orElseThrow(() -> new ResponseStatusException(HttpStatus.NOT_FOUND, "Reservation not found"));
 }
 public Instant now() { return clock.instant(); }
 public record HoldResult(UUID holdId, UUID seatId, String buyerId, Instant expiresAt) {}
 public record ConfirmationResult(Reservation reservation, boolean expired, boolean replayed) {}
}

`src/main/java/com/nextwork/ticketing/booking/TicketController.java`:

package com.nextwork.ticketing.booking;
import java.util.*;
import com.nextwork.ticketing.booking.ApiModels.*;
import org.springframework.http.*;
import org.springframework.web.bind.annotation.*;
@RestController
@RequestMapping("/api")
public class TicketController {
 private final ReservationService service;
 public TicketController(ReservationService service) { this.service = service; }
 @GetMapping("/events/{eventId}/seats") public List<SeatView> seats(@PathVariable UUID eventId) {
  return service.listSeats(eventId).stream().map(s -> SeatView.from(s, service.now())).toList();
 }
 @PostMapping("/events/{eventId}/holds") public ResponseEntity<HoldResponse> hold(@PathVariable UUID eventId, @RequestBody HoldRequest request) {
  var result = service.createHold(eventId, request.seatId(), request.buyerId());
  return new ResponseEntity<>(new HoldResponse(result.holdId(), result.seatId(), result.buyerId(), result.expiresAt()), HttpStatus.CREATED);
 }
 @PostMapping("/holds/{holdId}/confirm") public ResponseEntity<?> confirm(@PathVariable UUID holdId,
   @RequestHeader(value = "Idempotency-Key", required = false) String suppliedKey) {
  String key = suppliedKey == null || suppliedKey.isBlank() ? "legacy-" + UUID.randomUUID() : suppliedKey;
  var result = service.confirm(holdId, key);
  var builder = ResponseEntity.status(result.expired() ? HttpStatus.GONE : HttpStatus.CREATED)
   .header("Idempotency-Replayed", Boolean.toString(result.replayed()));
  return result.expired() ? builder.body(new StatusResponse("EXPIRED")) : builder.body(ReservationView.from(result.reservation()));
 }
 @DeleteMapping("/holds/{holdId}") public ResponseEntity<Void> release(@PathVariable UUID holdId) { service.release(holdId); return ResponseEntity.noContent().build(); }
 @GetMapping("/reservations/{id}") public ReservationView reservation(@PathVariable UUID id) { return ReservationView.from(service.getReservation(id)); }
}

`src/test/java/com/nextwork/ticketing/IdempotencyIntegrationTest.java`:

package com.nextwork.ticketing;
import static org.junit.jupiter.api.Assertions.*;
import java.net.URI;
import java.net.http.*;
import java.util.*;
import java.util.concurrent.*;
import java.util.regex.*;
import org.junit.jupiter.api.*;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.*;
import org.springframework.boot.test.web.server.LocalServerPort;
import org.springframework.boot.testcontainers.service.connection.ServiceConnection;
import org.springframework.jdbc.core.JdbcTemplate;
import org.testcontainers.containers.PostgreSQLContainer;
import org.testcontainers.junit.jupiter.*;
@Testcontainers
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
class IdempotencyIntegrationTest {
 static final UUID EVENT = UUID.fromString("20000000-0000-0000-0000-000000000001");
 static final UUID A1 = UUID.fromString("10000000-0000-0000-0000-000000000001");
 static final UUID A2 = UUID.fromString("10000000-0000-0000-0000-000000000002");
 @Container @ServiceConnection static final PostgreSQLContainer<?> POSTGRES = new PostgreSQLContainer<>("postgres:18.6-alpine3.24");
 @LocalServerPort int port; @Autowired JdbcTemplate jdbc; final HttpClient client = HttpClient.newHttpClient();
 @BeforeEach void reset() {
  jdbc.update("DELETE FROM idempotency_records"); jdbc.update("DELETE FROM reservations");
  jdbc.update("UPDATE seats SET state='AVAILABLE', hold_id=NULL, held_by=NULL, hold_expires_at=NULL, version=0");
 }
 @Test void concurrentSameKeyReplaysOnePurchase() throws Exception {
  UUID hold = createHold(A1, "buyer-a"); String path = "/api/holds/" + hold + "/confirm";
  List<HttpResponse<String>> responses = concurrent(path, "same-key");
  assertEquals(List.of(201, 201), responses.stream().map(HttpResponse::statusCode).sorted().toList());
  assertEquals(1, jdbc.queryForObject("SELECT COUNT(*) FROM reservations WHERE seat_id=?", Integer.class, A1));
  assertEquals(1, jdbc.queryForObject("SELECT COUNT(*) FROM idempotency_records WHERE idempotency_key='same-key'", Integer.class));
 }
 @Test void sameKeyWithDifferentFingerprintIsRejected() throws Exception {
  UUID first = createHold(A1, "buyer-a"), second = createHold(A2, "buyer-b");
  assertEquals(201, post("/api/holds/" + first + "/confirm", "shared-key").statusCode());
  assertEquals(409, post("/api/holds/" + second + "/confirm", "shared-key").statusCode());
  assertEquals(1, jdbc.queryForObject("SELECT COUNT(*) FROM reservations", Integer.class));
 }
 @Test void failedTransactionDoesNotConsumeTheKey() throws Exception {
  String key = "rollback-safe";
  assertEquals(404, post("/api/holds/" + UUID.randomUUID() + "/confirm", key).statusCode());
  assertEquals(0, jdbc.queryForObject("SELECT COUNT(*) FROM idempotency_records WHERE idempotency_key=?", Integer.class, key));
  assertEquals(201, post("/api/holds/" + createHold(A1, "buyer-a") + "/confirm", key).statusCode());
 }
 UUID createHold(UUID seat, String buyer) throws Exception {
  String body = "{\"seatId\":\"%s\",\"buyerId\":\"%s\"}".formatted(seat, buyer);
  HttpRequest request = HttpRequest.newBuilder(URI.create("http://localhost:" + port + "/api/events/" + EVENT + "/holds"))
   .header("Content-Type", "application/json").POST(HttpRequest.BodyPublishers.ofString(body)).build();
  HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
  Matcher matcher = Pattern.compile("\"holdId\":\"([^\"]+)\"").matcher(response.body()); assertTrue(matcher.find());
  return UUID.fromString(matcher.group(1));
 }
 HttpResponse<String> post(String path, String key) throws Exception {
  HttpRequest request = HttpRequest.newBuilder(URI.create("http://localhost:" + port + path))
   .header("Content-Type", "application/json").header("Idempotency-Key", key)
   .POST(HttpRequest.BodyPublishers.ofString("{}")).build();
  return client.send(request, HttpResponse.BodyHandlers.ofString());
 }
 List<HttpResponse<String>> concurrent(String path, String key) throws Exception {
  try (ExecutorService executor = Executors.newVirtualThreadPerTaskExecutor()) {
   Future<HttpResponse<String>> first = executor.submit(() -> post(path, key));
   Future<HttpResponse<String>> second = executor.submit(() -> post(path, key));
   return List.of(first.get(), second.get());
  }
 }
}

That is the correctness boundary secured. Next, you will place admission outside this transaction and reproduce why one local semaphore cannot coordinate two JVMs.

Expose the Local Admission Failure

Your PostgreSQL transaction now turns concurrent retries into one purchase. This step tests whether one admission limit can protect that transaction across two application instances.

A Java Semaphore controls permits inside one JVM. You will place that gate outside the booking transaction before testing the limit through both application ports.

In this step, get ready to:

• Add a process-local admission lease around purchase confirmation.
• Expose the admission source in each confirmation response.
• Race both application instances before checking the database invariant.

Build the process-local gate

Admission belongs outside ReservationService.confirm() so a rejected request never starts the booking transaction. An AutoCloseable lease guarantees that every acquired permit has one release path.

• Create src/main/java/com/nextwork/ticketing/admission/AdmissionService.java from the VS Code Explorer sidebar.
• Paste this first section into AdmissionService.java to define the service boundary:

package com.nextwork.ticketing.admission;
import java.time.Duration;
import java.util.*;
import java.util.concurrent.Semaphore;
import java.util.concurrent.atomic.AtomicBoolean;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.dao.DataAccessException;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.data.redis.core.script.RedisScript;
import org.springframework.http.HttpStatus;
import org.springframework.stereotype.Service;
import org.springframework.web.server.ResponseStatusException;
@Service
public class AdmissionService {
 private static final String KEY = "admission:confirm";
 private static final RedisScript<Long> ACQUIRE = RedisScript.of("""
   local time = redis.call('TIME')
   local now = (tonumber(time[1]) * 1000) + math.floor(tonumber(time[2]) / 1000)
   redis.call('ZREMRANGEBYSCORE', KEYS[1], '-inf', now)
   if redis.call('ZCARD', KEYS[1]) >= tonumber(ARGV[2]) then return 0 end
   redis.call('ZADD', KEYS[1], now + tonumber(ARGV[1]), ARGV[3])
   redis.call('EXPIRE', KEYS[1], tonumber(ARGV[4]))
   return 1
   """, Long.class);
 private static final RedisScript<Long> RELEASE = RedisScript.of("return redis.call('ZREM', KEYS[1], ARGV[1])", Long.class);

What does this section establish?

• The @Service annotation makes the admission boundary available through Spring dependency injection.
• The imports provide the semaphore plus the lease lifecycle types used by local admission.
• The shared admission scripts stay encapsulated inside the same service for the next coordination step.

• Continue the class with its settings plus the local acquisition path by pasting this code directly below the previous section:

 private final StringRedisTemplate redis;
 private final String mode;
 private final String failurePolicy;
 private final int limit;
 private final long leaseTtlMillis;
 private final Semaphore localSemaphore;
 public AdmissionService(StringRedisTemplate redis,
   @Value("${app.admission.mode:off}") String mode,
   @Value("${app.admission.failure-policy:fail-open}") String failurePolicy,
   @Value("${app.admission.limit:8}") int limit,
   @Value("${app.admission.lease-ttl:2s}") Duration leaseTtl) {
  this.redis = redis; this.mode = mode; this.failurePolicy = failurePolicy; this.limit = limit;
  leaseTtlMillis = leaseTtl.toMillis(); localSemaphore = new Semaphore(limit);
 }
 public Lease acquire() {
  return switch (mode.toLowerCase(Locale.ROOT)) {
   case "off" -> Lease.noop("off");
   case "local" -> acquireLocal();
   case "redis" -> acquireRedis();
   default -> throw new IllegalStateException("Unknown admission mode: " + mode);
  };
 }
 private Lease acquireLocal() {
  if (!localSemaphore.tryAcquire()) throw new ResponseStatusException(HttpStatus.TOO_MANY_REQUESTS, "Local admission limit reached");
  return new Lease("local", localSemaphore::release);
 }

How does local admission work?

• The constructor creates one localSemaphore for each application process.
• The off mode returns a no-operation lease for tests that do not use admission.
• The local mode calls tryAcquire() without waiting.
• An exhausted semaphore returns HTTP status 429 before confirmation reaches the transaction.

• Finish AdmissionService.java by pasting the remaining methods below acquireLocal().

 private Lease acquireRedis() {
  String token = UUID.randomUUID().toString();
  long keyTtlSeconds = Math.max(1, ((leaseTtlMillis + 999) / 1000) * 2);
  try {
   Long admitted = redis.execute(ACQUIRE, List.of(KEY), Long.toString(leaseTtlMillis), Integer.toString(limit), token, Long.toString(keyTtlSeconds));
   if (!Long.valueOf(1).equals(admitted)) throw new ResponseStatusException(HttpStatus.TOO_MANY_REQUESTS, "Redis admission limit reached");
   return new Lease("redis", () -> releaseRedis(token));
  } catch (DataAccessException failure) {
   if ("fail-open".equalsIgnoreCase(failurePolicy)) return Lease.noop("fail-open");
   throw new ResponseStatusException(HttpStatus.SERVICE_UNAVAILABLE, "Admission service unavailable", failure);
  }
 }
 private void releaseRedis(String token) {
  try { redis.execute(RELEASE, List.of(KEY), token); } catch (DataAccessException ignored) { }
 }
 public static final class Lease implements AutoCloseable {
  private final String source; private final Runnable release; private final AtomicBoolean closed = new AtomicBoolean();
  private Lease(String source, Runnable release) { this.source = source; this.release = release; }
  static Lease noop(String source) { return new Lease(source, () -> {}); }
  public String source() { return source; }
  public void close() { if (closed.compareAndSet(false, true)) release.run(); }
 }
}

How does the lease stay safe?

• The lease stores the admission source beside its release action.
• The close() method runs that release action once.
• The AtomicBoolean prevents a repeated close from releasing another request's permit.
• This experiment exercises the local path while both JVMs hold independent service instances.

• Save AdmissionService.java.
• Create src/main/java/com/nextwork/ticketing/booking/ConfirmationFacade.java from the VS Code Explorer sidebar.
• Wrap the transactional confirmation call by pasting this code into ConfirmationFacade.java.

package com.nextwork.ticketing.booking;
import java.time.Duration;
import java.util.UUID;
import com.nextwork.ticketing.admission.AdmissionService;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.http.HttpStatus;
import org.springframework.stereotype.Service;
import org.springframework.web.server.ResponseStatusException;
@Service
public class ConfirmationFacade {
 private final AdmissionService admission; private final ReservationService reservations; private final Duration holdDelay;
 public ConfirmationFacade(AdmissionService admission, ReservationService reservations,
   @Value("${app.admission.hold-delay:0ms}") Duration holdDelay) {
  this.admission = admission; this.reservations = reservations; this.holdDelay = holdDelay;
 }
 public ConfirmationEnvelope confirm(UUID holdId, String key) {
  try (var lease = admission.acquire()) {
   try { Thread.sleep(holdDelay.toMillis()); }
   catch (InterruptedException interrupted) {
    Thread.currentThread().interrupt();
    throw new ResponseStatusException(HttpStatus.SERVICE_UNAVAILABLE, "Request interrupted", interrupted);
   }
   return new ConfirmationEnvelope(reservations.confirm(holdId, key), lease.source());
  }
 }
 public record ConfirmationEnvelope(ReservationService.ConfirmationResult result, String admissionSource) {}
}

What does the facade protect?

• The facade acquires admission before calling the transactional reservation service.
• The configured holdDelay keeps an acquired permit occupied during the experiment.
• The try-with-resources block releases the lease after success or failure.
• The ConfirmationEnvelope carries the transaction result plus the admission source.

• Save ConfirmationFacade.java.
• Compile the new admission boundary by running the existing test suite:

mvn test

What does this check prove?

Maven compiles both new services before running the existing tests. Admission stays in off mode during those tests.

You should see the Part 1 tests plus the idempotency tests pass. The PostgreSQL confirmation transaction still owns purchase correctness.

Seeing a compilation failure?

Check that each file path matches its package declaration. Confirm that every code section appears in the order shown.

Use this debugging prompt to trace the first compiler error.

Route confirmations through the facade

The controller still calls ReservationService.confirm() directly. Routing that endpoint through ConfirmationFacade makes admission part of every HTTP confirmation request.

• Replace the contents of src/main/java/com/nextwork/ticketing/booking/TicketController.java with this version:

package com.nextwork.ticketing.booking;
import java.util.*;
import com.nextwork.ticketing.booking.ApiModels.*;
import org.springframework.http.*;
import org.springframework.web.bind.annotation.*;
@RestController
@RequestMapping("/api")
public class TicketController {
 private final ReservationService service; private final ConfirmationFacade confirmations;
 public TicketController(ReservationService service, ConfirmationFacade confirmations) { this.service = service; this.confirmations = confirmations; }
 @GetMapping("/events/{eventId}/seats") public List<SeatView> seats(@PathVariable UUID eventId) {
  return service.listSeats(eventId).stream().map(s -> SeatView.from(s, service.now())).toList();
 }
 @PostMapping("/events/{eventId}/holds") public ResponseEntity<HoldResponse> hold(@PathVariable UUID eventId, @RequestBody HoldRequest request) {
  var result = service.createHold(eventId, request.seatId(), request.buyerId());
  return new ResponseEntity<>(new HoldResponse(result.holdId(), result.seatId(), result.buyerId(), result.expiresAt()), HttpStatus.CREATED);
 }
 @PostMapping("/holds/{holdId}/confirm") public ResponseEntity<?> confirm(@PathVariable UUID holdId,
   @RequestHeader(value = "Idempotency-Key", required = false) String suppliedKey) {
  String key = suppliedKey == null || suppliedKey.isBlank() ? "legacy-" + UUID.randomUUID() : suppliedKey;
  var envelope = confirmations.confirm(holdId, key); var result = envelope.result();
  var builder = ResponseEntity.status(result.expired() ? HttpStatus.GONE : HttpStatus.CREATED)
   .header("X-Admission-Source", envelope.admissionSource())
   .header("Idempotency-Replayed", Boolean.toString(result.replayed()));
  return result.expired() ? builder.body(new StatusResponse("EXPIRED")) : builder.body(ReservationView.from(result.reservation()));
 }
 @DeleteMapping("/holds/{holdId}") public ResponseEntity<Void> release(@PathVariable UUID holdId) { service.release(holdId); return ResponseEntity.noContent().build(); }
 @GetMapping("/reservations/{id}") public ReservationView reservation(@PathVariable UUID id) { return ReservationView.from(service.getReservation(id)); }
}

What changed in the controller?

• The constructor receives ConfirmationFacade beside the existing reservation service.
• The confirmation endpoint delegates to confirmations.confirm().
• The X-Admission-Source header reveals which admission path handled the request.
• The existing Idempotency-Replayed header still identifies a replayed transaction result.

• Save TicketController.java.
• Verify the complete HTTP path by running the test suite again:

mvn test

What should pass?

You should see the full test suite pass. The controller now reaches the same idempotent transaction through the admission facade.

That is the code path ready. The next check runs it in two separate JVMs.

Spring context failing to start?

Confirm that both new classes use the @Service annotation. Check that the controller constructor receives ConfirmationFacade.

Use this context debugging prompt to inspect the startup failure.

✔️ Awesome, I've got everything!

Your tests pass. Save all three Java files before building the two application containers.

ⓧ I'd like to double check the full code

Compare your admission service with this complete file.

package com.nextwork.ticketing.admission;
import java.time.Duration;
import java.util.*;
import java.util.concurrent.Semaphore;
import java.util.concurrent.atomic.AtomicBoolean;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.dao.DataAccessException;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.data.redis.core.script.RedisScript;
import org.springframework.http.HttpStatus;
import org.springframework.stereotype.Service;
import org.springframework.web.server.ResponseStatusException;
@Service
public class AdmissionService {
 private static final String KEY = "admission:confirm";
 private static final RedisScript<Long> ACQUIRE = RedisScript.of("""
   local time = redis.call('TIME')
   local now = (tonumber(time[1]) * 1000) + math.floor(tonumber(time[2]) / 1000)
   redis.call('ZREMRANGEBYSCORE', KEYS[1], '-inf', now)
   if redis.call('ZCARD', KEYS[1]) >= tonumber(ARGV[2]) then return 0 end
   redis.call('ZADD', KEYS[1], now + tonumber(ARGV[1]), ARGV[3])
   redis.call('EXPIRE', KEYS[1], tonumber(ARGV[4]))
   return 1
   """, Long.class);
 private static final RedisScript<Long> RELEASE = RedisScript.of("return redis.call('ZREM', KEYS[1], ARGV[1])", Long.class);
 private final StringRedisTemplate redis;
 private final String mode;
 private final String failurePolicy;
 private final int limit;
 private final long leaseTtlMillis;
 private final Semaphore localSemaphore;
 public AdmissionService(StringRedisTemplate redis,
   @Value("${app.admission.mode:off}") String mode,
   @Value("${app.admission.failure-policy:fail-open}") String failurePolicy,
   @Value("${app.admission.limit:8}") int limit,
   @Value("${app.admission.lease-ttl:2s}") Duration leaseTtl) {
  this.redis = redis; this.mode = mode; this.failurePolicy = failurePolicy; this.limit = limit;
  leaseTtlMillis = leaseTtl.toMillis(); localSemaphore = new Semaphore(limit);
 }
 public Lease acquire() {
  return switch (mode.toLowerCase(Locale.ROOT)) {
   case "off" -> Lease.noop("off");
   case "local" -> acquireLocal();
   case "redis" -> acquireRedis();
   default -> throw new IllegalStateException("Unknown admission mode: " + mode);
  };
 }
 private Lease acquireLocal() {
  if (!localSemaphore.tryAcquire()) throw new ResponseStatusException(HttpStatus.TOO_MANY_REQUESTS, "Local admission limit reached");
  return new Lease("local", localSemaphore::release);
 }
 private Lease acquireRedis() {
  String token = UUID.randomUUID().toString();
  long keyTtlSeconds = Math.max(1, ((leaseTtlMillis + 999) / 1000) * 2);
  try {
   Long admitted = redis.execute(ACQUIRE, List.of(KEY), Long.toString(leaseTtlMillis), Integer.toString(limit), token, Long.toString(keyTtlSeconds));
   if (!Long.valueOf(1).equals(admitted)) throw new ResponseStatusException(HttpStatus.TOO_MANY_REQUESTS, "Redis admission limit reached");
   return new Lease("redis", () -> releaseRedis(token));
  } catch (DataAccessException failure) {
   if ("fail-open".equalsIgnoreCase(failurePolicy)) return Lease.noop("fail-open");
   throw new ResponseStatusException(HttpStatus.SERVICE_UNAVAILABLE, "Admission service unavailable", failure);
  }
 }
 private void releaseRedis(String token) {
  try { redis.execute(RELEASE, List.of(KEY), token); } catch (DataAccessException ignored) { }
 }
 public static final class Lease implements AutoCloseable {
  private final String source; private final Runnable release; private final AtomicBoolean closed = new AtomicBoolean();
  private Lease(String source, Runnable release) { this.source = source; this.release = release; }
  static Lease noop(String source) { return new Lease(source, () -> {}); }
  public String source() { return source; }
  public void close() { if (closed.compareAndSet(false, true)) release.run(); }
 }
}

What should match?

The constructor must create localSemaphore from limit. The local path must return a lease whose release action restores that permit.

Compare your facade with this complete file.

package com.nextwork.ticketing.booking;
import java.time.Duration;
import java.util.UUID;
import com.nextwork.ticketing.admission.AdmissionService;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.http.HttpStatus;
import org.springframework.stereotype.Service;
import org.springframework.web.server.ResponseStatusException;
@Service
public class ConfirmationFacade {
 private final AdmissionService admission; private final ReservationService reservations; private final Duration holdDelay;
 public ConfirmationFacade(AdmissionService admission, ReservationService reservations,
   @Value("${app.admission.hold-delay:0ms}") Duration holdDelay) {
  this.admission = admission; this.reservations = reservations; this.holdDelay = holdDelay;
 }
 public ConfirmationEnvelope confirm(UUID holdId, String key) {
  try (var lease = admission.acquire()) {
   try { Thread.sleep(holdDelay.toMillis()); }
   catch (InterruptedException interrupted) {
    Thread.currentThread().interrupt();
    throw new ResponseStatusException(HttpStatus.SERVICE_UNAVAILABLE, "Request interrupted", interrupted);
   }
   return new ConfirmationEnvelope(reservations.confirm(holdId, key), lease.source());
  }
 }
 public record ConfirmationEnvelope(ReservationService.ConfirmationResult result, String admissionSource) {}
}

What should match?

The try-with-resources block must surround the delay plus the call to reservations.confirm(). This keeps admission outside the transaction.

Compare your controller with this complete file.

package com.nextwork.ticketing.booking;
import java.util.*;
import com.nextwork.ticketing.booking.ApiModels.*;
import org.springframework.http.*;
import org.springframework.web.bind.annotation.*;
@RestController
@RequestMapping("/api")
public class TicketController {
 private final ReservationService service; private final ConfirmationFacade confirmations;
 public TicketController(ReservationService service, ConfirmationFacade confirmations) { this.service = service; this.confirmations = confirmations; }
 @GetMapping("/events/{eventId}/seats") public List<SeatView> seats(@PathVariable UUID eventId) {
  return service.listSeats(eventId).stream().map(s -> SeatView.from(s, service.now())).toList();
 }
 @PostMapping("/events/{eventId}/holds") public ResponseEntity<HoldResponse> hold(@PathVariable UUID eventId, @RequestBody HoldRequest request) {
  var result = service.createHold(eventId, request.seatId(), request.buyerId());
  return new ResponseEntity<>(new HoldResponse(result.holdId(), result.seatId(), result.buyerId(), result.expiresAt()), HttpStatus.CREATED);
 }
 @PostMapping("/holds/{holdId}/confirm") public ResponseEntity<?> confirm(@PathVariable UUID holdId,
   @RequestHeader(value = "Idempotency-Key", required = false) String suppliedKey) {
  String key = suppliedKey == null || suppliedKey.isBlank() ? "legacy-" + UUID.randomUUID() : suppliedKey;
  var envelope = confirmations.confirm(holdId, key); var result = envelope.result();
  var builder = ResponseEntity.status(result.expired() ? HttpStatus.GONE : HttpStatus.CREATED)
   .header("X-Admission-Source", envelope.admissionSource())
   .header("Idempotency-Replayed", Boolean.toString(result.replayed()));
  return result.expired() ? builder.body(new StatusResponse("EXPIRED")) : builder.body(ReservationView.from(result.reservation()));
 }
 @DeleteMapping("/holds/{holdId}") public ResponseEntity<Void> release(@PathVariable UUID holdId) { service.release(holdId); return ResponseEntity.noContent().build(); }
 @GetMapping("/reservations/{id}") public ReservationView reservation(@PathVariable UUID id) { return ReservationView.from(service.getReservation(id)); }
}

What should match?

The confirmation endpoint must call confirmations.confirm() before adding both response headers.

Race both application instances

The existing Docker Compose stack passes the same admission settings to both application services. The lab delay keeps each permit occupied for 500ms so the requests overlap.

Building both application containers can take a few minutes. The command runs the services in the background after their images are ready.

• Start both application instances with one local permit by running:

ADMISSION_MODE=local ADMISSION_LIMIT=1 ADMISSION_HOLD_DELAY=500ms docker compose up -d --build

What does this configuration control?

• The local mode selects the semaphore path in each process.
• The limit of 1 creates one permit inside each JVM.
• The configured delay keeps each acquired permit busy during confirmation.
• The application services listen through host ports 8081 plus 8082.

You should see PostgreSQL become healthy. You should also see both application services start.

One application service missing?

Check the build output for the first failed service. Confirm that host ports 8081 plus 8082 are available.

Use this container troubleshooting prompt for environment-specific help.

The race needs one valid hold plus one shared idempotency key. The existing Part 1 seat gives both confirmation requests the same purchase target.

• Create a hold through the first application instance by running:

curl --silent --show-error -X POST "http://localhost:8081/api/events/20000000-0000-0000-0000-000000000001/holds" -H 'Content-Type: application/json' -d '{"seatId":"10000000-0000-0000-0000-000000000001","buyerId":"local-race-buyer"}'

What should the hold request return?

You should see a JSON response containing holdId plus the selected seat. That hold becomes the common target for both confirmation requests.

• Copy the returned holdId value here: your hold ID.

No hold ID returned?

Confirm that app-1 is running on port 8081. The selected seat must also be available.

Use this hold debugging prompt to diagnose the response.

Before you run the race, do you expect one deployment-wide admission or one admission from each process?

• Send the same confirmation to both application ports concurrently by running:

curl --parallel --silent --show-error \
  --dump-header - --output /tmp/app1.json \
  -X POST "http://localhost:8081/api/holds/[[HOLD_ID="your hold ID"]]/confirm" \
  -H 'Idempotency-Key: local-race-1' -H 'Content-Type: application/json' -d '{}' \
  --next \
  --dump-header - --output /tmp/app2.json \
  -X POST "http://localhost:8082/api/holds/[[HOLD_ID="your hold ID"]]/confirm" \
  -H 'Idempotency-Key: local-race-1' -H 'Content-Type: application/json' -d '{}'

What did the race expose?

You should see two 201 responses. Both header sets should contain X-Admission-Source: local.

One response should report a replay through Idempotency-Replayed. Both JVMs admitted work despite the deployment limit of one.

This shortfall is the intended result. Each application process owns a separate semaphore with its own permit.

Did the requests miss each other?

Confirm that both application services started with the 500ms delay. Confirm that both requests use the same hold ID plus the same idempotency key.

Use this race debugging prompt to inspect the experiment.

Admission failed at the deployment boundary. Before you query the database, how many reservations do you expect for the target seat?

• Count the reservation plus the completed idempotency rows by running:

docker compose exec -T postgres psql -U tickets -d tickets -c "SELECT COUNT(*) AS reservations_for_seat FROM reservations WHERE seat_id='10000000-0000-0000-0000-000000000001'::uuid; SELECT COUNT(*) AS completed_rows_for_key FROM idempotency_records WHERE idempotency_key='local-race-1' AND response_status=201;"

What does PostgreSQL prove?

You should see reservations_for_seat equal 1. You should also see completed_rows_for_key equal 1.

The local coordination policy admitted too much work. Transactional idempotency plus the seat constraint still produced one purchase.

Database count different from one?

Inspect the two HTTP status lines first. A missing reservation means the confirmation requests did not complete successfully.

Use this invariant debugging prompt to trace the database state.

You have exposed the boundary between coordination and correctness. Next, you will replace the two independent permit pools with one atomic shared lease gate.

Coordinate with Redis Leases

Your two application instances proved that one semaphore per JVM cannot enforce a shared admission limit. The booking transaction in PostgreSQL still protected the seat.

The admission decision now needs shared state in Redis. Cleanup must happen in the same server operation as counting. Lease creation must happen there too.

In this step, get ready to:

• Create atomic Lua scripts for acquiring shared leases and releasing unique tokens.
• Apply fail-open or fail-closed behavior when Redis is unavailable.
• Prove lease expiry and stale-release safety with Testcontainers.

Build the atomic lease scripts

A shared sorted set holds active lease tokens under admission:confirm. Each token has an expiry timestamp as its score.

One short script uses Redis server time to remove expired leases before checking capacity. This keeps the read-decide-write sequence atomic.

• In AdmissionService.java, find the StringRedisTemplate import.
• Add this import directly below it:

import org.springframework.data.redis.core.script.RedisScript;

Why add this import?

Spring Data Redis represents each typed Lua script as a RedisScript<Long>. The return value tells the application whether Redis admitted the request.

• Find the KEY constant near the top of AdmissionService.
• Add the acquire and release scripts directly below it:

 private static final RedisScript<Long> ACQUIRE = RedisScript.of("""
   local time = redis.call('TIME')
   local now = (tonumber(time[1]) * 1000) + math.floor(tonumber(time[2]) / 1000)
   redis.call('ZREMRANGEBYSCORE', KEYS[1], '-inf', now)
   if redis.call('ZCARD', KEYS[1]) >= tonumber(ARGV[2]) then return 0 end
   redis.call('ZADD', KEYS[1], now + tonumber(ARGV[1]), ARGV[3])
   redis.call('EXPIRE', KEYS[1], tonumber(ARGV[4]))
   return 1
   """, Long.class);
 private static final RedisScript<Long> RELEASE = RedisScript.of("return redis.call('ZREM', KEYS[1], ARGV[1])", Long.class);

How do the scripts protect capacity?

• TIME supplies one shared clock for both API instances.
• ZREMRANGEBYSCORE removes leases whose expiry score has passed.
• ZCARD compares the active lease count with the configured limit.
• ZADD stores a unique token with its expiry timestamp.
• EXPIRE applies a safety TTL to the entire sorted-set key.
• ZREM removes only the token supplied by the lease owner.

Why a lease gate?

Rate limiting controls how many requests arrive during a time window. This gate controls how many confirmations are admitted at the same time.

The lease can expire while a slow confirmation is still running. PostgreSQL remains the authority for idempotency and seat ownership.

Handle Redis outcomes safely

Every admitted request needs its own UUID token. That identity prevents an expired lease owner from releasing a newer lease.

A Redis connection failure follows the configured policy. Fail-open preserves booking availability. Fail-closed protects the database from new confirmation traffic.

• In AdmissionService.java, find the existing acquireRedis() method.
• Replace the entire method with this implementation:

 private Lease acquireRedis() {
  String token = UUID.randomUUID().toString();
  long keyTtlSeconds = Math.max(1, ((leaseTtlMillis + 999) / 1000) * 2);
  try {
   Long admitted = redis.execute(ACQUIRE, List.of(KEY), Long.toString(leaseTtlMillis), Integer.toString(limit), token, Long.toString(keyTtlSeconds));
   if (!Long.valueOf(1).equals(admitted)) throw new ResponseStatusException(HttpStatus.TOO_MANY_REQUESTS, "Redis admission limit reached");
   return new Lease("redis", () -> releaseRedis(token));
  } catch (DataAccessException failure) {
   if ("fail-open".equalsIgnoreCase(failurePolicy)) return Lease.noop("fail-open");
   throw new ResponseStatusException(HttpStatus.SERVICE_UNAVAILABLE, "Admission service unavailable", failure);
  }
 }

What does acquisition decide?

• UUID.randomUUID() gives each admitted request a unique lease token.
• keyTtlSeconds keeps the Redis key alive beyond an individual lease expiry.
• redis.execute() sends the shared key through KEYS. It sends lease inputs through ARGV.
• A zero result becomes HTTP 429 before the booking transaction begins.
• DataAccessException produces a bypass lease under fail-open. It produces HTTP 503 under fail-closed.

• Add the releaseRedis() method directly below acquireRedis().

 private void releaseRedis(String token) {
  try { redis.execute(RELEASE, List.of(KEY), token); } catch (DataAccessException ignored) { }
 }

Why ignore release failures?

The lease already has an expiry score and a safety TTL. A release outage cannot undo a completed PostgreSQL transaction.

The token-checked script also protects a replacement lease from a stale caller. Only the matching member can be removed.

• Save AdmissionService.java.

Before you run the existing tests, predict whether the default admission mode requires a live Redis service.

• Compile the Redis path and run the existing test suite by running:

mvn test

What does this test run prove?

Maven compiles the new scripts and Redis methods before running the tests. The existing PostgreSQL tests still use the default admission mode of off.

You should see the existing tests pass. This confirms the Redis path compiles without making Redis a requirement for the original correctness suite.

Seeing a compilation failure?

Check that the RedisScript import sits with the other Spring Data Redis imports. Confirm that both script constants remain inside AdmissionService.

Check that releaseRedis() sits outside acquireRedis().

Help me compare my AdmissionService with the required Redis lease implementation.

✔️ Awesome, I've got everything!

Your shared lease implementation compiles. Make sure AdmissionService.java is saved before continuing.

ⓧ I'd like to double check the full code

Compare your file with the complete implementation below.

package com.nextwork.ticketing.admission;
import java.time.Duration;
import java.util.*;
import java.util.concurrent.Semaphore;
import java.util.concurrent.atomic.AtomicBoolean;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.dao.DataAccessException;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.data.redis.core.script.RedisScript;
import org.springframework.http.HttpStatus;
import org.springframework.stereotype.Service;
import org.springframework.web.server.ResponseStatusException;
@Service
public class AdmissionService {
 private static final String KEY = "admission:confirm";
 private static final RedisScript<Long> ACQUIRE = RedisScript.of("""
   local time = redis.call('TIME')
   local now = (tonumber(time[1]) * 1000) + math.floor(tonumber(time[2]) / 1000)
   redis.call('ZREMRANGEBYSCORE', KEYS[1], '-inf', now)
   if redis.call('ZCARD', KEYS[1]) >= tonumber(ARGV[2]) then return 0 end
   redis.call('ZADD', KEYS[1], now + tonumber(ARGV[1]), ARGV[3])
   redis.call('EXPIRE', KEYS[1], tonumber(ARGV[4]))
   return 1
   """, Long.class);
 private static final RedisScript<Long> RELEASE = RedisScript.of("return redis.call('ZREM', KEYS[1], ARGV[1])", Long.class);
 private final StringRedisTemplate redis;
 private final String mode;
 private final String failurePolicy;
 private final int limit;
 private final long leaseTtlMillis;
 private final Semaphore localSemaphore;
 public AdmissionService(StringRedisTemplate redis,
   @Value("${app.admission.mode:off}") String mode,
   @Value("${app.admission.failure-policy:fail-open}") String failurePolicy,
   @Value("${app.admission.limit:8}") int limit,
   @Value("${app.admission.lease-ttl:2s}") Duration leaseTtl) {
  this.redis = redis; this.mode = mode; this.failurePolicy = failurePolicy; this.limit = limit;
  leaseTtlMillis = leaseTtl.toMillis(); localSemaphore = new Semaphore(limit);
 }
 public Lease acquire() {
  return switch (mode.toLowerCase(Locale.ROOT)) {
   case "off" -> Lease.noop("off");
   case "local" -> acquireLocal();
   case "redis" -> acquireRedis();
   default -> throw new IllegalStateException("Unknown admission mode: " + mode);
  };
 }
 private Lease acquireLocal() {
  if (!localSemaphore.tryAcquire()) throw new ResponseStatusException(HttpStatus.TOO_MANY_REQUESTS, "Local admission limit reached");
  return new Lease("local", localSemaphore::release);
 }
 private Lease acquireRedis() {
  String token = UUID.randomUUID().toString();
  long keyTtlSeconds = Math.max(1, ((leaseTtlMillis + 999) / 1000) * 2);
  try {
   Long admitted = redis.execute(ACQUIRE, List.of(KEY), Long.toString(leaseTtlMillis), Integer.toString(limit), token, Long.toString(keyTtlSeconds));
   if (!Long.valueOf(1).equals(admitted)) throw new ResponseStatusException(HttpStatus.TOO_MANY_REQUESTS, "Redis admission limit reached");
   return new Lease("redis", () -> releaseRedis(token));
  } catch (DataAccessException failure) {
   if ("fail-open".equalsIgnoreCase(failurePolicy)) return Lease.noop("fail-open");
   throw new ResponseStatusException(HttpStatus.SERVICE_UNAVAILABLE, "Admission service unavailable", failure);
  }
 }
 private void releaseRedis(String token) {
  try { redis.execute(RELEASE, List.of(KEY), token); } catch (DataAccessException ignored) { }
 }
 public static final class Lease implements AutoCloseable {
  private final String source; private final Runnable release; private final AtomicBoolean closed = new AtomicBoolean();
  private Lease(String source, Runnable release) { this.source = source; this.release = release; }
  static Lease noop(String source) { return new Lease(source, () -> {}); }
  public String source() { return source; }
  public void close() { if (closed.compareAndSet(false, true)) release.run(); }
 }
}

Prove expiry and shared coordination

A timing-sensitive lease needs an executable test. The test holds one token past its TTL before proving that an old owner cannot remove the replacement token.

The final acquisition proves that an explicit release frees capacity. This covers rejection, expiry, stale release safety and normal release behavior.

• In the code editor from earlier, create RedisAdmissionIntegrationTest.java inside src/test/java/com/nextwork/ticketing using the new-file control in the file sidebar.
• Paste this integration test into the new file:

package com.nextwork.ticketing;
import static org.junit.jupiter.api.Assertions.*;
import com.nextwork.ticketing.admission.AdmissionService;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.boot.testcontainers.service.connection.ServiceConnection;
import org.springframework.test.context.*;
import org.springframework.web.server.ResponseStatusException;
import org.testcontainers.containers.*;
import org.testcontainers.junit.jupiter.*;
import org.testcontainers.utility.DockerImageName;
@Testcontainers
@SpringBootTest(properties = {"app.admission.mode=redis", "app.admission.failure-policy=fail-closed", "app.admission.limit=1", "app.admission.lease-ttl=1s"})
class RedisAdmissionIntegrationTest {
 @Container @ServiceConnection static final PostgreSQLContainer<?> POSTGRES = new PostgreSQLContainer<>("postgres:18.6-alpine3.24");
 @Container static final GenericContainer<?> REDIS = new GenericContainer<>(DockerImageName.parse("redis:8.10.2-alpine")).withExposedPorts(6379);
 @DynamicPropertySource static void redisProperties(DynamicPropertyRegistry registry) {
  registry.add("spring.data.redis.host", REDIS::getHost);
  registry.add("spring.data.redis.port", () -> REDIS.getMappedPort(6379));
 }
 @Autowired AdmissionService admission;
 @Test void leaseExpiresAndStaleReleaseCannotRemoveTheNewOwner() throws Exception {
  var first = admission.acquire();
  ResponseStatusException full = assertThrows(ResponseStatusException.class, admission::acquire); assertEquals(429, full.getStatusCode().value());
  Thread.sleep(1100); var second = admission.acquire(); first.close();
  ResponseStatusException stillFull = assertThrows(ResponseStatusException.class, admission::acquire); assertEquals(429, stillFull.getStatusCode().value());
  second.close(); try (var third = admission.acquire()) { assertEquals("redis", third.source()); }
 }
}

What does this test prove?

• The first acquisition fills the one-lease limit.
• The next acquisition receives HTTP 429 while that lease is active.
• The 1s TTL expires before the second token is admitted.
• Closing the stale first lease cannot remove the second token.
• Closing the second lease permits a third acquisition from Redis.

• Save RedisAdmissionIntegrationTest.java.

The first container-backed run may take longer while Docker prepares the pinned images. Before you run it, predict which acquisition should receive 429.

• Run the complete test suite with the new Redis integration test by running:

mvn test

What should the suite confirm?

Testcontainers starts isolated PostgreSQL and Redis services for the integration test. The test exercises the real Lua scripts through AdmissionService.

You should see RedisAdmissionIntegrationTest pass with the rest of the Maven suite. The passing method confirms rejection, expiry, stale-token safety and release.

Does the Redis test fail?

Confirm Docker Desktop from the earlier setup is still running. Testcontainers needs access to its container engine.

Check that the test uses the Redis image and port shown in the complete code. Confirm that redisProperties() registers the mapped host and port.

Help me diagnose why RedisAdmissionIntegrationTest fails during mvn test.

✔️ Awesome, I've got everything!

Your Redis lease test now covers every required ownership transition. Keep the test file saved.

ⓧ I'd like to double check the full code

Compare your integration test with the complete file below.

package com.nextwork.ticketing;
import static org.junit.jupiter.api.Assertions.*;
import com.nextwork.ticketing.admission.AdmissionService;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.boot.testcontainers.service.connection.ServiceConnection;
import org.springframework.test.context.*;
import org.springframework.web.server.ResponseStatusException;
import org.testcontainers.containers.*;
import org.testcontainers.junit.jupiter.*;
import org.testcontainers.utility.DockerImageName;
@Testcontainers
@SpringBootTest(properties = {"app.admission.mode=redis", "app.admission.failure-policy=fail-closed", "app.admission.limit=1", "app.admission.lease-ttl=1s"})
class RedisAdmissionIntegrationTest {
 @Container @ServiceConnection static final PostgreSQLContainer<?> POSTGRES = new PostgreSQLContainer<>("postgres:18.6-alpine3.24");
 @Container static final GenericContainer<?> REDIS = new GenericContainer<>(DockerImageName.parse("redis:8.10.2-alpine")).withExposedPorts(6379);
 @DynamicPropertySource static void redisProperties(DynamicPropertyRegistry registry) {
  registry.add("spring.data.redis.host", REDIS::getHost);
  registry.add("spring.data.redis.port", () -> REDIS.getMappedPort(6379));
 }
 @Autowired AdmissionService admission;
 @Test void leaseExpiresAndStaleReleaseCannotRemoveTheNewOwner() throws Exception {
  var first = admission.acquire();
  ResponseStatusException full = assertThrows(ResponseStatusException.class, admission::acquire); assertEquals(429, full.getStatusCode().value());
  Thread.sleep(1100); var second = admission.acquire(); first.close();
  ResponseStatusException stillFull = assertThrows(ResponseStatusException.class, admission::acquire); assertEquals(429, stillFull.getStatusCode().value());
  second.close(); try (var third = admission.acquire()) { assertEquals("redis", third.source()); }
 }
}

Your automated test proves lease ownership inside one application context. The final check sends real traffic to both running instances.

• Recreate both API services in Redis mode by running:

ADMISSION_MODE=redis ADMISSION_LIMIT=1 ADMISSION_HOLD_DELAY=500ms docker compose up -d --force-recreate app-1 app-2

What does this restart change?

Docker Compose recreates both API services with Redis admission enabled. Each instance now executes the same Lua script against admission:confirm.

The admission limit is 1. The lab delay keeps the winning lease active long enough for the competing request to reach Redis.

You should see both application services recreated. Their mapped ports remain 8081 and 8082.

Before you repeat the race, predict whether both instances can acquire the same shared capacity slot.

• Return to the concurrent confirmation race from the previous step.
• Repeat the race against ports 8081 and 8082 using the same hold ID and idempotency key.

You should see one admitted confirmation with X-Admission-Source: redis. The competing request should return 429.

What changed across the two instances?

Both JVMs now count active work from one Redis sorted set. The second instance sees the first instance's lease before it opens a booking transaction.

Redis only sheds concurrent load. PostgreSQL still claims the idempotency key and decides whether the reservation is valid.

Before the final test run, predict whether the shared runtime check changed any PostgreSQL correctness behavior.

• Confirm the complete automated suite still passes by running:

mvn test

What does the final run confirm?

The Redis lease-expiry test passes alongside the PostgreSQL idempotency tests. Admission coordination now changes load handling without replacing the booking invariant.

That is the shared coordination gap closed. Both API instances now respect one Redis-backed admission limit while PostgreSQL continues to protect each purchase.

Your two instances now coordinate admission through an atomic Redis lease. Next, you will stop Redis deliberately and measure how each failure policy changes availability and protection.

Test Failure Policies and Load

Your two API instances now use a shared Redis lease gate. The existing PostgreSQL transaction still protects every purchase.

Redis now sits on the synchronous request path. An outage forces you to choose between booking availability or admission protection. This step exercises both policies before measuring the system under load.

In this step, get ready to:

• Compare fail-open behavior against fail-closed behavior during a Redis outage.
• Build a Dockerized 1,000-iteration ticket-drop workload.
• Measure both admission modes with database assertions.

Exercise both outage policies

Fail-open bypasses admission when Redis cannot respond within the bounded 200ms timeout. Fail-closed returns 503 before ReservationService.confirm() opens its transaction.

Why test both policies?

Fail-open preserves booking availability during a coordination outage. PostgreSQL absorbs the admitted work.

Fail-closed preserves the admission limit by rejecting new work. The choice becomes an explicit operating policy.

The existing Docker Compose stack lets you stop Redis without stopping either API instance or the database.

• Return to the terminal running the repository stack from earlier.
• Set ADMISSION_FAILURE_POLICY to fail-open in the Redis-mode launch environment.
• Recreate app-1 using the Redis-mode Compose command from the previous step.
• Recreate app-2 using the same environment.
• Create a fresh hold through the existing hold endpoint.
• Prepare the confirmation request with a fresh idempotency key.

Before you stop Redis, do you expect fail-open to preserve the request or reject it?

• Stop only Redis by running this command:

docker compose stop redis

What does this command do?

This stops the redis service. PostgreSQL plus both API instances remain available.

• Send the prepared confirmation request to port 8081.

You should receive a booking response after the bounded Redis timeout. The response header should report X-Admission-Source: fail-open.

• Start Redis again by running this command:

docker compose start redis

What does this command do?

This restarts the existing Redis container. The API instances can use shared admission again.

• Retry the same confirmation request with the same idempotency key.

You should receive 201 with Idempotency-Replayed: true. PostgreSQL replays the committed reservation.

Fail-open request not completing?

Confirm both application containers remain running after Redis stops. Check that the active policy value is fail-open.

Confirm both bounded Redis timeout properties still use 200ms.

Ask for help with the outage behavior.

Fail-closed moves the rejection boundary ahead of the booking transaction. A post-recovery retry proves that the failed attempt did not consume its idempotency key.

• Set ADMISSION_FAILURE_POLICY to fail-closed in the Redis-mode launch environment.
• Recreate app-1 using that environment.
• Recreate app-2 using that environment.
• Create another fresh hold through the existing hold endpoint.
• Prepare its confirmation request with a new idempotency key.
• Stop only Redis by running this command:

docker compose stop redis

What changes under fail-closed?

The same Redis outage now blocks admission. The facade cannot enter the PostgreSQL booking transaction.

Before you send the request, do you expect the hold confirmation to reach PostgreSQL?

• Send the new confirmation request to port 8082.

You should receive 503. The idempotency key remains available because the booking transaction never opened.

• Start Redis again by running this command:

docker compose start redis

What does recovery prove?

Redis becomes available to both application instances again. The same request can now pass admission.

• Retry the failed confirmation with the same idempotency key.

You should receive 201. The successful retry proves the earlier 503 did not create a completed idempotency record.

Still receiving 503 after recovery?

Confirm the Redis container has started. Confirm both API instances still use ADMISSION_MODE=redis.

Ask for help tracing the recovery path.

Build the ticket-drop workload

A Grafana k6 shared-iterations scenario distributes exactly 1,000 iterations across 50 virtual users. Each iteration creates one hold before confirming it through the other API instance.

• Switch back to Visual Studio Code.
• Create a load folder at the top level of the repository.
• Create ticket-drop.js inside the load folder.

You should now see load/ticket-drop.js beside the existing project files.

• Add the scenario configuration plus metric counters to load/ticket-drop.js by copying this first chunk:

import http from 'k6/http';
import { check, sleep } from 'k6';
import exec from 'k6/execution';
import { Counter } from 'k6/metrics';
const EVENT = '20000000-0000-0000-0000-000000000001';
const BASES = ['http://app-1:8080', 'http://app-2:8080'];
const ATTEMPTS = 1000;
const attempts = new Counter('ticket_attempts');
const completed = new Counter('completed_reservations');
const admissionRejections = new Counter('redis_admission_rejections');
const retries = new Counter('retried_requests');
const replays = new Counter('idempotency_replays');
const conflicts = new Counter('idempotency_conflicts');
const temporaryFailures = new Counter('temporary_failures');
const unexpected = new Counter('unexpected_responses');
export const options = {
 scenarios: {ticket_drop: {executor: 'shared-iterations', vus: 50, iterations: ATTEMPTS, maxDuration: '2m'}},
 thresholds: {unexpected_responses: ['count==0']},
};
export function setup() {
 const response = http.get(`${BASES[0]}/api/events/${EVENT}/seats`);
 const seats = response.json().filter((seat) => seat.label.startsWith('L')).slice(0, ATTEMPTS).map((seat) => seat.id);
 if (seats.length < ATTEMPTS) throw new Error(`Expected ${ATTEMPTS} load seats, found ${seats.length}`);
 return {seats};
}

What does this code do?

• The scenario fixes the workload at 1,000 shared iterations across 50 virtual users.
• The custom counters separate purchases from admission rejections or temporary failures.
• The setup() function loads exactly 1,000 fictional seats before the measured iterations begin.
• The threshold fails the run when unexpected_responses is greater than zero.

• Save load/ticket-drop.js.
• Confirm the editor recognizes options plus setup() without syntax warnings.

Seeing a JavaScript warning?

Check that each import starts at the beginning of its line. Confirm the closing brace for setup() appears after return {seats}.

Ask for help checking the first workload chunk.

• Add the confirmation helper plus iteration flow below setup() by copying this chunk:

function confirm(base, holdId, key) {
 return http.post(`${base}/api/holds/${holdId}/confirm`, '{}', {
  headers: {'Content-Type': 'application/json', 'Idempotency-Key': key}, tags: {operation: 'confirm'},
 });
}
export default function (data) {
 const index = exec.scenario.iterationInTest;
 const firstBase = BASES[index % 2], secondBase = BASES[(index + 1) % 2];
 const key = `load-confirm-${index}`; attempts.add(1);
 const hold = http.post(`${firstBase}/api/events/${EVENT}/holds`, JSON.stringify({seatId: data.seats[index], buyerId: `buyer-${index}`}), {
  headers: {'Content-Type': 'application/json'}, tags: {operation: 'hold'},
 });
 if (!check(hold, {'hold created': (response) => response.status === 201})) { unexpected.add(1); return; }
 let result = confirm(secondBase, hold.json().holdId, key);
 if (result.status === 429) { admissionRejections.add(1); retries.add(1); sleep(0.05); result = confirm(firstBase, hold.json().holdId, key); }
 if (result.status === 201) {
  completed.add(1); const replay = confirm(firstBase, hold.json().holdId, key);
  if (replay.status === 201) replays.add(1); else if (replay.status === 429) admissionRejections.add(1);
  else if (replay.status === 409) conflicts.add(1); else if (replay.status === 503) temporaryFailures.add(1); else unexpected.add(1);
 } else if (result.status === 429) admissionRejections.add(1);
 else if (result.status === 409) conflicts.add(1); else if (result.status === 503) temporaryFailures.add(1); else unexpected.add(1);
}

How does each iteration behave?

• Each iteration alternates its first API instance.
• Confirmation targets the other instance to exercise shared state.
• A 429 increments the rejection counter before one retry.
• A successful confirmation is repeated with the same key to measure idempotency replay.

• Save load/ticket-drop.js.
• Confirm the final closing brace appears after the response classification branches.

Workload file still showing a syntax warning?

Check the braces around the successful 201 branch. Confirm the final response branches remain inside the default function.

Ask for help checking the completed workload.

✔️ Awesome, I've got everything!

Great. Your Dockerized load profile now creates holds across both application instances.

ⓧ I'd like to double check the full code

Compare load/ticket-drop.js with the complete workload below.

import http from 'k6/http';
import { check, sleep } from 'k6';
import exec from 'k6/execution';
import { Counter } from 'k6/metrics';
const EVENT = '20000000-0000-0000-0000-000000000001';
const BASES = ['http://app-1:8080', 'http://app-2:8080'];
const ATTEMPTS = 1000;
const attempts = new Counter('ticket_attempts');
const completed = new Counter('completed_reservations');
const admissionRejections = new Counter('redis_admission_rejections');
const retries = new Counter('retried_requests');
const replays = new Counter('idempotency_replays');
const conflicts = new Counter('idempotency_conflicts');
const temporaryFailures = new Counter('temporary_failures');
const unexpected = new Counter('unexpected_responses');
export const options = {
 scenarios: {ticket_drop: {executor: 'shared-iterations', vus: 50, iterations: ATTEMPTS, maxDuration: '2m'}},
 thresholds: {unexpected_responses: ['count==0']},
};
export function setup() {
 const response = http.get(`${BASES[0]}/api/events/${EVENT}/seats`);
 const seats = response.json().filter((seat) => seat.label.startsWith('L')).slice(0, ATTEMPTS).map((seat) => seat.id);
 if (seats.length < ATTEMPTS) throw new Error(`Expected ${ATTEMPTS} load seats, found ${seats.length}`);
 return {seats};
}
function confirm(base, holdId, key) {
 return http.post(`${base}/api/holds/${holdId}/confirm`, '{}', {
  headers: {'Content-Type': 'application/json', 'Idempotency-Key': key}, tags: {operation: 'confirm'},
 });
}
export default function (data) {
 const index = exec.scenario.iterationInTest;
 const firstBase = BASES[index % 2], secondBase = BASES[(index + 1) % 2];
 const key = `load-confirm-${index}`; attempts.add(1);
 const hold = http.post(`${firstBase}/api/events/${EVENT}/holds`, JSON.stringify({seatId: data.seats[index], buyerId: `buyer-${index}`}), {
  headers: {'Content-Type': 'application/json'}, tags: {operation: 'hold'},
 });
 if (!check(hold, {'hold created': (response) => response.status === 201})) { unexpected.add(1); return; }
 let result = confirm(secondBase, hold.json().holdId, key);
 if (result.status === 429) { admissionRejections.add(1); retries.add(1); sleep(0.05); result = confirm(firstBase, hold.json().holdId, key); }
 if (result.status === 201) {
  completed.add(1); const replay = confirm(firstBase, hold.json().holdId, key);
  if (replay.status === 201) replays.add(1); else if (replay.status === 429) admissionRejections.add(1);
  else if (replay.status === 409) conflicts.add(1); else if (replay.status === 503) temporaryFailures.add(1); else unexpected.add(1);
 } else if (result.status === 429) admissionRejections.add(1);
 else if (result.status === 409) conflicts.add(1); else if (result.status === 503) temporaryFailures.add(1); else unexpected.add(1);
}

What should match?

The complete file contains one setup phase plus one iteration function. It records every expected response class in a dedicated counter.

Run both profiles and document the evidence

The comparison uses a fresh PostgreSQL volume for each profile. This keeps completed seats from the first run out of the second run.

Resetting the volume removes only the local PostgreSQL test data for this repository. Your source files stay in place.

• Return to the repository terminal.
• Remove the current containers plus PostgreSQL volume by running this command:

docker compose down -v

What does this command remove?

This removes the Compose containers plus the named PostgreSQL volume. The next startup applies the migrations to an empty database.

Before you run the baseline, which custom counters do you expect to remain at zero when admission is disabled?

• Start the admission-off baseline before running the Dockerized workload with these commands:

ADMISSION_MODE=off docker compose up -d --build
docker compose --profile load run --rm k6

What does the baseline measure?

The first command starts both API instances with admission disabled. The second command runs the profiled k6 service once.

This run can take up to two minutes while 50 virtual users share 1,000 iterations.

You should see http_reqs plus a request rate. You should also see med plus p(95) latency values.

• Keep the baseline summary visible for comparison.
• Run the database assertions with these commands:

docker compose exec -T postgres psql -U tickets -d tickets -c "SELECT COUNT(*) FROM reservations;"
docker compose exec -T postgres psql -U tickets -d tickets -c "SELECT seat_id, COUNT(*) FROM reservations GROUP BY seat_id HAVING COUNT(*) > 1;"
docker compose exec -T postgres psql -U tickets -d tickets -c "SELECT idempotency_key, COUNT(*) FROM idempotency_records GROUP BY idempotency_key HAVING COUNT(*) > 1;"

What do these assertions prove?

• The first query counts completed reservation rows.
• The second query identifies seats with multiple reservation owners.
• The third query identifies duplicate idempotency-key rows.

The last two queries should return no rows. That means the baseline produced zero duplicate seat groups plus zero duplicate key groups.

Baseline run failing?

Confirm the migration created 1,000 load seats. Confirm both API containers are running before the k6 container starts.

Ask for help with the baseline workload.

The Redis profile uses the same workload against a fresh database. Only the admission mode changes.

• Reset the baseline data before starting the Redis profile with these commands:

docker compose down -v
ADMISSION_MODE=redis ADMISSION_LIMIT=8 ADMISSION_HOLD_DELAY=25ms docker compose up -d --build

What changes for this run?

The database starts empty again. Both API instances now use the shared Redis lease gate with a limit of eight.

Before you run the Redis profile, do you expect throughput or admission rejections to change?

• Run the same 1,000-iteration workload against Redis admission:

docker compose --profile load run --rm k6

What should you compare?

Compare http_reqs plus request rate across the two runs. Compare med plus p(95) as the latency measures.

Use the custom counters to separate completed purchases from admission rejections or retries.

The values come from your Mac. Keep the actual output because the project never estimates missing measurements.

• Review the application logs for deadlock or serialization conflict entries.
• Count only entries that actually occurred.
• Run the database assertions again with these commands:

docker compose exec -T postgres psql -U tickets -d tickets -c "SELECT COUNT(*) FROM reservations;"
docker compose exec -T postgres psql -U tickets -d tickets -c "SELECT seat_id, COUNT(*) FROM reservations GROUP BY seat_id HAVING COUNT(*) > 1;"
docker compose exec -T postgres psql -U tickets -d tickets -c "SELECT idempotency_key, COUNT(*) FROM idempotency_records GROUP BY idempotency_key HAVING COUNT(*) > 1;"

Why repeat the assertions?

The second run exercises the shared Redis gate. These queries verify that PostgreSQL correctness still holds under that profile.

You should see a reservation count from the first query. The final two queries should return no rows.

Seeing duplicate rows or unexpected responses?

Confirm the database volume was removed between runs. Check that unexpected_responses identifies the status path that the workload did not classify.

Ask for help interpreting the evidence.

The repository README now turns the implementation into reproducible evidence. It documents where Redis may shed work plus where PostgreSQL remains authoritative.

• Switch back to README.md in Visual Studio Code.
• Select the existing README content.
• Replace it with this architecture plus correctness section:

# Distributed Ticket Reservation API: Coordination Without Compromising Correctness

Two Spring Boot API instances share Redis admission state and a PostgreSQL booking database. Redis may shed load, but PostgreSQL remains the authority for idempotency and seat ownership.

## Architecture

```mermaid
flowchart LR
 C[Concurrent buyers] --> A1[API instance 1]
 C --> A2[API instance 2]
 A1 --> G[Redis Lua lease gate]
 A2 --> G
 A1 --> I[PostgreSQL idempotency key]
 A2 --> I
 I --> L[PESSIMISTIC_WRITE seat transaction]
 L --> P[(PostgreSQL)]
 P --> U[UNIQUE reservations.seat_id]
 K[Dockerized k6] --> A1
 K --> A2
```

## Correctness boundary

A Redis lease can expire while a slow request is still running, and fail-open can admit more work than the configured limit. Redis never authorizes a sale. PostgreSQL claims the idempotency key, locks the seat, writes the reservation, and enforces one reservation per seat in one transaction.

What does this section document?

The diagram places Redis before both application instances. The correctness boundary assigns final authority to PostgreSQL.

• Save README.md.
• Preview the Markdown document.

You should see both API instances converge on one Redis gate plus one PostgreSQL booking path.

Architecture diagram not rendering?

Confirm the Mermaid fence begins before flowchart LR.

Ask for help checking the diagram syntax.

• Place the cursor after the correctness boundary.
• Append the run commands plus failure-mode comparison:

## Run

```bash
mvn test
mvn package
docker compose up -d --build
```

The instances listen on `http://localhost:8081` and `http://localhost:8082`.

## Failure modes

- `local`: one semaphore per JVM, so two instances can each admit a request.
- `redis`: one atomic shared lease set coordinates both instances.
- `fail-open`: bypass Redis after the 200 ms timeout and preserve booking availability.
- `fail-closed`: return `503` before opening a booking transaction.

## Load comparison

Run each case from a fresh database:

```bash
ADMISSION_MODE=off docker compose up -d --build
docker compose --profile load run --rm k6
docker compose down -v
ADMISSION_MODE=redis ADMISSION_LIMIT=8 ADMISSION_HOLD_DELAY=25ms docker compose up -d --build
docker compose --profile load run --rm k6
```

Why include reproducible commands?

The commands make both measurements repeatable from a fresh database. The failure-mode list records the operational trade-off behind each configuration.

• Save README.md.
• Preview the new run plus failure-mode sections.

You should see separate baseline plus Redis commands. The four admission behaviors should appear above them.

Commands rendering on one line?

Confirm both command groups remain inside fenced Bash blocks. Keep every command on its own line.

Ask for help checking the Markdown fences.

• Place the cursor after the load commands.
• Append the comparison table:

| Metric | Admission off | Redis admission |
| --- | ---: | ---: |
| Total HTTP requests | Record from `http_reqs` | Record from `http_reqs` |
| Requests per second | Record actual rate | Record actual rate |
| p50 latency | Record `med` | Record `med` |
| p95 latency | Record `p(95)` | Record `p(95)` |
| Completed reservations | Record counter and SQL | Record counter and SQL |
| Admission rejections | Record counter | Record counter |
| Retried requests | Record counter | Record counter |
| Idempotency replays | Record counter | Record counter |
| Idempotency conflicts | Record counter | Record counter |
| PostgreSQL deadlock or serialization conflicts | Record actual log count | Record actual log count |
| Duplicate confirmed seats | Must be `0` from SQL | Must be `0` from SQL |

Do not estimate missing values.

How should you use this table?

Each row points to either k6 output or a PostgreSQL assertion. The final instruction prevents estimated results from entering the repository.

• Save README.md.
• Preview the comparison table.

You should see one baseline column plus one Redis-admission column. Every row names its evidence source.

Table columns misaligned?

Confirm every metric row contains two separator characters. Keep the alignment row directly below the header.

Ask for help checking the table.

• Place the cursor after the comparison table.
• Append the database assertions plus outage notes plus cleanup commands:

## Database assertions

```bash
docker compose exec -T postgres psql -U tickets -d tickets -c "SELECT COUNT(*) FROM reservations;"
docker compose exec -T postgres psql -U tickets -d tickets -c "SELECT seat_id, COUNT(*) FROM reservations GROUP BY seat_id HAVING COUNT(*) > 1;"
docker compose exec -T postgres psql -U tickets -d tickets -c "SELECT idempotency_key, COUNT(*) FROM idempotency_records GROUP BY idempotency_key HAVING COUNT(*) > 1;"
```

The last two queries must return no rows.

## Outage proof

Start a fresh fail-open stack and run `bash scripts/redis-outage-proof.sh`. The script reports request failures separately and exits nonzero only when a database invariant or Redis recovery check fails.

## Cleanup

```bash
docker compose stop
docker compose down -v
```

What completes the evidence trail?

The SQL commands verify reservation plus idempotency invariants directly. The outage section points to the next proof exercise.

The cleanup commands provide both pause plus deletion paths for the local stack.

• Save README.md.
• Preview the completed README.

You should see the architecture diagram plus correctness boundary. You should also see failure modes plus load commands plus SQL assertions plus cleanup instructions.

README preview missing its final sections?

Confirm each heading begins on its own line. Confirm every fenced command block has a closing fence.

Ask for help checking the completed document.

✔️ Awesome, I've got everything!

Your repository now explains the design plus the commands that reproduce its evidence.

ⓧ I'd like to double check the full code

Compare README.md with the complete document below.

# Distributed Ticket Reservation API: Coordination Without Compromising Correctness

Two Spring Boot API instances share Redis admission state and a PostgreSQL booking database. Redis may shed load, but PostgreSQL remains the authority for idempotency and seat ownership.

## Architecture

```mermaid
flowchart LR
 C[Concurrent buyers] --> A1[API instance 1]
 C --> A2[API instance 2]
 A1 --> G[Redis Lua lease gate]
 A2 --> G
 A1 --> I[PostgreSQL idempotency key]
 A2 --> I
 I --> L[PESSIMISTIC_WRITE seat transaction]
 L --> P[(PostgreSQL)]
 P --> U[UNIQUE reservations.seat_id]
 K[Dockerized k6] --> A1
 K --> A2
```

## Correctness boundary

A Redis lease can expire while a slow request is still running, and fail-open can admit more work than the configured limit. Redis never authorizes a sale. PostgreSQL claims the idempotency key, locks the seat, writes the reservation, and enforces one reservation per seat in one transaction.

## Run

```bash
mvn test
mvn package
docker compose up -d --build
```

The instances listen on `http://localhost:8081` and `http://localhost:8082`.

## Failure modes

- `local`: one semaphore per JVM, so two instances can each admit a request.
- `redis`: one atomic shared lease set coordinates both instances.
- `fail-open`: bypass Redis after the 200 ms timeout and preserve booking availability.
- `fail-closed`: return `503` before opening a booking transaction.

## Load comparison

Run each case from a fresh database:

```bash
ADMISSION_MODE=off docker compose up -d --build
docker compose --profile load run --rm k6
docker compose down -v
ADMISSION_MODE=redis ADMISSION_LIMIT=8 ADMISSION_HOLD_DELAY=25ms docker compose up -d --build
docker compose --profile load run --rm k6
```

| Metric | Admission off | Redis admission |
| --- | ---: | ---: |
| Total HTTP requests | Record from `http_reqs` | Record from `http_reqs` |
| Requests per second | Record actual rate | Record actual rate |
| p50 latency | Record `med` | Record `med` |
| p95 latency | Record `p(95)` | Record `p(95)` |
| Completed reservations | Record counter and SQL | Record counter and SQL |
| Admission rejections | Record counter | Record counter |
| Retried requests | Record counter | Record counter |
| Idempotency replays | Record counter | Record counter |
| Idempotency conflicts | Record counter | Record counter |
| PostgreSQL deadlock or serialization conflicts | Record actual log count | Record actual log count |
| Duplicate confirmed seats | Must be `0` from SQL | Must be `0` from SQL |

Do not estimate missing values.

## Database assertions

```bash
docker compose exec -T postgres psql -U tickets -d tickets -c "SELECT COUNT(*) FROM reservations;"
docker compose exec -T postgres psql -U tickets -d tickets -c "SELECT seat_id, COUNT(*) FROM reservations GROUP BY seat_id HAVING COUNT(*) > 1;"
docker compose exec -T postgres psql -U tickets -d tickets -c "SELECT idempotency_key, COUNT(*) FROM idempotency_records GROUP BY idempotency_key HAVING COUNT(*) > 1;"
```

The last two queries must return no rows.

## Outage proof

Start a fresh fail-open stack and run `bash scripts/redis-outage-proof.sh`. The script reports request failures separately and exits nonzero only when a database invariant or Redis recovery check fails.

## Cleanup

```bash
docker compose stop
docker compose down -v
```

What should match?

The complete README contains the architecture plus correctness boundary. It also contains reproducible load commands plus unfilled evidence prompts.

Before the final database check, do you expect any seat to have more than one reservation owner?

• Run the duplicate-seat assertion one final time:

docker compose exec -T postgres psql -U tickets -d tickets -c "SELECT seat_id, COUNT(*) FROM reservations GROUP BY seat_id HAVING COUNT(*) > 1;"

What should this prove?

This groups reservations by seat. It returns only seats whose reservation count exceeds one.

You should see no rows. That result means the duplicate confirmed-seat count is zero.

That is the full proof: Redis changes admission behavior while PostgreSQL preserves purchase correctness.

Secret mission

Redis Outage During a Ticket Drop

A shell proof races both API instances while Redis is unavailable. PostgreSQL decides whether one seat owner survives. It also decides whether one completed idempotency result survives.

Clean Up Your Resources

Clean Up Your Resources

Everything here runs through local Docker Compose resources, so keeping the stack costs $0. Decide whether to keep your resources running, pause them for later, or delete the local runtime data.

Resources you used:

• A PostgreSQL 18.6 container with the postgres-data volume holding reservation and idempotency data.
• A Redis 8.10.2 container providing shared admission leases.
• Two application containers named app-1 and app-2.
• Temporary Grafana k6 2.3.0 and Testcontainers containers created during load and integration runs.
• Local application images produced by the Compose build.

Keep everything running

No action is needed if you are still testing the distributed API. Your containers and PostgreSQL data remain available for more experiments.

• Return the Compose services to a running state by running this command from the repository:

docker compose up -d

What does this command do?

This command creates or starts the services in the background. Your terminal returns while PostgreSQL, Redis, app-1 and app-2 continue running.

• Confirm the terminal reports the four services in a running state.
• Keep the postgres-data volume so your reservation and load-test records remain available.
• Keep the repository and cached application images for future experiments.

Pause - I'll come back to this later

Shut down the running services to free local memory while preserving the containers and PostgreSQL data.

• Pause every service in the local stack by running this command from the repository:

docker compose stop

What does this command do?

This command stops the Compose services without deleting their containers or named volume. Your PostgreSQL rows remain available for the next start.

• Confirm the terminal reports each service as stopped.
• Keep the postgres-data volume so the database returns with the same records.
• Keep the repository and local application images untouched.

Delete - I don't want to use this again

Deleting the PostgreSQL volume is permanent. Your source files and README remain untouched.

• Remove the Compose containers and PostgreSQL data by running this command from the repository:

docker compose down -v

What does this command do?

This command removes the Compose containers plus the named PostgreSQL volume. Your repository files and cached application images remain untouched.

• Confirm the command completes without reporting a remaining Compose service.
• Return to Docker Desktop from earlier.
• Select the local container list.
• Remove any stopped one-off containers associated with Grafana k6 or Testcontainers.
• Select the local image list.
• Remove the application images produced by this repository's Compose build.
• Confirm the project containers, the postgres-data volume and the application images are absent.

Nice Work!

Nice Work!

You did it! You completed Part 2 with a two-instance Spring Boot API that coordinates admission through Redis while PostgreSQL remains the authority for every purchase.

You've learned how to:

• Implement transactional idempotency inside the PostgreSQL purchase transaction. Concurrent retries now replay one committed reservation. Fingerprint mismatches return a conflict. Failed attempts roll back their key claims safely.
• Expose the limit of JVM-local state by running one semaphore in each API instance. You replaced that local gate with an atomic Redis lease gate. Short Lua scripts make each admission decision atomic. Unique tokens protect lease release ownership.
• Exercise fail-open and fail-closed outage policies with bounded Redis timeouts. A Docker workload running Grafana k6 captures throughput plus latency percentiles. PostgreSQL queries prove that duplicate confirmed seats remain at zero.
• Complete the Secret Mission by stopping Redis during simultaneous same-seat retries across both API instances. The outage proof separates request failures from correctness violations. It confirms one reservation for the target seat. It confirms one completed idempotency row. It also confirms successful recovery after Redis restarts.

Ready to quiz yourself?