Scalability Lab: Break It, Then Fix It

Stress-test a Spring Boot API, watch it break, then fix it with 4 scalability patterns.

Introduction

30 Second Summary

Every app works fine with one user. The real test is what happens when thousands of people hit your system at the same time.

In this project, you will build a local scalability lab that lets you break a service under load, then fix it by layering in real architecture patterns one at a time.

What You'll Build

A Spring Boot REST API backed by PostgreSQL, instrumented with Prometheus metrics and visualized in Grafana, that you systematically stress-test and improve with four scalability patterns.

By the end of this project, you'll have:

  • A running multi-service Docker Compose stack demonstrating caching, load balancing, horizontal scaling, and backpressure with real-time metrics.
  • Grafana dashboards showing concrete before/after metrics that prove why each pattern matters.
  • A k6 load test script that generates burst traffic to expose bottlenecks on demand.
  • Secret Mission: Implement a circuit breaker pattern so your service degrades gracefully when a downstream dependency fails.

Are there any prerequisites?

This project assumes you have Docker Desktop installed, Java development experience, and basic Docker Compose familiarity.

You do not need prior experience with Redis, Nginx, or Prometheus.

Set Up the Project

Before you can observe systems breaking under load, you need a lab to run experiments in. This step builds that lab: a Spring Boot API backed by PostgreSQL, fronted by Nginx, cached with Redis, and fully instrumented with Prometheus and Grafana.

You will scaffold the project from Spring Initializr, then wire up the entire Docker infrastructure stack. By the end of this step, a single docker compose up command will spin up every service your lab needs.

In this step, get ready to:
  • Create the Spring Boot project with all required dependencies from Spring Initializr.
  • Build the Docker Compose infrastructure stack with six services.
  • Configure the application properties and verify the full stack starts.
Create the Spring Boot project structure

Spring Initializr generates a ready-to-build project skeleton with the Maven wrapper, directory structure, and your chosen dependencies already wired in.

  • Open your browser and navigate to start.spring.io.
  • Set the project type to Maven and the language to Java.
  • Under Project Metadata, set Group to com.scalabilitylab and Artifact to scalability-lab.
  • Set the Java version to 21.
  • In the Dependencies search box, search for and add each of the following:
  • Spring Web
  • Spring Data JPA
  • PostgreSQL Driver
  • Spring Boot Actuator
  • Spring Data Redis (search "Redis")
  • Spring Cache Abstraction (search "Cache")
  • Prometheus (under Observability, adds Micrometer Registry Prometheus)
  • Click Generate to download the project zip file.
  • Extract the zip file to your Desktop. You should now have a folder called scalability-lab on your Desktop.
  • Open a terminal by pressing Cmd+Space (macOS) or the Windows key (Windows), typing Terminal, and pressing Enter.
  • Navigate into the project folder by running this command:
cd ~/Desktop/scalability-lab

Spring Initializr may generate a slightly different Spring Boot version than 3.5.3. To keep everyone on the same version, replace your pom.xml with the exact content below.

  • Open pom.xml in your text editor and replace its entire contents with this:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.5.3</version>
        <relativePath/>
    </parent>
    <groupId>com.scalabilitylab</groupId>
    <artifactId>scalability-lab</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>scalability-lab</name>
    <description>Scalability Lab: Break It Then Fix It</description>
    <properties>
        <java.version>21</java.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-jpa</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-redis</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-cache</artifactId>
        </dependency>
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-registry-prometheus</artifactId>
        </dependency>
        <dependency>
            <groupId>org.postgresql</groupId>
            <artifactId>postgresql</artifactId>
            <scope>runtime</scope>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>
</project>

What does this pom.xml include?

  • spring-boot-starter-web provides the embedded Tomcat server and REST controller support.
  • spring-boot-starter-data-jpa adds Hibernate and Spring Data repositories for database access.
  • spring-boot-starter-actuator exposes health and metrics endpoints your monitoring stack will scrape.
  • spring-boot-starter-data-redis and spring-boot-starter-cache wire up Redis-backed caching.
  • micrometer-registry-prometheus formats metrics in Prometheus exposition format so Prometheus can scrape them.
  • Save pom.xml.

✔️ Awesome, I've got everything!

Great. Make sure you saved pom.xml before moving on.

ⓧ I'd like to double check the full code

Your pom.xml should match this exactly:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.5.3</version>
        <relativePath/>
    </parent>
    <groupId>com.scalabilitylab</groupId>
    <artifactId>scalability-lab</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>scalability-lab</name>
    <description>Scalability Lab: Break It Then Fix It</description>
    <properties>
        <java.version>21</java.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-jpa</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-redis</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-cache</artifactId>
        </dependency>
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-registry-prometheus</artifactId>
        </dependency>
        <dependency>
            <groupId>org.postgresql</groupId>
            <artifactId>postgresql</artifactId>
            <scope>runtime</scope>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>
</project>
Create the Docker Compose infrastructure stack

Your scalability lab runs six services: the Spring Boot API, PostgreSQL for persistence, Redis for caching, Nginx as a load balancer, Prometheus for metrics collection, and Grafana for visualization.

You will create four configuration files that wire everything together.

  • Create a file called docker-compose.yml in the root of your scalability-lab folder and paste in this content:
services:
  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: scalabilitylab
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
    ports:
      - "5432:5432"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      timeout: 3s
      retries: 5

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5

  api:
    build: .
    deploy:
      replicas: 3
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    environment:
      SPRING_DATASOURCE_URL: jdbc:postgresql://postgres:5432/scalabilitylab

  nginx:
    image: nginx:1.27-alpine
    ports:
      - "80:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - api

  prometheus:
    image: prom/prometheus:v3.4.1
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro

  grafana:
    image: grafana/grafana:11.6.0
    ports:
      - "3000:3000"
    environment:
      GF_SECURITY_ADMIN_PASSWORD: admin
      GF_AUTH_ANONYMOUS_ENABLED: "true"
      GF_AUTH_ANONYMOUS_ORG_ROLE: Admin
    depends_on:
      - prometheus

What does this compose file define?

  • postgres and redis have healthchecks so the API waits for them to be ready before starting.
  • api builds from the local Dockerfile and starts 3 replicas for horizontal scaling experiments.
  • nginx acts as a reverse proxy that distributes traffic across all API replicas using round-robin.
  • prometheus scrapes metrics from the API replicas every 5 seconds.
  • grafana has anonymous admin access enabled so you can jump straight into dashboards without logging in.
  • Save docker-compose.yml.

Seeing YAML indentation errors?

YAML is whitespace-sensitive. Make sure you use spaces (not tabs) and that nested keys are indented consistently. Check that depends_on and condition are aligned with other keys at the same level.

Help me fix a YAML indentation error in my docker-compose.yml

Next, Docker needs to know how to build your Spring Boot app into a container image.

  • Create a file called Dockerfile (no extension) in the root of your scalability-lab folder and paste in this content:
FROM eclipse-temurin:21-jdk AS builder
WORKDIR /app
COPY pom.xml .
COPY .mvn .mvn
COPY mvnw .
RUN chmod +x mvnw && ./mvnw dependency:go-offline -B
COPY src src
RUN ./mvnw package -DskipTests -B

FROM eclipse-temurin:21-jre
WORKDIR /app
COPY --from=builder /app/target/*.jar app.jar
EXPOSE 8080
ENTRYPOINT ["java", "-jar", "app.jar"]

What does this Dockerfile do?

This is a multi-stage build. The first stage (builder) uses the full JDK to compile your app and package it as a JAR.

The second stage copies only the compiled JAR into a slim JRE image. This produces a smaller, production-style container without build tools baked in.

  • Save Dockerfile.

Now create the Nginx configuration that tells it how to route traffic to your API replicas.

  • Create a file called nginx.conf in the root of your scalability-lab folder and paste in this content:
events {
    worker_connections 1024;
}

http {
    upstream api_servers {
        server api:8080;
    }

    server {
        listen 80;

        location / {
            proxy_pass http://api_servers;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_http_version 1.1;
            proxy_set_header Connection "";
        }
    }
}

How does Nginx load balance here?

The upstream api_servers block points to the Docker Compose service name api. Docker's internal DNS resolves this name to all running containers in that service.

Nginx defaults to round-robin distribution. Each new request goes to the next available replica in the pool.

  • Save nginx.conf.

Finally, Prometheus needs a configuration file telling it where to find your app's metrics.

  • Create a file called prometheus.yml in the root of your scalability-lab folder and paste in this content:
global:
  scrape_interval: 5s

scrape_configs:
  - job_name: "spring-boot"
    metrics_path: "/actuator/prometheus"
    dns_sd_configs:
      - names:
          - "api"
        type: "A"
        port: 8080

How does Prometheus find all 3 API replicas?

The dns_sd_configs section uses Docker's DNS service discovery. The name api resolves to all container IP addresses in that service. Prometheus scrapes each one individually at the /actuator/prometheus endpoint every 5 seconds.

  • Save prometheus.yml.
  • Confirm all your new files exist by running:
ls

You should see docker-compose.yml, Dockerfile, nginx.conf, prometheus.yml, pom.xml, and the src directory listed in the output.

✔️ Awesome, I've got everything!

Great. Make sure all four new files are saved before moving on.

ⓧ I'd like to double check the full code

Your docker-compose.yml should match this:

services:
  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: scalabilitylab
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
    ports:
      - "5432:5432"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      timeout: 3s
      retries: 5

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5

  api:
    build: .
    deploy:
      replicas: 3
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    environment:
      SPRING_DATASOURCE_URL: jdbc:postgresql://postgres:5432/scalabilitylab

  nginx:
    image: nginx:1.27-alpine
    ports:
      - "80:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - api

  prometheus:
    image: prom/prometheus:v3.4.1
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro

  grafana:
    image: grafana/grafana:11.6.0
    ports:
      - "3000:3000"
    environment:
      GF_SECURITY_ADMIN_PASSWORD: admin
      GF_AUTH_ANONYMOUS_ENABLED: "true"
      GF_AUTH_ANONYMOUS_ORG_ROLE: Admin
    depends_on:
      - prometheus

Your Dockerfile should match this:

FROM eclipse-temurin:21-jdk AS builder
WORKDIR /app
COPY pom.xml .
COPY .mvn .mvn
COPY mvnw .
RUN chmod +x mvnw && ./mvnw dependency:go-offline -B
COPY src src
RUN ./mvnw package -DskipTests -B

FROM eclipse-temurin:21-jre
WORKDIR /app
COPY --from=builder /app/target/*.jar app.jar
EXPOSE 8080
ENTRYPOINT ["java", "-jar", "app.jar"]

Your nginx.conf should match this:

events {
    worker_connections 1024;
}

http {
    upstream api_servers {
        server api:8080;
    }

    server {
        listen 80;

        location / {
            proxy_pass http://api_servers;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_http_version 1.1;
            proxy_set_header Connection "";
        }
    }
}

Your prometheus.yml should match this:

global:
  scrape_interval: 5s

scrape_configs:
  - job_name: "spring-boot"
    metrics_path: "/actuator/prometheus"
    dns_sd_configs:
      - names:
          - "api"
        type: "A"
        port: 8080
Configure application properties and verify the build

Spring Boot uses application.properties to configure database connections, caching, and metric endpoints. The file already exists from the Initializr scaffold, but you need to replace its contents with your lab configuration.

  • Open src/main/resources/application.properties and replace its entire contents with this:
spring.application.name=scalability-lab
spring.datasource.url=jdbc:postgresql://postgres:5432/scalabilitylab
spring.datasource.username=postgres
spring.datasource.password=postgres
spring.jpa.hibernate.ddl-auto=update
spring.jpa.open-in-view=false

spring.data.redis.host=redis
spring.data.redis.port=6379
spring.cache.type=redis
spring.cache.redis.time-to-live=30s

management.endpoints.web.exposure.include=health,prometheus
management.endpoint.prometheus.enabled=true

server.tomcat.threads.max=10
app.max-inflight=20

What do these properties control?

  • spring.datasource.* connects to the PostgreSQL container using Docker's internal hostname postgres.
  • spring.data.redis.* connects to the Redis container. The cache TTL of 30s means cached data expires after 30 seconds.
  • management.endpoints.web.exposure.include exposes the health and Prometheus metrics endpoints so monitoring tools can reach them.
  • server.tomcat.threads.max=10 deliberately constrains the thread pool. This makes the server easier to overwhelm during load tests so you can observe backpressure behavior.
  • app.max-inflight=20 is a custom property you will use later for a load-shedding filter.
  • Save application.properties.
  • Verify the project compiles by running this in your terminal:
./mvnw package -DskipTests

You should see output ending with BUILD SUCCESS. This confirms your dependencies resolve and the app compiles.

Seeing BUILD FAILURE?

Check that your pom.xml matches the version above exactly. A common issue is a missing closing tag or an incorrect groupId on one of the dependencies.

Also confirm Java 21 is active by running java --version in your terminal. The output should show version 21.

Help me debug a Maven BUILD FAILURE in my scalability-lab project

  • Start the full infrastructure stack by running:
docker compose up -d

This builds the API image and starts all six services in the background. The first run takes a few minutes because Docker downloads base images and Maven downloads dependencies inside the build stage.

  • Verify all containers are running by checking:
docker compose ps

You should see all services listed with a status of Up or running. The api service should show 3 replicas.

Containers exiting or not starting?

Make sure Docker Desktop is running. If the api containers are restarting, check their logs with docker compose logs api. A common cause is a typo in application.properties (especially the datasource URL).

If postgres fails to start, ensure port 5432 is not already in use by another local PostgreSQL instance.

Help me troubleshoot Docker containers that won't start in my scalability lab

✔️ Awesome, I've got everything!

Great. Your entire lab infrastructure is now running.

ⓧ I'd like to double check the full code

Your src/main/resources/application.properties should match this:

spring.application.name=scalability-lab
spring.datasource.url=jdbc:postgresql://postgres:5432/scalabilitylab
spring.datasource.username=postgres
spring.datasource.password=postgres
spring.jpa.hibernate.ddl-auto=update
spring.jpa.open-in-view=false

spring.data.redis.host=redis
spring.data.redis.port=6379
spring.cache.type=redis
spring.cache.redis.time-to-live=30s

management.endpoints.web.exposure.include=health,prometheus
management.endpoint.prometheus.enabled=true

server.tomcat.threads.max=10
app.max-inflight=20

Your lab infrastructure is fully operational. Next up, you will build the baseline API, hit it with load, and watch it break in real time.

Build the Baseline API and Observe It Breaking

Your infrastructure is running, but there is no application code for it to serve yet. In this step, you will build a simple REST API with a deliberately slow database query, hammer it with simulated traffic, and watch it buckle in real time.

Seeing a system fail under load is the foundation for understanding why scalability patterns exist. Without a measurable baseline, every optimization is guesswork.

In this step, get ready to:
  • Build a Spring Boot REST API with a simulated slow database query and seed data.
  • Write and run a k6 load test script that generates burst traffic against the API.
  • Observe the system breaking in Grafana dashboards with real-time metrics.
Implement the product API with a slow database query

The API will expose a single endpoint that fetches products from PostgreSQL. A 200ms Thread.sleep simulates a complex database join, making the endpoint slow enough to reveal contention under load.

  • In your editor, open src/main/java/com/scalabilitylab/ScalabilityLabApplication.java and replace its contents with the following:
package com.scalabilitylab;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cache.annotation.EnableCaching;

@SpringBootApplication
@EnableCaching
public class ScalabilityLabApplication {
    public static void main(String[] args) {
        SpringApplication.run(ScalabilityLabApplication.class, args);
    }
}

What does @EnableCaching do?

The @EnableCaching annotation activates Spring's caching infrastructure. You will not see caching take effect in this step because the baseline test is designed to show the system without help. The annotation is here so the app is ready for the caching pattern you will layer in next.

  • Create a new file at src/main/java/com/scalabilitylab/Product.java and paste the following entity class:
package com.scalabilitylab;

import jakarta.persistence.Entity;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Id;
import java.io.Serializable;

// JPA entity representing a product in the database
@Entity
public class Product implements Serializable {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;
    private String name;
    private String description;
    private double price;

    // No-arg constructor required by JPA
    public Product() {}

    public Product(String name, String description, double price) {
        this.name = name;
        this.description = description;
        this.price = price;
    }

    public Long getId() { return id; }
    public void setId(Long id) { this.id = id; }
    public String getName() { return name; }
    public void setName(String name) { this.name = name; }
    public String getDescription() { return description; }
    public void setDescription(String description) { this.description = description; }
    public double getPrice() { return price; }
    public void setPrice(double price) { this.price = price; }
}

Why does Product implement Serializable?

When you cache objects in Redis later, the objects must be serializable so they can be stored as byte arrays. Adding Serializable now prevents a runtime error when caching kicks in.

  • Create src/main/java/com/scalabilitylab/ProductRepository.java with this repository interface:
package com.scalabilitylab;

import org.springframework.data.jpa.repository.JpaRepository;

// Spring Data JPA generates the SQL queries for us
public interface ProductRepository extends JpaRepository<Product, Long> {
}
  • Create src/main/java/com/scalabilitylab/ProductService.java with the deliberately slow service:
package com.scalabilitylab;

import org.springframework.cache.annotation.Cacheable;
import org.springframework.stereotype.Service;
import java.util.List;

@Service
public class ProductService {

    private final ProductRepository productRepository;

    public ProductService(ProductRepository productRepository) {
        this.productRepository = productRepository;
    }

    // Simulates a slow database query with Thread.sleep
    // @Cacheable will be activated once Redis caching is layered in
    @Cacheable("products")
    public List<Product> getAllProducts() {
        try {
            Thread.sleep(200);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
        return productRepository.findAll();
    }
}

What does this service do?

Every call to getAllProducts() waits 200ms before hitting the database. This simulates the latency of a complex query joining multiple tables.

The @Cacheable("products") annotation will not help yet because every request is a cache miss against a fresh Redis instance. You will observe this in the baseline metrics.

  • Create src/main/java/com/scalabilitylab/ProductController.java to expose the REST endpoint:
package com.scalabilitylab;

import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import java.util.List;

// REST controller exposing the product list endpoint
@RestController
@RequestMapping("/api/products")
public class ProductController {

    private final ProductService productService;

    public ProductController(ProductService productService) {
        this.productService = productService;
    }

    @GetMapping
    public List<Product> getProducts() {
        return productService.getAllProducts();
    }
}
  • Create src/main/java/com/scalabilitylab/DataSeeder.java to populate the database on startup:
package com.scalabilitylab;

import org.springframework.boot.CommandLineRunner;
import org.springframework.stereotype.Component;

// Seeds the database with 50 sample products on first startup
@Component
public class DataSeeder implements CommandLineRunner {

    private final ProductRepository productRepository;

    public DataSeeder(ProductRepository productRepository) {
        this.productRepository = productRepository;
    }

    @Override
    public void run(String... args) {
        if (productRepository.count() == 0) {
            for (int i = 1; i <= 50; i++) {
                productRepository.save(new Product(
                    "Product " + i,
                    "Description for product " + i,
                    9.99 + i
                ));
            }
        }
    }
}
  • Create src/main/java/com/scalabilitylab/BackpressureFilter.java to protect the server from being overwhelmed:
package com.scalabilitylab;

import jakarta.servlet.Filter;
import jakarta.servlet.FilterChain;
import jakarta.servlet.ServletException;
import jakarta.servlet.ServletRequest;
import jakarta.servlet.ServletResponse;
import jakarta.servlet.http.HttpServletResponse;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;
import java.io.IOException;
import java.util.concurrent.atomic.AtomicInteger;

// Rejects requests with HTTP 503 when too many are in flight
@Component
public class BackpressureFilter implements Filter {

    private final AtomicInteger inFlight = new AtomicInteger(0);

    @Value("${app.max-inflight:20}")
    private int maxInFlight;

    @Override
    public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain)
            throws IOException, ServletException {
        int current = inFlight.incrementAndGet();
        try {
            if (current > maxInFlight) {
                HttpServletResponse httpResponse = (HttpServletResponse) response;
                httpResponse.setStatus(503);
                httpResponse.getWriter().write("{\"error\":\"Service overloaded, try again later\"}");
                return;
            }
            chain.doFilter(request, response);
        } finally {
            inFlight.decrementAndGet();
        }
    }
}

What does the backpressure filter do?

This filter tracks how many requests are currently being processed using an AtomicInteger counter. When the count exceeds maxInFlight (set to 20 in application.properties), it immediately returns HTTP 503 instead of queuing the request.

This is a load shedding mechanism. Returning 503 fast is better than letting every request queue up and timeout after 30+ seconds. You will see this in action during the load test.

  • Rebuild the Docker image and restart the stack with a single API instance by running:
docker compose up -d --build --scale api=1

Why scale to 1 instance?

Your docker-compose.yml is configured for 3 replicas, but you want to establish a single-instance baseline first. The --scale api=1 flag overrides the replica count so you can measure performance without load balancing.

  • Wait for the containers to become healthy, then verify the API responds by running:
curl http://localhost/api/products
  • You should see a JSON array of 50 products in your terminal output.

Not seeing any products?

  • Run docker compose logs api and look for connection errors. If you see a PostgreSQL connection refused message, the database may not be healthy yet. Wait 10 seconds and try again.
  • Check that all Java files are in the correct package path under src/main/java/com/scalabilitylab/.

help me debug why my Spring Boot API is not returning products

✔️ Awesome, I've got everything!

Great. Make sure you have saved all seven Java files before moving on.

ⓧ I'd like to double check the full code

Here are all the Java source files you should have at this point:

package com.scalabilitylab;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cache.annotation.EnableCaching;

@SpringBootApplication
@EnableCaching
public class ScalabilityLabApplication {
    public static void main(String[] args) {
        SpringApplication.run(ScalabilityLabApplication.class, args);
    }
}
package com.scalabilitylab;

import jakarta.persistence.Entity;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Id;
import java.io.Serializable;

@Entity
public class Product implements Serializable {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;
    private String name;
    private String description;
    private double price;

    public Product() {}

    public Product(String name, String description, double price) {
        this.name = name;
        this.description = description;
        this.price = price;
    }

    public Long getId() { return id; }
    public void setId(Long id) { this.id = id; }
    public String getName() { return name; }
    public void setName(String name) { this.name = name; }
    public String getDescription() { return description; }
    public void setDescription(String description) { this.description = description; }
    public double getPrice() { return price; }
    public void setPrice(double price) { this.price = price; }
}
package com.scalabilitylab;

import org.springframework.data.jpa.repository.JpaRepository;

public interface ProductRepository extends JpaRepository<Product, Long> {
}
package com.scalabilitylab;

import org.springframework.cache.annotation.Cacheable;
import org.springframework.stereotype.Service;
import java.util.List;

@Service
public class ProductService {

    private final ProductRepository productRepository;

    public ProductService(ProductRepository productRepository) {
        this.productRepository = productRepository;
    }

    @Cacheable("products")
    public List<Product> getAllProducts() {
        try {
            Thread.sleep(200);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
        return productRepository.findAll();
    }
}
package com.scalabilitylab;

import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import java.util.List;

@RestController
@RequestMapping("/api/products")
public class ProductController {

    private final ProductService productService;

    public ProductController(ProductService productService) {
        this.productService = productService;
    }

    @GetMapping
    public List<Product> getProducts() {
        return productService.getAllProducts();
    }
}
package com.scalabilitylab;

import org.springframework.boot.CommandLineRunner;
import org.springframework.stereotype.Component;

@Component
public class DataSeeder implements CommandLineRunner {

    private final ProductRepository productRepository;

    public DataSeeder(ProductRepository productRepository) {
        this.productRepository = productRepository;
    }

    @Override
    public void run(String... args) {
        if (productRepository.count() == 0) {
            for (int i = 1; i <= 50; i++) {
                productRepository.save(new Product(
                    "Product " + i,
                    "Description for product " + i,
                    9.99 + i
                ));
            }
        }
    }
}
package com.scalabilitylab;

import jakarta.servlet.Filter;
import jakarta.servlet.FilterChain;
import jakarta.servlet.ServletException;
import jakarta.servlet.ServletRequest;
import jakarta.servlet.ServletResponse;
import jakarta.servlet.http.HttpServletResponse;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;
import java.io.IOException;
import java.util.concurrent.atomic.AtomicInteger;

@Component
public class BackpressureFilter implements Filter {

    private final AtomicInteger inFlight = new AtomicInteger(0);

    @Value("${app.max-inflight:20}")
    private int maxInFlight;

    @Override
    public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain)
            throws IOException, ServletException {
        int current = inFlight.incrementAndGet();
        try {
            if (current > maxInFlight) {
                HttpServletResponse httpResponse = (HttpServletResponse) response;
                httpResponse.setStatus(503);
                httpResponse.getWriter().write("{\"error\":\"Service overloaded, try again later\"}");
                return;
            }
            chain.doFilter(request, response);
        } finally {
            inFlight.decrementAndGet();
        }
    }
}
Write and run the k6 load test

A single curl request tells you nothing about how the system behaves under concurrent load. k6 is a load testing tool that simulates many virtual users hitting your API simultaneously.

  • Create a file called loadtest.js in your project root (the scalability-lab folder) with the following k6 script:
import http from "k6/http";
import { check, sleep } from "k6";

// Ramp up to 50 virtual users, hold for 20s, then ramp down
export const options = {
  stages: [
    { duration: "10s", target: 50 },
    { duration: "20s", target: 50 },
    { duration: "5s", target: 0 },
  ],
};

export default function () {
  // Hit the products endpoint through Nginx
  const res = http.get("http://host.docker.internal/api/products");
  check(res, {
    "status is 200": (r) => r.status === 200,
    "response time < 500ms": (r) => r.timings.duration < 500,
  });
  sleep(0.1);
}

What does this script do?

  • stages ramps from 0 to 50 virtual users over 10 seconds, holds at 50 for 20 seconds, then ramps back down over 5 seconds.
  • http.get("http://host.docker.internal/api/products") sends requests to your Nginx load balancer. The host.docker.internal hostname resolves to your host machine from inside the k6 Docker container.
  • check() verifies each response is HTTP 200 and under 500ms. Failed checks appear in the k6 summary.
  • sleep(0.1) adds a small pause between iterations to simulate realistic user behavior.
  • Save loadtest.js, then run the load test using the k6 Docker image by running:
docker run --rm -i grafana/k6 run - <loadtest.js

What does this command do?

The < operator pipes your local loadtest.js file into the k6 container via stdin. The - tells k6 to read the script from stdin instead of a file path inside the container.

If you are on Windows using PowerShell, run cat loadtest.js | docker run --rm -i grafana/k6 run - instead.

  • Watch the test run for about 35 seconds. When it finishes, k6 prints a summary showing request counts, response times, and check pass/fail rates.

You should see a significant number of failed checks. Many requests will exceed 500ms, and some will receive HTTP 503 responses from the backpressure filter.

Seeing connection refused errors?

  • Make sure your Docker Compose stack is still running with docker compose ps. The nginx and api containers must be healthy.
  • On Linux, you may need to add --add-host=host.docker.internal:host-gateway to the docker run command so the k6 container can reach your host's port 80.

help me troubleshoot k6 connection refused errors when running my load test in Docker

Observe the baseline metrics in Grafana

The numbers in the k6 summary are useful, but real-time graphs tell a much richer story. Grafana is already running and connected to Prometheus, which is scraping your API's metrics endpoint every 5 seconds.

  • Open your browser and navigate to http://localhost:3000. Grafana loads without a login because anonymous admin access is enabled.
  • Click Connections in the left sidebar.
  • Click Add new connection.
  • Search for Prometheus and select it.
  • Click Add new data source in the upper right.
  • In the URL field, enter http://prometheus:9090.
  • Scroll down and click Save & test. You should see a green confirmation message.

Why use http://prometheus:9090?

Grafana runs inside the same Docker network as Prometheus. The service name prometheus resolves to the Prometheus container's IP via Docker DNS. You cannot use localhost here because that would point to the Grafana container itself.

Now create a dashboard to visualize the API's health.

  • Click Dashboards in the left sidebar.
  • Click New and select New Dashboard.
  • Click the + Add visualization button.
  • Select Prometheus as the data source.
  • In the query editor, enter the following PromQL query to see request rate:
rate(http_server_requests_seconds_count{uri="/api/products"}[1m])
  • Click Run queries to see the graph populate.
  • Set the panel title to Request Rate and click Apply.
  • Add a second panel with this query to see p95 latency:
histogram_quantile(0.95, rate(http_server_requests_seconds_bucket{uri="/api/products"}[1m]))
  • Title this panel P95 Latency and click Apply.
  • Add a third panel with this query to see the error rate (HTTP 5xx responses):
rate(http_server_requests_seconds_count{uri="/api/products", status=~"5.."}[1m])
  • Title this panel Error Rate and click Apply.
  • Click the save icon (floppy disk) in the top right and name your dashboard Scalability Lab.

Now re-run the k6 test while watching the dashboard.

  • Switch back to your terminal and run the load test again:
docker run --rm -i grafana/k6 run - <loadtest.js
  • Switch to your browser and watch the Grafana dashboard update in real time. Set the time range to Last 5 minutes and enable auto-refresh (every 5s) from the top-right dropdown.

You should see p95 latency spike above 200ms as every request hits the slow Thread.sleep, and the error rate panel should show 503 responses when the backpressure filter kicks in.

What do these baseline numbers mean?

With a single API instance, 10 Tomcat threads, and a 200ms sleep per request, the server can handle roughly 50 requests per second at best. Anything beyond that queues up or gets rejected with a 503.

Record the p95 latency and error rate you see. In the next steps, you will layer in caching and horizontal scaling, then re-run the same test to prove the improvement with real numbers.

Dashboard panels showing No Data?

  • Check that Prometheus is scraping your API by opening http://localhost:9090/targets in your browser. You should see the spring-boot job with at least one target in the UP state.
  • If the target is DOWN, the API container may not have started yet. Run docker compose logs api to check for startup errors.
  • Make sure the data source URL in Grafana is http://prometheus:9090 (not localhost).

help me figure out why my Grafana dashboard is showing No Data for Prometheus metrics

Your baseline is established. A single API instance with no caching crumbles under 50 concurrent users. Next, you will layer in Redis caching and watch the p95 latency drop dramatically.

Add Redis Caching

In the last step, you watched your API struggle under 50 concurrent users. Every single request waited 200ms for the simulated database query, and the server had no way to avoid that work.

Your first scalability fix is caching. Instead of hitting the database on every request, your app stores the result in Redis and serves it directly from memory on subsequent calls. In this step, you'll confirm how the caching configuration works, rebuild the stack, and observe the dramatic improvement in your Grafana dashboard.

In this step, get ready to:
  • Confirm the Redis caching configuration already in your project.
  • Rebuild the stack and re-run the k6 load test to observe cache hits.
  • Verify cached data in Redis CLI and understand cache TTL trade-offs.
Confirm the Redis caching setup

Your project already has all the pieces for Redis caching wired together. Let's walk through each piece so you understand how they connect.

  • In your editor, open pom.xml and find these two dependencies:
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-cache</artifactId>
</dependency>

What do these dependencies do?

spring-boot-starter-data-redis provides the connection to your Redis container using the Lettuce client.

spring-boot-starter-cache enables Spring's caching abstraction, letting you cache method results with annotations instead of writing cache logic manually.

  • Open ScalabilityLabApplication.java and find the @EnableCaching annotation:
@SpringBootApplication
@EnableCaching
public class ScalabilityLabApplication {
    public static void main(String[] args) {
        SpringApplication.run(ScalabilityLabApplication.class, args);
    }
}

What does @EnableCaching do?

This annotation activates Spring's caching infrastructure at the application level. Without it, @Cacheable annotations on individual methods would be ignored.

  • Open ProductService.java and find the @Cacheable("products") annotation on getAllProducts():
@Cacheable("products")
public List<Product> getAllProducts() {
    try {
        Thread.sleep(200);
    } catch (InterruptedException e) {
        Thread.currentThread().interrupt();
    }
    return productRepository.findAll();
}

What does @Cacheable do here?

The first time this method is called, it executes normally (including the 200ms sleep simulating a slow query) and stores the result in a Redis cache named products.

Every subsequent call skips the method body entirely and returns the cached result from Redis in single-digit milliseconds.

  • Open application.properties and find the Redis cache configuration:
spring.data.redis.host=redis
spring.data.redis.port=6379
spring.cache.type=redis
spring.cache.redis.time-to-live=30s

What do these properties control?

spring.data.redis.host=redis points to the Redis container by its Docker Compose service name. spring.cache.type=redis tells Spring to use Redis as the cache backend instead of an in-memory map.

spring.cache.redis.time-to-live=30s sets a 30-second expiry on cached entries. After 30 seconds, the next request triggers a fresh database call and re-populates the cache.

Rebuild the stack and re-run the load test

Now that you understand how the caching pieces fit together, let's rebuild the API containers to ensure a clean state and observe the caching effect under load.

  • Rebuild and restart the stack by running this command:
docker compose up -d --build

Why rebuild?

The --build flag recompiles your Spring Boot application inside the Docker image. The -d flag runs everything in the background so you can continue using your terminal.

  • Wait for all containers to report healthy status by running:
docker compose ps
  • Confirm all services show Up or healthy in the status column before continuing.

Containers not starting?

If the API containers show Restarting status, they may be waiting for PostgreSQL or Redis to become healthy. Wait 30 seconds and run docker compose ps again.

If containers still fail, check logs with docker compose logs api to see the specific error.

help me debug why my Docker containers aren't starting

  • Re-run the k6 load test by running:
cat loadtest.js | docker run --rm -i grafana/k6 run -
  • Watch the k6 output in your terminal as it completes. You should see a much higher percentage of requests passing the response time < 500ms check compared to your baseline run.

What's happening during the test?

The very first request from k6 is a cache miss. It goes through the full 200ms Thread.sleep path, queries PostgreSQL, and stores the result in Redis.

Every subsequent request during the 30-second test window hits the Redis cache and returns in single-digit milliseconds. That's a 10-100x latency reduction for the vast majority of requests.

  • Switch to your browser and open Grafana at http://localhost:3000.
  • Open your dashboard and observe the p95 latency panel.
  • Compare the numbers to your baseline from the previous step. You should see the p95 latency drop dramatically after the initial spike.

Reading the Grafana graph

Look for a sharp spike at the start of the test (the cache miss) followed by a flat line near zero for the remaining duration. That flat line is your Redis cache in action.

Not seeing latency improvement in Grafana?

Make sure you selected the correct time range in Grafana (last 5 minutes). If the graph looks identical to your baseline, the API containers may not have rebuilt properly.

Run docker compose logs api | grep "cache" to confirm the application started with Redis cache enabled.

help me understand why my Redis cache doesn't seem to be working

Verify cached data in Redis

Grafana shows the effect, but let's prove the mechanism by inspecting Redis directly. You'll connect to the Redis container and see the cached data sitting in memory.

  • Connect to the Redis CLI inside the container by running:
docker compose exec redis redis-cli
  • List all keys stored in Redis by running:
KEYS *
  • You should see a key starting with products:: in the output. This confirms Spring Boot cached the result of getAllProducts() in Redis.

How does the key format work?

Spring Data Redis prefixes each cache entry with the cache name followed by :: and then the method's arguments as the key. Since getAllProducts() takes no arguments, Spring uses a default key, giving you something like products::SimpleKey [].

  • Exit the Redis CLI by typing:
exit

What about stale data?

Your cache TTL is set to 30 seconds. This means if a product's price changes in PostgreSQL, the API continues serving the old price for up to 30 seconds.

This is the fundamental caching trade-off: performance vs data freshness. A shorter TTL keeps data more current but means more cache misses. A longer TTL improves hit rates but serves staler data.

In a system design interview, you'd choose the TTL based on how stale the data can be for your use case. Product catalogs can tolerate 30-60 seconds of staleness. User session data or inventory counts may need much shorter TTLs.

KEYS * returns empty?

The cache entry expires after 30 seconds. If you waited too long after the k6 test finished, the key has already expired.

Hit the API endpoint once to re-populate the cache, then check again:

curl http://localhost/api/products > /dev/null && docker compose exec redis redis-cli KEYS "*"

help me verify that my Spring Boot Redis cache is storing data correctly

You've just proved the first scalability pattern with real metrics. A single Redis cache layer turned your 200ms-per-request bottleneck into sub-millisecond responses for the vast majority of traffic. Next up, you'll tackle what happens when even cached responses can't keep up with volume by distributing load across multiple API replicas.

Add Horizontal Scaling with Nginx Load Balancer

In the previous step, Redis caching slashed your p95 latency by serving repeat requests from memory instead of hitting the database. That solves the per-request speed problem, but a single server still has finite CPU, memory, and network bandwidth.

Horizontal scaling solves this by running multiple identical copies of your API behind a load balancer. In this step, you will examine how Nginx distributes traffic across 3 API replicas using round-robin, then prove it works by measuring throughput in Grafana.

In this step, get ready to:
  • Examine the Nginx load balancer and replica configuration.
  • Run the k6 load test through the Nginx load balancer on port 80.
  • Compare single-instance vs 3-replica metrics in Grafana and Prometheus.
Examine the Nginx load balancer and replica configuration

Your Docker Compose stack already has the pieces for horizontal scaling wired together. Before running a load test, let's understand how traffic flows from k6 through Nginx to your API replicas.

  • Switch back to your editor and open docker-compose.yml.
  • Find the api service and locate the deploy section:
  api:
    build: .
    deploy:
      replicas: 3
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    environment:
      SPRING_DATASOURCE_URL: jdbc:postgresql://postgres:5432/scalabilitylab

What does replicas: 3 do?

Docker Compose spins up 3 identical containers from the same image, each running your Spring Boot API on port 8080. Docker's internal DNS resolves the service name api to all 3 container IPs automatically.

No code changes required. The same JAR runs in each container, sharing the same PostgreSQL database and Redis cache.

  • Now open nginx.conf and examine the upstream block:
events {
    worker_connections 1024;
}

http {
    upstream api_servers {
        server api:8080;
    }

    server {
        listen 80;

        location / {
            proxy_pass http://api_servers;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_http_version 1.1;
            proxy_set_header Connection "";
        }
    }
}

How does Nginx distribute traffic?

The upstream api_servers block lists server api:8080 as the backend. Because Docker DNS resolves api to all 3 replica IPs, Nginx automatically load-balances across all of them using round-robin (the default strategy).

Round-robin sends each new request to the next server in the list, cycling through all 3. This is the most common load balancing strategy you will encounter in system design interviews.

  • Finally, open prometheus.yml and examine the dns_sd_configs section:
scrape_configs:
  - job_name: "spring-boot"
    metrics_path: "/actuator/prometheus"
    dns_sd_configs:
      - names:
          - "api"
        type: "A"
        port: 8080

How does Prometheus find all replicas?

Instead of hardcoding 3 separate targets, Prometheus uses DNS service discovery (dns_sd_configs) to resolve the api hostname. Docker DNS returns all container IPs for that service, so Prometheus automatically scrapes every replica.

  • Rebuild and restart the full stack to ensure all 3 replicas are healthy by running:
docker compose up -d --build
  • Wait about 30 seconds for all containers to become healthy, then verify the replicas are running:
docker compose ps

You should see 3 containers for the api service, plus one each for postgres, redis, nginx, prometheus, and grafana.

Don't see 3 API containers?

Make sure your docker-compose.yml has deploy: replicas: 3 nested under the api service. Check docker compose logs api for startup errors like database connection failures.

help me debug why my API replicas aren't starting

Run the k6 load test through the Nginx load balancer

Now that 3 replicas are running behind Nginx on port 80, let's hit the system with the same 50-user burst test from earlier. This time, Nginx distributes the load evenly across all 3 instances.

  • Run the k6 load test by running this command:
docker run --rm -i --add-host=host.docker.internal:host-gateway grafana/k6 run - < loadtest.js

What does this command do?

  • grafana/k6 runs k6 inside a Docker container.
  • --add-host=host.docker.internal:host-gateway lets the k6 container reach your host machine's port 80 (where Nginx is listening).
  • run - reads the test script from standard input, piped in with < loadtest.js.
  • The test ramps to 50 virtual users for 30 seconds, sending requests to http://host.docker.internal/api/products (port 80 = Nginx).

Wait for the test to complete. You should see k6 output showing the request rate, response times, and check pass percentages.

  • Return to Grafana at http://localhost:3000 in your browser.
  • Open the dashboard you imported earlier.

You should see request throughput roughly 3x higher than your single-instance baseline. Latency also drops because each replica handles fewer concurrent connections.

Why does throughput triple?

Each API replica has its own Tomcat thread pool. With 3 replicas, you have 3x the threads available to handle requests simultaneously. Nginx spreads incoming requests evenly, so no single instance gets overwhelmed.

Seeing connection refused errors in k6?

Make sure Nginx is running on port 80. Run docker compose ps and confirm the nginx container shows 0.0.0.0:80->80/tcp. If another process is using port 80, stop it first or change the Nginx port mapping in docker-compose.yml.

help me fix k6 connection errors to my Nginx load balancer

Compare metrics and verify Prometheus targets

The final proof of horizontal scaling is seeing all 3 replicas actively serving traffic in your monitoring stack.

  • Open the Prometheus targets page at http://localhost:9090/targets in your browser.
  • Under the spring-boot job, confirm you see 3 targets listed, each with a different IP address and state UP.

What am I looking at?

Each target represents one API container that Prometheus is scraping every 5 seconds. The different IPs confirm Docker created 3 separate containers, and Prometheus discovered them all through DNS service discovery.

  • Return to Grafana and compare your metrics from this test to your earlier baseline (before caching and scaling).

Note these key differences between a single instance and 3 replicas:

  • Request throughput (requests/second) should be roughly 3x higher.
  • P95 latency should be lower because each replica handles fewer connections.
  • Error rate should be lower (or zero) because no single instance is overwhelmed.

The interview talking point

Horizontal scaling gives you linear throughput gains: 3 replicas handle roughly 3x the traffic. It also provides redundancy. If one replica crashes, Nginx routes traffic to the other two automatically.

The trade-off is complexity. You now need stateless services (session data in Redis, not in memory), shared databases, and health checks to detect failed replicas.

Your system now handles 3x the traffic by running identical copies behind a load balancer. Next up, you will add backpressure to protect the system when even horizontal scaling is not enough.

Add Backpressure and Load Shedding

Your horizontally scaled cluster with caching handles 50 concurrent users comfortably. But what happens when traffic doubles overnight?

Without protection, every incoming request queues up, threads exhaust, and the entire system grinds to a halt. In this step, you will understand the backpressure pattern already built into your API and prove it works by hammering the system with 100 virtual users.

In this step, get ready to:
  • Understand the thread pool constraint and backpressure filter in your API.
  • Run k6 at 100 virtual users to trigger load shedding.
  • Analyze the interview talking point: fast-fail 503 vs queuing and timeout.
Understand the thread pool and backpressure filter

Your API already has two layers of protection built in. The first is a constrained thread pool that simulates a real-world server with limited capacity.

  • Open src/main/resources/application.properties and find these two lines near the bottom:
server.tomcat.threads.max=10
app.max-inflight=20

What do these settings do?

server.tomcat.threads.max=10 limits the embedded Tomcat server to 10 worker threads. In production, the default is 200. You are deliberately constraining it to simulate a server under heavy contention.

app.max-inflight=20 sets the threshold for your custom backpressure filter. Once more than 20 requests are being processed simultaneously, new requests get rejected immediately with HTTP 503.

The second layer is a custom servlet filter that enforces that in-flight limit. This is the core of the load shedding pattern.

  • Open src/main/java/com/scalabilitylab/BackpressureFilter.java and review the full filter:
package com.scalabilitylab;

import jakarta.servlet.Filter;
import jakarta.servlet.FilterChain;
import jakarta.servlet.ServletException;
import jakarta.servlet.ServletRequest;
import jakarta.servlet.ServletResponse;
import jakarta.servlet.http.HttpServletResponse;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;
import java.io.IOException;
import java.util.concurrent.atomic.AtomicInteger;

@Component
public class BackpressureFilter implements Filter {

    private final AtomicInteger inFlight = new AtomicInteger(0);

    @Value("${app.max-inflight:20}")
    private int maxInFlight;

    @Override
    public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain)
            throws IOException, ServletException {
        int current = inFlight.incrementAndGet();
        try {
            if (current > maxInFlight) {
                HttpServletResponse httpResponse = (HttpServletResponse) response;
                httpResponse.setStatus(503);
                httpResponse.getWriter().write("{\"error\":\"Service overloaded, try again later\"}");
                return;
            }
            chain.doFilter(request, response);
        } finally {
            inFlight.decrementAndGet();
        }
    }
}

How does this filter protect the system?

  • AtomicInteger inFlight tracks how many requests are currently being processed across all threads. It is thread-safe so concurrent requests cannot corrupt the count.
  • Every request increments the counter on entry and decrements it in the finally block. This guarantees cleanup even if the request throws an exception.
  • When the counter exceeds maxInFlight, the filter immediately returns HTTP 503 without touching the database, cache, or any downstream logic. The rejected request costs almost zero resources.

Why a custom filter instead of Tomcat's built-in queue?

Tomcat's thread pool already rejects connections when its queue is full, but that happens at the TCP level with a connection reset. Your custom filter rejects cleanly at the HTTP level with a structured JSON response.

Clients can parse the 503, back off, and retry intelligently. This is the difference between a system that crashes ungracefully and one that degrades with clear signals.

Run k6 at 100 virtual users to trigger load shedding

With 3 replicas running 10 threads each (30 threads total) and a max in-flight limit of 20 per replica, your cluster can handle roughly 60 concurrent requests before shedding load. Sending 100 virtual users should push past that threshold.

  • Rebuild the API containers to ensure the backpressure filter is active by running this command:
docker compose up -d --build
  • Wait about 30 seconds for all 3 API replicas to start and register as healthy.
  • Verify the replicas are running by running:
docker compose ps

You should see 3 instances of the api service listed as running.

  • Flush the Redis cache so the 200ms database delay is active for the test by running:
docker compose exec redis redis-cli FLUSHALL
  • Run the k6 load test at 100 VUs for 30 seconds by running this command:
cat loadtest.js | docker run --rm -i grafana/k6 run --vus 100 --duration 30s -

What does this command do?

  • cat loadtest.js pipes your load test script into the k6 Docker container via stdin.
  • --vus 100 --duration 30s overrides the script's built-in stages (50 VUs) with a flat 100 virtual users for 30 seconds. CLI flags take precedence over script-defined options in k6.

When the test finishes, you should see a summary showing some failed checks. The status is 200 check will report a pass rate below 100% because some requests received HTTP 503 responses.

  • Return to Grafana at http://localhost:3000 in your browser.
  • Switch to your dashboard and observe the metrics from the last 5 minutes.

You should see 503 responses appearing in the HTTP status code breakdown. The key observation: those 503 responses returned in single-digit milliseconds, while the successful requests still completed normally.

Not seeing any 503 responses?

  • Make sure all 3 API replicas finished rebuilding. Run docker compose ps and confirm all containers show a healthy status.
  • If you skipped the FLUSHALL step, the Redis cache might serve responses instantly for all requests. The in-flight count never builds up when responses return from cache in under 1ms.
  • Check that app.max-inflight=20 is present in your application.properties file.

Why am I not seeing HTTP 503 responses when load testing with 100 VUs?

Compare throughput with and without backpressure

This is a critical system design interview talking point. Look at what the Grafana dashboard tells you about how the system behaved under extreme load.

Why is returning 503 immediately better than letting requests queue and timeout?

Without backpressure, every request queues behind the overloaded threads. Each waiting request holds a connection, consumes memory, and eventually times out after 30+ seconds. The client waits the full timeout duration and gets nothing useful. The server spends resources on requests it will never finish.

With backpressure, excess requests fail immediately in under 5 milliseconds. The client gets a clear 503 signal, can retry with exponential backoff, or route to a different region. The server preserves its resources for the requests it can actually complete. Total successful throughput stays high instead of collapsing to zero.

In your Grafana dashboard, compare: without backpressure the p95 latency spikes to 30+ seconds and the error rate climbs toward 100%. With backpressure, p95 stays low for accepted requests and the 503s return instantly. The system recovers the moment load drops.

  • In your Grafana dashboard, note the following metrics from your 100 VU test:
  • The p95 latency for successful (200) responses stays low because the server is not overloaded.
  • The 503 responses return in under 5ms because the filter short-circuits before any real work happens.
  • Once the k6 test ends, the error rate drops to zero immediately. The system recovers without any restart or manual intervention.

Extra for Experts: How would you tune these values in production?

In a real system, server.tomcat.threads.max would be set based on your server's CPU cores and expected I/O wait time. A common starting point is 2x CPU cores for I/O-bound workloads.

The app.max-inflight threshold should be determined by load testing in staging. Set it just below the point where latency starts spiking. You want to shed load before the system degrades, not after.

You have now proven all four scalability patterns with measurable evidence: caching, horizontal scaling, load balancing, and backpressure. Next up in the Secret Mission, you will add a circuit breaker to protect against cascading failures from downstream services.

Secret mission

Implement a Circuit Breaker

Your system handles load spikes gracefully now. But what happens when a downstream service your API depends on goes completely offline? Without protection, every request waits for a timeout, consuming threads and cascading the failure to your own users. In this secret mission, you will implement the circuit breaker pattern using resilience4j to detect downstream failures and fail fast with a graceful fallback.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally using Docker containers, so there are no ongoing cloud costs.

Resources you used:

  • Docker containers (PostgreSQL, Redis, Nginx, Prometheus, Grafana, API replicas).
  • Docker volumes for PostgreSQL and Redis data persistence.
  • Project folder scalability-lab on your Desktop.

Keep everything running

No action needed. Choose this if you're still actively experimenting with different scalability patterns or want to tweak your Grafana dashboards.

  • Your containers stay running and accessible at their respective ports (localhost:80 for the API, localhost:3000 for Grafana, localhost:9090 for Prometheus).
  • Docker Desktop will continue using system memory while the containers are running.

Pause - I'll come back to this later

Shut down running containers to free up memory while preserving all data. You can restart them later without rebuilding.

  • Stop all containers by running this command from your scalability-lab folder:
docker compose stop
  • When you're ready to resume, run docker compose start from the same folder to restart everything.

Delete - I don't want to use this again

Remove all project containers, networks, and volumes to free up disk space completely.

  • Remove all containers and volumes by running this command from your scalability-lab folder:
docker compose down -v
  • Delete the project folder. On macOS/Linux, run:
cd ~/Desktop
rm -rf scalability-lab
  • On Windows, run:
cd ~\Desktop
rmdir /s /q scalability-lab
  • Optionally, remove the Docker images that were pulled during the project by running:
docker image prune -a

What does docker compose down -v do?

The -v flag tells Docker to also remove the named volumes declared in your Compose file. Without it, your PostgreSQL and Redis data would persist on disk even after the containers are gone.

Nice Work!

Nice Work!

Nice work! You just built a complete scalability laboratory that proves, with real metrics, how architectural patterns transform a system from crumbling under load to handling traffic gracefully.

You've learned how to:

  • Add Redis caching to a Spring Boot API and prove with Grafana dashboards that cache hits reduce p95 latency by 10-100x.
  • Deploy 3 horizontal replicas behind an Nginx load balancer, tripling throughput with round-robin distribution.
  • Implement backpressure and load shedding that returns HTTP 503 immediately instead of letting every request queue and timeout, keeping the system recoverable under burst traffic.
  • Secret Mission: Implement a circuit breaker with resilience4j that protects the system from cascading failures when downstream services go down.

Ready to quiz yourself?