Design and Break a Distributed System
Build a local distributed system, stress-test it with k6, and fix it.
Introduction
30 Second Summary
Every time you order food, stream a video, or check a delivery status, you are relying on a distributed system. When those systems slow down under heavy traffic, someone has to hunt down the bottleneck and fix it.
In this project, you will design a distributed order-processing system from scratch, build it locally with Docker Compose, then deliberately stress-test it to failure using k6 load testing. You will identify the bottleneck, apply a caching fix, and measure the improvement.
What You'll Build
A local distributed order-processing microservice with an Nginx load balancer, FastAPI application replicas, a Redis cache, and a PostgreSQL database, all orchestrated with Docker Compose.
By the end of this project, you'll have:
- A documented High-Level Design and Low-Level Design with component responsibilities, API contracts, and a caching strategy.
- A working Docker Compose stack running 5 containers (Nginx, 2 FastAPI replicas, Redis, PostgreSQL) on your machine.
- A before/after performance story showing a bottleneck identified with k6 load testing and fixed with a design-level Redis caching change.
- Secret Mission: Add connection pooling and database query optimization as a second-layer fix, then run a final load test to show compounding performance gains.
Are there any prerequisites?
None. This project includes full setup instructions for Python, Docker Desktop, and k6. You just need a computer running Windows 11 with at least 8 GB of RAM.
Before We Start
You are about to build a distributed order-processing system from scratch. That means multiple services running together, talking to each other, and breaking under pressure.
Before you write any code, you need three tools on your machine. Python runs the application logic, Docker Desktop runs all the services in containers, and k6 stress-tests everything to find where it breaks.
In this step, get ready to:
- Install Python 3.13 for building the FastAPI order service.
- Install Docker Desktop with WSL 2 for running containers locally.
- Install k6 for load testing the system under stress.
Open a terminal
You will use the Windows Terminal for all commands in this project.
- Press the Windows key to open your search bar.
- Type Terminal and press Enter to open it.
Install Python 3.13
Python is the language you will use to build the FastAPI order service. This project requires version 3.13 or higher.
- Check if Python is already installed by running this command in your terminal:
python --versionLook at what your terminal prints and pick the tab that matches what you see.
✔️ I see version 3.13 or higher
Python is ready to go. Skip ahead to the next substep.
ⓧ I see an older version
You have Python installed, but you need version 3.13 or higher for this project. You can install it alongside your existing version.
- Open your browser and go to:
- https://www.python.org/downloads/release/python-31314/
- Scroll down to the Files section and click the Windows installer (64-bit) link to download it.
- Run the downloaded installer.
- On the first screen, check the box labelled Add python.exe to PATH.
- Click Install Now.
- After the install finishes, close and reopen your terminal.
- Verify the new version by running:
python --versionYou should see Python 3.13.14 or similar.
ⓧ Command not found
Python is not installed yet. Follow these steps to install it.
- Open your browser and go to:
- https://www.python.org/downloads/release/python-31314/
- Scroll down to the Files section and click the Windows installer (64-bit) link to download it.
- Run the downloaded installer.
- On the first screen, check the box labelled Add python.exe to PATH. This is important. Without it, your terminal will not recognise the python command.
- Click Install Now.
- After the install finishes, close and reopen your terminal.
- Verify Python is working by running:
python --versionYou should see Python 3.13.14 or similar.
Still seeing command not found after installing?
Make sure you checked Add python.exe to PATH during the install. If you missed it, re-run the installer, click Modify, and enable the PATH option.
Also make sure you closed and reopened your terminal after installing. The old terminal session does not pick up PATH changes.
Help me troubleshoot why Python isn't being recognised in my terminal after installation.
Install Docker Desktop
Docker Desktop lets you run multiple services in isolated containers on your machine. It needs WSL 2 (Windows Subsystem for Linux) as its backend engine.
- Check if Docker is already installed by running this command:
docker compose versionPick the tab that matches what you see.
✔️ I see a version number
Docker Desktop is installed and Docker Compose is available. Skip ahead to the next substep.
- Make sure Docker Desktop is running. Press the Windows key, type Docker Desktop, and press Enter to open it if it is not already running.
ⓧ Command not found
Docker is not installed. You will set up WSL 2 first, then install Docker Desktop.
Enable WSL 2
- Open PowerShell as Administrator. Press the Windows key, type PowerShell, then click Run as administrator on the right panel.
- If a User Account Control prompt appears, click Yes to allow it.
- Enable WSL 2 by running this command:
wsl --installWhat does this command do?
This enables the Windows Subsystem for Linux feature and installs a default Linux distribution. Docker Desktop uses WSL 2 as its lightweight virtual machine backend to run Linux containers.
- Restart your computer if prompted. WSL 2 requires a reboot to finish setup.
- After restarting, reopen your terminal.
Download and install Docker Desktop
- Open your browser and go to:
- https://www.docker.com/products/docker-desktop/
- Click the download button to get Docker Desktop Installer.exe.
- Run Docker Desktop Installer.exe.
- The installer asks which installation mode you prefer. Choose Per-user installation. This does not require administrator privileges and uses the WSL 2 backend automatically.
- Wait for the installation to complete.
- Launch Docker Desktop. Press the Windows key, type Docker Desktop, and press Enter.
- Docker Desktop may take a minute to start for the first time. Wait until the Docker icon in your system tray shows a steady state (not animating).
- Verify Docker Compose is available by running this command in your terminal:
docker compose versionYou should see a version number printed in the terminal.
Docker Desktop not starting?
Make sure you restarted your computer after running wsl --install. WSL 2 requires a full reboot to activate.
If you see a message about virtualization, check that hardware virtualization is enabled in your BIOS settings. Most modern machines have it on by default.
Help me get Docker Desktop running on my Windows 11 machine.
Install k6
k6 is a load testing tool that simulates many users hitting your service at once. You will use it to find where the system breaks under pressure.
- Check if k6 is already installed by running:
k6 versionPick the tab that matches what you see.
✔️ I see a version number
k6 is ready. You are all set.
ⓧ Command not found
k6 is not installed. Install it using the Windows package manager.
- Install k6 by running this command:
winget install k6 --source wingetWhat does this command do?
This uses winget, the built-in Windows package manager, to download and install k6 from the official source. No manual download required.
- After it finishes, close and reopen your terminal so the new command is available.
- Verify k6 is installed by running:
k6 versionYou should see a version number printed.
winget not recognised?
The winget tool comes pre-installed on Windows 11. If your terminal does not recognise it, make sure you are using the default Windows Terminal (not an older Command Prompt shortcut).
Help me install k6 on Windows when winget is not working.
Your development environment is ready. Next up, you will design the architecture of your distributed system before writing a single line of code.
Set Up Your Tools
Your tools are installed and verified. Now you need a dedicated workspace on your machine where all the project files will live.
In this step, you'll create the project folder structure and open it in a text editor so you're ready to start building.
In this step, get ready to:
- Create the project directory and application subfolder.
- Open the project in VS Code.
Create the project directory
Your project needs two folders. The root folder holds Docker configuration and infrastructure files. A subfolder called app/ holds the Python application code.
- Move to your Desktop by running this command:
cd ~/Desktop- Create your project folder and the application subfolder by running these commands:
mkdir order-system
cd order-system
mkdir app- Confirm the folders were created by running:
lsYou should see app listed in the output.
Don't see the app folder?
- If you see an error saying the directory already exists, that means it was created previously. Run cd order-system to enter it, then run ls to check whether app is already inside.
- If app is missing, run mkdir app again from inside the order-system folder.
- Make sure your terminal prompt shows you are inside order-system before running ls.
Help me troubleshoot my project directory setup.
Open VS Code to the project folder
You'll use VS Code as your text editor for this project. Any editor works, but VS Code has built-in terminal and file explorer panels that make managing multiple files easier.
- Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
- Type Visual Studio Code and press Enter to open it.
- Click File in the top menu bar.
- Click Open Folder.
- Navigate to your Desktop in the file dialog.
- Select the order-system folder.
- Click Select Folder.
VS Code opens with the order-system folder in the Explorer sidebar on the left. You should see the app subfolder listed.
Using a different editor?
Any text editor works for this project. Just make sure you can see both the order-system root folder and the app subfolder in your editor's file tree.
VS Code not found?
If searching for Visual Studio Code returns no results, download and install it from code.visualstudio.com. Run the installer with default settings, then try searching again.
Help me install VS Code on Windows.
Your workspace is ready. Next up, you'll design the architecture of your distributed system before writing a single line of code.
Design the Architecture
Your tools are installed and your workspace is ready. Now it is time to design the system before writing a single line of code.
In this step, you will create two architecture documents that guide every decision in the build phase. These are the same documents that real engineering teams produce before building distributed systems.
In this step, get ready to:
- Create a High-Level Design (HLD) with a component diagram and responsibilities.
- Create a Low-Level Design (LLD) with API contract, data model, and caching strategy.
- Confirm your project directory structure is ready for development.
Create the High-Level Design
A High-Level Design (HLD) shows the big picture of your system. It names every major component and shows how data flows between them.
Your order-processing system has five components arranged in a pipeline. Grab a piece of paper, open a notes app, or use any tool you like to sketch the following diagram.
- Write out your component flow diagram using the layout below as your reference:
Client
|
v
Nginx (Load Balancer)
| |
v v
FastAPI-1 FastAPI-2
| |
v v
Redis Cache
|
v
PostgreSQL (Database)What is a High-Level Design?
An HLD is a blueprint that shows which components exist and how they connect. It does not specify implementation details like code or configuration.
Engineering teams use HLDs to align on architecture before anyone starts coding. It prevents costly rework by catching design issues early.
- Below your diagram, document the responsibility of each component:
- Nginx (Load Balancer): Receives all incoming HTTP requests from the client and distributes them evenly across the two FastAPI replicas.
- FastAPI Service (x2 replicas): Handles the application logic for creating, retrieving, and listing orders. Two copies run in parallel so one replica can serve requests while the other is busy.
- Redis Cache: Stores recently accessed order data in memory so repeated requests skip the database entirely.
- PostgreSQL (Database): Stores all order records permanently. This is the single source of truth for your data.
Now identify where bottlenecks typically hide. Write these three locations in your notes.
- Network/Load Balancer layer: Nginx can become a single point of failure if it cannot handle the incoming request volume.
- Application layer: The FastAPI replicas can run out of CPU or memory if each request does expensive work (like complex queries or unoptimized loops).
- Database layer: PostgreSQL can become overwhelmed when every single request hits the database directly, even for the same data that was just fetched a moment ago.
Why identify bottleneck locations upfront?
Performance engineers do not guess where problems live. They design systems with known pressure points, then use load testing to confirm which layer fails first under stress.
Your HLD now gives you a map. When k6 reveals a bottleneck later, you will know exactly which layer to investigate.
Create the Low-Level Design
A Low-Level Design (LLD) zooms into the details. It specifies the exact API endpoints, data shapes, and strategies your code will implement.
- Document the API contract with these three endpoints:
- GET /orders returns a list of all orders.
- GET /orders/{id} returns a single order by its ID.
- POST /orders creates a new order and saves it to the database.
- Document the data model for the Order entity with these fields:
- id (integer, primary key, auto-generated).
- customer_name (string, required).
- product (string, required).
- quantity (integer, required).
- status (string, defaults to pending).
- created_at (datetime, auto-generated on creation).
- Document the caching strategy:
- Cache target: GET /orders/{id} responses stored in Redis.
- TTL: 30 seconds. After 30 seconds, the cached entry expires and the next request fetches fresh data from PostgreSQL.
- Cache miss: Query PostgreSQL, store the result in Redis, then return the response.
- Cache hit: Return the cached response directly, skipping PostgreSQL entirely.
What is a TTL?
TTL stands for Time To Live. It tells Redis how many seconds to keep a cached value before automatically deleting it.
A 30-second TTL means cached data may be slightly stale, but your database never gets hammered by repeated identical requests. This trade-off is standard for read-heavy APIs.
- Document the load balancing strategy:
- Method: Nginx round-robin across two replicas (api-1 and api-2).
- Behaviour: Each incoming request goes to the next replica in rotation. Request 1 goes to api-1, request 2 goes to api-2, request 3 goes back to api-1, and so on.
Why round-robin instead of random distribution?
Round-robin guarantees even distribution across replicas. Random distribution can cluster requests on one replica by chance, especially under light load.
For this project, round-robin keeps both FastAPI instances equally busy so you can observe bottleneck behaviour consistently.
Confirm your project directory structure
Your order-system folder and its app/ subdirectory are already in place from the previous step. Here is how the files will map to your architecture.
- Switch back to VS Code with your order-system folder open.
- Confirm your folder structure matches this layout:
order-system/
├── app/ # FastAPI application code goes here
├── compose.yaml # Docker Compose orchestration (created next step)
├── Dockerfile # Container image definition (created next step)
├── nginx.conf # Load balancer config (created next step)
├── requirements.txt # Python dependencies (created next step)
└── load-test.js # k6 test script (created in Step 4)What does this structure mean?
The app/ directory holds your FastAPI service code. Every other file sits at the project root because Docker Compose expects the compose.yaml and Dockerfile in the same directory where you run the build command.
- Verify your app/ folder is visible in the VS Code Explorer sidebar. You will add your Python code here in the next step.
Your architecture is fully documented. Next up, you will turn this design into a working multi-container system using Docker Compose.
Build the Distributed System
Your architecture is designed. You have a documented HLD showing how data flows from client to Nginx to your FastAPI replicas, and an LLD specifying the exact API contract and caching strategy.
Now it is time to turn that design into running code. In this step, you will write the FastAPI order service, package it with Docker, wire all five containers together with Docker Compose, and verify the system responds to real HTTP requests.
In this step, get ready to:
- Write the FastAPI order service with three endpoints (list, get, create).
- Create the Dockerfile, Nginx config, and compose.yaml to orchestrate all containers.
- Start the system and verify the API responds through the load balancer.
Write the FastAPI order service
The order service is the application layer of your distributed system. It connects to PostgreSQL for persistence and Redis for caching. For now, you will deliberately skip the caching logic on the get-order endpoint. This creates a real bottleneck you will discover later through load testing.
- Create the requirements.txt file in the root of your order-system folder by pasting the following:
fastapi[standard]==0.138.0
sqlalchemy==2.0.41
psycopg2-binary==2.9.10
redis==5.3.0What do these packages do?
- fastapi[standard] is the web framework that handles HTTP requests and automatic API docs.
- sqlalchemy is the ORM that maps Python objects to database rows.
- psycopg2-binary is the PostgreSQL driver that SQLAlchemy uses under the hood.
- redis is the Python client for communicating with the Redis cache server.
- Create an empty app/__init__.py file inside the app directory. This tells Python to treat app as a package.
Now for the main application file. You will build it in two parts.
- Create app/main.py and paste the following imports, configuration, and data models:
import os
import json
import time
from datetime import datetime
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from sqlalchemy import create_engine, Column, Integer, String, Float, DateTime, text
from sqlalchemy.orm import sessionmaker, declarative_base
import redis
# Create the FastAPI application instance
app = FastAPI(title="Order Service")
# Database and cache connection strings from environment variables
DATABASE_URL = os.environ.get(
"DATABASE_URL",
"postgresql://orders_user:orders_pass@postgres:5432/orders_db",
)
REDIS_URL = os.environ.get("REDIS_URL", "redis://redis:6379/0")
# Set up SQLAlchemy engine and session factory
engine = create_engine(DATABASE_URL, pool_pre_ping=True)
SessionLocal = sessionmaker(bind=engine)
Base = declarative_base()
# Redis client for caching (used later)
redis_client = redis.Redis.from_url(REDIS_URL, decode_responses=True)
# Cache time-to-live in seconds
CACHE_TTL = 30
# SQLAlchemy model representing an order in the database
class Order(Base):
__tablename__ = "orders"
id = Column(Integer, primary_key=True, index=True)
customer_name = Column(String, nullable=False)
product = Column(String, nullable=False)
quantity = Column(Integer, nullable=False)
status = Column(String, default="pending")
created_at = Column(DateTime, default=datetime.utcnow)
# Pydantic model for incoming order creation requests
class OrderCreate(BaseModel):
customer_name: str
product: str
quantity: int
# Pydantic model for order responses
class OrderResponse(BaseModel):
id: int
customer_name: str
product: str
quantity: int
status: str
created_at: strWhat does this code do?
The top section sets up the connections your service needs. DATABASE_URL and REDIS_URL are read from environment variables so Docker Compose can inject them at runtime.
The Order class defines the database table schema. OrderCreate validates incoming POST requests. OrderResponse shapes the JSON sent back to clients.
- Add the following endpoint functions directly below the code you just pasted in app/main.py:
# Create tables on startup if they don't exist
@app.on_event("startup")
def on_startup():
Base.metadata.create_all(bind=engine)
# List all orders (limited to 50)
@app.get("/orders")
def list_orders():
db = SessionLocal()
try:
orders = db.query(Order).limit(50).all()
return [
OrderResponse(
id=o.id,
customer_name=o.customer_name,
product=o.product,
quantity=o.quantity,
status=o.status,
created_at=o.created_at.isoformat(),
)
for o in orders
]
finally:
db.close()
# Get a single order by ID (NO caching - this is the deliberate bottleneck)
@app.get("/orders/{order_id}")
def get_order(order_id: int):
db = SessionLocal()
try:
order = db.query(Order).filter(Order.id == order_id).first()
if not order:
raise HTTPException(status_code=404, detail="Order not found")
response = OrderResponse(
id=order.id,
customer_name=order.customer_name,
product=order.product,
quantity=order.quantity,
status=order.status,
created_at=order.created_at.isoformat(),
)
return response
finally:
db.close()
# Create a new order
@app.post("/orders", status_code=201)
def create_order(order_in: OrderCreate):
db = SessionLocal()
try:
order = Order(
customer_name=order_in.customer_name,
product=order_in.product,
quantity=order_in.quantity,
)
db.add(order)
db.commit()
db.refresh(order)
return OrderResponse(
id=order.id,
customer_name=order.customer_name,
product=order.product,
quantity=order.quantity,
status=order.status,
created_at=order.created_at.isoformat(),
)
finally:
db.close()
# Simple health check endpoint
@app.get("/health")
def health():
return {"status": "ok"}Why is there no caching on get_order?
Notice that get_order queries PostgreSQL directly on every single request, even if someone fetches the same order 1000 times. This is intentional. Your LLD specifies a Redis cache here, but you are skipping it for now.
In the load testing step, you will discover this bottleneck through real metrics. Then you will add the cache fix and measure the improvement.
- Save app/main.py.
✔️ Awesome, I've got everything!
Great. Double check you have saved both requirements.txt and app/main.py.
ⓧ I'd like to double check the full code
Here is the full app/main.py file:
import os
import json
import time
from datetime import datetime
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from sqlalchemy import create_engine, Column, Integer, String, Float, DateTime, text
from sqlalchemy.orm import sessionmaker, declarative_base
import redis
app = FastAPI(title="Order Service")
DATABASE_URL = os.environ.get(
"DATABASE_URL",
"postgresql://orders_user:orders_pass@postgres:5432/orders_db",
)
REDIS_URL = os.environ.get("REDIS_URL", "redis://redis:6379/0")
engine = create_engine(DATABASE_URL, pool_pre_ping=True)
SessionLocal = sessionmaker(bind=engine)
Base = declarative_base()
redis_client = redis.Redis.from_url(REDIS_URL, decode_responses=True)
CACHE_TTL = 30
class Order(Base):
__tablename__ = "orders"
id = Column(Integer, primary_key=True, index=True)
customer_name = Column(String, nullable=False)
product = Column(String, nullable=False)
quantity = Column(Integer, nullable=False)
status = Column(String, default="pending")
created_at = Column(DateTime, default=datetime.utcnow)
class OrderCreate(BaseModel):
customer_name: str
product: str
quantity: int
class OrderResponse(BaseModel):
id: int
customer_name: str
product: str
quantity: int
status: str
created_at: str
@app.on_event("startup")
def on_startup():
Base.metadata.create_all(bind=engine)
@app.get("/orders")
def list_orders():
db = SessionLocal()
try:
orders = db.query(Order).limit(50).all()
return [
OrderResponse(
id=o.id,
customer_name=o.customer_name,
product=o.product,
quantity=o.quantity,
status=o.status,
created_at=o.created_at.isoformat(),
)
for o in orders
]
finally:
db.close()
@app.get("/orders/{order_id}")
def get_order(order_id: int):
db = SessionLocal()
try:
order = db.query(Order).filter(Order.id == order_id).first()
if not order:
raise HTTPException(status_code=404, detail="Order not found")
response = OrderResponse(
id=order.id,
customer_name=order.customer_name,
product=order.product,
quantity=order.quantity,
status=order.status,
created_at=order.created_at.isoformat(),
)
return response
finally:
db.close()
@app.post("/orders", status_code=201)
def create_order(order_in: OrderCreate):
db = SessionLocal()
try:
order = Order(
customer_name=order_in.customer_name,
product=order_in.product,
quantity=order_in.quantity,
)
db.add(order)
db.commit()
db.refresh(order)
return OrderResponse(
id=order.id,
customer_name=order.customer_name,
product=order.product,
quantity=order.quantity,
status=order.status,
created_at=order.created_at.isoformat(),
)
finally:
db.close()
@app.get("/health")
def health():
return {"status": "ok"}Write the Dockerfile and compose.yaml
With the application code written, you need to package it as a container image and define how all five services connect to each other.
- Create a Dockerfile in the root of your order-system folder and paste the following:
FROM python:3.13
WORKDIR /code
# Install Python dependencies
COPY ./requirements.txt /code/requirements.txt
RUN pip install --no-cache-dir --upgrade -r /code/requirements.txt
# Copy application code
COPY ./app /code/app
# Run FastAPI with proxy headers for Nginx
CMD ["fastapi", "run", "app/main.py", "--port", "80", "--proxy-headers"]What does this Dockerfile do?
- FROM python:3.13 uses the official Python 3.13 base image.
- Requirements are copied and installed before the app code. This means Docker caches the dependency layer and only reinstalls when requirements.txt changes.
- --proxy-headers tells FastAPI to trust the forwarded headers from Nginx so client IPs are logged correctly.
- Save Dockerfile.
- Create nginx.conf in the root of your order-system folder and paste the following:
upstream api {
server api-1:80;
server api-2:80;
}
server {
listen 80;
location / {
proxy_pass http://api;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}What does this Nginx config do?
The upstream api block defines a group of backend servers. Nginx distributes incoming requests across api-1 and api-2 using round-robin. These names match the service names in your compose.yaml, and Docker Compose networking resolves them automatically.
- Save nginx.conf.
- Create compose.yaml in the root of your order-system folder and paste the following:
services:
nginx:
image: nginx:alpine
ports:
- "80:80"
volumes:
- ./nginx.conf:/etc/nginx/conf.d/default.conf
depends_on:
- api-1
- api-2
api-1:
build: .
environment:
- DATABASE_URL=postgresql://orders_user:orders_pass@postgres:5432/orders_db
- REDIS_URL=redis://redis:6379/0
depends_on:
- postgres
- redis
api-2:
build: .
environment:
- DATABASE_URL=postgresql://orders_user:orders_pass@postgres:5432/orders_db
- REDIS_URL=redis://redis:6379/0
depends_on:
- postgres
- redis
postgres:
image: postgres:16-alpine
environment:
- POSTGRES_USER=orders_user
- POSTGRES_PASSWORD=orders_pass
- POSTGRES_DB=orders_db
volumes:
- pgdata:/var/lib/postgresql/data
redis:
image: redis:7-alpine
volumes:
pgdata:What does this compose.yaml do?
This file defines your entire distributed system as five services. nginx acts as the load balancer on port 80. api-1 and api-2 are identical FastAPI replicas built from your Dockerfile. postgres stores orders persistently. redis provides the cache layer.
The depends_on keys ensure services start in the right order. The pgdata volume persists database data between container restarts.
- Save compose.yaml.
File not in the right place?
Make sure Dockerfile, nginx.conf, compose.yaml, and requirements.txt are all in the root of your order-system folder (not inside the app subdirectory). The app/main.py and app/__init__.py files go inside the app folder.
Help me check my file structure
Start the system and verify
With all files in place, you can now bring up the entire distributed system with a single command.
- Open a terminal and navigate to your order-system folder.
- Start all five containers by running this command:
docker compose up --build -dWhat does this command do?
- docker compose up creates and starts all services defined in compose.yaml.
- --build rebuilds the Docker images from your Dockerfile before starting.
- -d runs containers in detached mode (in the background) so you get your terminal back.
- Wait for the build to complete. The first run downloads base images and installs packages, which takes a few minutes.
- Verify all containers are running by executing:
docker compose psYou should see 5 services listed with a status of "running" or "Up": nginx, api-1, api-2, redis, and postgres.
Containers not running?
If a container shows "exited" status, check its logs by running docker compose logs api-1 (replace api-1 with the service name that failed). Common issues include port 80 already in use or Docker Desktop not running.
Help me troubleshoot my containers
- Test the API responds through Nginx by running:
curl http://localhost/ordersYou should see an empty JSON array [] because the database has no orders yet.
- Seed the database with test orders by running these commands:
curl -X POST http://localhost/orders -H "Content-Type: application/json" -d "{\"customer_name\": \"Alice\", \"product\": \"Widget\", \"quantity\": 3}"
curl -X POST http://localhost/orders -H "Content-Type: application/json" -d "{\"customer_name\": \"Bob\", \"product\": \"Gadget\", \"quantity\": 1}"
curl -X POST http://localhost/orders -H "Content-Type: application/json" -d "{\"customer_name\": \"Charlie\", \"product\": \"Sprocket\", \"quantity\": 5}"Each command should return a JSON object with the created order and a 201 status code.
- Confirm the orders are stored by running:
curl http://localhost/ordersYou should see a JSON array with three orders listed.
Seeing connection refused?
Make sure Docker Desktop is running and all 5 containers are in the "running" state. If curl reports a connection error, Nginx might still be starting up. Wait 10 seconds and try again.
Help me debug my connection issue
Your distributed system is alive. Five containers are working together. Next up, you will write a k6 load test to hammer this system and find exactly where it breaks.
Load Test and Find the Bottleneck
Your distributed system is up and running. Five containers are working together to serve order data through a load balancer. But how does this architecture actually perform under pressure?
In this step, you will write a k6 load test script that simulates dozens of users hitting your API simultaneously. You will run it, watch the metrics, and pinpoint exactly which component buckles first.
In this step, get ready to:
- Write a k6 load test script that simulates realistic traffic.
- Run the baseline load test and record performance metrics.
- Identify and document the bottleneck in your system.
Write the k6 load test script
A load test script tells k6 what traffic pattern to simulate. You define how many virtual users (VUs) ramp up, how long they sustain load, and what endpoints they hit.
- Create a new file called load-test.js in the root of your project folder (the same level as compose.yaml).
- Add the test configuration with the k6 imports and stage definitions by pasting this code:
import http from "k6/http";
import { check, sleep } from "k6";
// Define the load test stages
export const options = {
stages: [
{ duration: "30s", target: 50 }, // Ramp up to 50 virtual users
{ duration: "30s", target: 50 }, // Hold at 50 virtual users
{ duration: "10s", target: 0 }, // Ramp down to 0
],
};What does this code do?
The stages array defines a traffic ramp pattern. Over 30 seconds, k6 gradually spins up 50 virtual users. It holds that load for another 30 seconds, then ramps back down over 10 seconds.
This pattern mimics real-world traffic spikes. The sustained 50-user phase is where bottlenecks become visible because your system has no time to recover between requests.
- Below the options block, add the default function that defines what each virtual user does:
export default function () {
// Pick a random order ID between 1 and 10
const orderId = Math.floor(Math.random() * 10) + 1;
const res = http.get(`http://localhost/orders/${orderId}`);
// Validate the response
check(res, {
"status is 200": (r) => r.status === 200,
"response time < 500ms": (r) => r.timings.duration < 500,
});
// Brief pause between requests to simulate real user behavior
sleep(0.5);
}What does this code do?
Every virtual user runs this function in a loop for the duration of the test. Each iteration picks a random order ID (1 through 10), sends a GET request to /orders/{id} through the Nginx load balancer, and validates the response.
The check function tracks two things: whether the response returned a 200 status, and whether it came back in under 500ms. The sleep(0.5) call adds a half-second pause between requests so traffic feels like real users, not a flood of simultaneous connections.
- Save load-test.js.
- Confirm the file exists by running:
ls load-test.jsYou should see load-test.js listed in the output.
File not found?
Make sure you created the file in the same folder as your compose.yaml. If you are in a different directory, cd back to your project root before running ls.
Help me find my load-test.js file
✔️ Awesome, I've got everything!
Great. Make sure you have saved your file before moving on.
ⓧ I'd like to double check the full code
import http from "k6/http";
import { check, sleep } from "k6";
export const options = {
stages: [
{ duration: "30s", target: 50 },
{ duration: "30s", target: 50 },
{ duration: "10s", target: 0 },
],
};
export default function () {
const orderId = Math.floor(Math.random() * 10) + 1;
const res = http.get(`http://localhost/orders/${orderId}`);
check(res, {
"status is 200": (r) => r.status === 200,
"response time < 500ms": (r) => r.timings.duration < 500,
});
sleep(0.5);
}Run the baseline load test
Your system currently has no caching on the GET /orders/{id} endpoint. Every single request goes directly to PostgreSQL. This baseline test captures how the system performs before any optimization.
- Open a second terminal window. You will use this to monitor container resource usage while the load test runs.
- In the second terminal, start watching container stats by running:
docker statsWhat does docker stats show?
This command streams live CPU and memory usage for each running container. During the load test, watch the postgres container. If it shows high CPU usage compared to the other containers, that confirms the database is working the hardest.
- Switch back to your first terminal. Make sure your Docker containers are still running by checking:
docker compose psYou should see all 5 containers listed with a "running" state.
- Execute the k6 load test by running:
k6 run load-test.jsWhat should I see?
The test takes about 70 seconds total (30s ramp-up + 30s sustained + 10s ramp-down). While it runs, k6 prints a progress bar showing virtual users and completed iterations.
When it finishes, k6 outputs a summary table with key metrics. Look for these values and write them down:
- http_req_duration (p95) tells you the 95th percentile response time.
- http_reqs tells you total requests per second.
- checks tells you how many responses passed your validations.
- Record your baseline results. Write down the p95 response time, the requests/second rate, and the check pass percentage.
Seeing connection refused errors?
If k6 reports connection errors, your Docker containers may have stopped. Run docker compose up --build again and wait for all services to start before re-running the test.
Help me troubleshoot k6 connection errors
Identify the bottleneck
The k6 results and docker stats output together tell a clear story about where your system struggles.
- Look at your docker stats output from the second terminal. The postgres container should have shown the highest CPU usage during the test.
- Compare the CPU usage of postgres against api-1, api-2, and nginx. The database container is doing the most work because every request results in a fresh query.
- Press Ctrl+C in the second terminal to stop docker stats.
Why is PostgreSQL the bottleneck?
Look at the get_order function in app/main.py. Every call to GET /orders/{id} opens a database connection, runs a SQL query, serializes the result, and closes the connection. With 50 virtual users calling this in a loop, PostgreSQL handles hundreds of identical queries per second for the same 10 orders.
Meanwhile, Redis is sitting idle. The system has a cache available but nobody is using it. This is the exact bottleneck your LLD predicted: the GET /orders/{id} endpoint needs a caching layer.
- Document your findings. Write down this conclusion: "Every GET /orders/{id} request hits PostgreSQL directly with no caching. Under load, the database becomes the bottleneck because it processes hundreds of redundant queries for the same order data."
You now have a baseline performance story with hard numbers. Next, you will implement the Redis caching fix from your LLD and measure exactly how much it improves response times.
Fix the Bottleneck and Measure Improvement
Your baseline load test told a clear story: every single GET /orders/{id} request travels all the way to PostgreSQL, even when the same order is requested hundreds of times per second.
Your LLD already specifies the fix: cache order lookups in Redis with a 30-second TTL. In this step, you will implement that cache layer, rebuild the system, and re-run the same k6 script to measure the improvement.
In this step, get ready to:
- Add Redis caching to the GET /orders/{id} endpoint.
- Rebuild the Docker containers with the updated code.
- Re-run the k6 load test and document the before/after comparison.
Implement the cache layer
The caching pattern you will implement is called cache-aside (also known as lazy loading). The application checks Redis first, and only queries PostgreSQL on a cache miss.
- In your text editor, open app/main.py.
- Find the get_order function (the one decorated with @app.get("/orders/{order_id}")).
- Replace the entire function with this updated version that checks Redis before hitting the database:
@app.get("/orders/{order_id}")
def get_order(order_id: int):
# Check Redis cache first
cached = redis_client.get(f"order:{order_id}")
if cached:
return json.loads(cached)
# Cache miss - query PostgreSQL
db = SessionLocal()
try:
order = db.query(Order).filter(Order.id == order_id).first()
if not order:
raise HTTPException(status_code=404, detail="Order not found")
response = OrderResponse(
id=order.id,
customer_name=order.customer_name,
product=order.product,
quantity=order.quantity,
status=order.status,
created_at=order.created_at.isoformat(),
)
# Store in Redis with 30-second TTL
redis_client.setex(
f"order:{order_id}", CACHE_TTL, json.dumps(response.model_dump())
)
return response
finally:
db.close()What does this code do?
The function now has two paths depending on whether the order is already cached.
- redis_client.get(f"order:{order_id}") checks if a cached copy exists in Redis. If it does, the function returns it immediately without touching PostgreSQL.
- On a cache miss, the function queries PostgreSQL as before, but now also stores the result in Redis using redis_client.setex() with a 30-second TTL (CACHE_TTL).
- After the first request for an order, every subsequent request within 30 seconds is served from Redis at in-memory speed.
- Save app/main.py.
Not sure you replaced the right function?
Make sure you replaced the function that starts with @app.get("/orders/{order_id}"), not @app.get("/orders") (the list endpoint). The list endpoint should remain unchanged.
Also confirm the redis_client and CACHE_TTL variables are still defined near the top of the file. They were created in Step 3.
Help me debug my get_order function changes
✔️ Awesome, I've got everything!
Great. Double check you have saved your file before moving on.
ⓧ I'd like to double check the full code
Here is the full app/main.py file with the caching fix applied:
import os
import json
import time
from datetime import datetime
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from sqlalchemy import create_engine, Column, Integer, String, Float, DateTime, text
from sqlalchemy.orm import sessionmaker, declarative_base
import redis
app = FastAPI(title="Order Service")
DATABASE_URL = os.environ.get(
"DATABASE_URL",
"postgresql://orders_user:orders_pass@postgres:5432/orders_db",
)
REDIS_URL = os.environ.get("REDIS_URL", "redis://redis:6379/0")
engine = create_engine(DATABASE_URL, pool_pre_ping=True)
SessionLocal = sessionmaker(bind=engine)
Base = declarative_base()
redis_client = redis.Redis.from_url(REDIS_URL, decode_responses=True)
CACHE_TTL = 30
class Order(Base):
__tablename__ = "orders"
id = Column(Integer, primary_key=True, index=True)
customer_name = Column(String, nullable=False)
product = Column(String, nullable=False)
quantity = Column(Integer, nullable=False)
status = Column(String, default="pending")
created_at = Column(DateTime, default=datetime.utcnow)
class OrderCreate(BaseModel):
customer_name: str
product: str
quantity: int
class OrderResponse(BaseModel):
id: int
customer_name: str
product: str
quantity: int
status: str
created_at: str
@app.on_event("startup")
def on_startup():
Base.metadata.create_all(bind=engine)
@app.get("/orders")
def list_orders():
db = SessionLocal()
try:
orders = db.query(Order).limit(50).all()
return [
OrderResponse(
id=o.id,
customer_name=o.customer_name,
product=o.product,
quantity=o.quantity,
status=o.status,
created_at=o.created_at.isoformat(),
)
for o in orders
]
finally:
db.close()
@app.get("/orders/{order_id}")
def get_order(order_id: int):
cached = redis_client.get(f"order:{order_id}")
if cached:
return json.loads(cached)
db = SessionLocal()
try:
order = db.query(Order).filter(Order.id == order_id).first()
if not order:
raise HTTPException(status_code=404, detail="Order not found")
response = OrderResponse(
id=order.id,
customer_name=order.customer_name,
product=order.product,
quantity=order.quantity,
status=order.status,
created_at=order.created_at.isoformat(),
)
redis_client.setex(
f"order:{order_id}", CACHE_TTL, json.dumps(response.model_dump())
)
return response
finally:
db.close()
@app.post("/orders", status_code=201)
def create_order(order_in: OrderCreate):
db = SessionLocal()
try:
order = Order(
customer_name=order_in.customer_name,
product=order_in.product,
quantity=order_in.quantity,
)
db.add(order)
db.commit()
db.refresh(order)
return OrderResponse(
id=order.id,
customer_name=order.customer_name,
product=order.product,
quantity=order.quantity,
status=order.status,
created_at=order.created_at.isoformat(),
)
finally:
db.close()
@app.get("/health")
def health():
return {"status": "ok"}Rebuild and re-run the load test
Your containers are still running the old code. You need to rebuild the images so Docker picks up the updated app/main.py.
- Rebuild and restart the system by running this command in your terminal:
docker compose up --build -d- Wait for the containers to finish starting, then verify all 5 are running:
docker compose ps- You should see all 5 containers (nginx, api-1, api-2, redis, postgres) with a status of Up.
Containers failing to start?
If api-1 or api-2 shows an error, check the logs for syntax issues in your code:
docker compose logs api-1Look for Python syntax errors or import failures. The most common cause is a missing comma or unmatched parenthesis in the redis_client.setex() call.
Help me debug my Docker container startup failure
Now re-run the exact same k6 load test script from Step 4. Using the same script ensures a fair comparison.
- Run the k6 load test by running:
k6 run load-test.js- Wait for the test to complete (about 70 seconds total: 30s ramp-up, 30s hold, 10s ramp-down).
- When the test finishes, look at the summary output. Note the http_req_duration p(95) value and the total http_reqs count.
What should I see?
The first few requests will still hit PostgreSQL (cache misses). But once each order is cached, all subsequent requests for that order within 30 seconds are served from Redis.
You should see a significantly lower p(95) response time compared to your baseline. The requests-per-second should also be higher because Redis responds much faster than PostgreSQL under load.
Metrics look the same as before?
If your p95 did not improve, the cache might not be working. Test it manually by running the same request twice:
curl http://localhost/orders/1If both responses return the same data but the second feels just as slow, check that redis_client.get() appears BEFORE the database query in your code. Also verify the Redis container is healthy in docker compose ps.
Help me figure out why my Redis cache isn't working
Document the before/after story
Performance work is only valuable if you can communicate the improvement clearly. A simple comparison table makes the impact obvious to anyone reading your documentation.
- Record your results in a comparison like this:
| Metric | Baseline (no cache) | With Redis Cache |
|---------------------|---------------------|------------------|
| p95 response time | [your value] ms | [your value] ms |
| Requests/second | [your value] | [your value] |
| Error rate | [your value] % | [your value] % |What does this code do?
This is a simple markdown table you can keep in a notes file or in the project README. Fill in the actual numbers from your two k6 runs.
The p95 response time is the most important metric. It tells you that 95% of requests completed faster than that threshold. A large drop here means the cache is absorbing load that used to hit the database.
- Fill in the table with your actual numbers from the baseline test (Step 4) and the test you just ran.
What is the trade-off of caching?
Caching improves speed but introduces staleness. With a 30-second TTL, if an order is updated in PostgreSQL, the cached version in Redis will still serve the old data for up to 30 seconds.
For a read-heavy order lookup endpoint, this is an acceptable trade-off. The data changes infrequently compared to how often it is read. In systems where real-time accuracy matters (stock prices, inventory counts), you would use shorter TTLs or cache invalidation strategies.
You now have a complete performance improvement story: you identified a bottleneck through load testing, applied a design-level fix, and proved it worked with data. This is exactly the workflow performance engineers follow in production systems. Next up, there is a Secret Mission that takes your optimization even further with connection pooling and query tuning.
Secret mission
Connection Pooling and Query Optimization
Your Redis cache fixed the single-order bottleneck. But what about the list endpoint that still hammers PostgreSQL on every request? In this mission, you'll add connection pooling and database indexing to squeeze even more performance out of your system.
Clean Up Your Resources
Clean Up Your Resources
Decide whether to keep your resources running, pause them to come back later, or delete them entirely.
This project runs entirely locally, so there are no ongoing costs. The Docker containers do consume RAM while running.
Resources you used:
- Docker containers (nginx, api-1, api-2, redis, postgres) running locally.
- Docker volume (pgdata) storing PostgreSQL data.
- Docker images built during the project.
- Local project files (app/main.py, compose.yaml, nginx.conf, load-test.js, Dockerfile, requirements.txt).
Keep everything running
No action needed. Choose this if you're still actively experimenting with the system or want to run more load tests.
- Your 5 Docker containers continue running and serving requests on port 80.
- Docker Desktop stays running in your system tray.
- Python, Docker Desktop, and k6 remain installed on your machine for future projects.
Pause - I'll come back to this later
Stop the containers to free up RAM without deleting your data. You can restart everything later with a single command.
- Stop all containers by running this command in your project folder:
docker compose stop- Your database data and project files remain on disk.
- To restart later, open a terminal in your project folder and run docker compose start.
Delete - I don't want to use this again
Remove all project containers, volumes, images, and files to start completely fresh.
- Stop and remove all containers plus the database volume by running this in your project folder:
docker compose down -v- Reclaim disk space from Docker images by running:
docker system prune -a- Type y when prompted to confirm.
- Delete your local project files. Navigate to the folder containing your project on your Desktop, then run:
macOS
cd ~/Desktop
rm -rf order-systemWindows
cd ~\Desktop
rmdir /s /q order-system- Confirm the folder is gone by running ls (macOS) or dir (Windows) and checking that your project folder no longer appears.
That's a Wrap!
That's a Wrap!
Nice work! You just designed, built, and stress-tested a distributed order-processing system from scratch.
You've learned how to:
- Design a distributed system with a documented High-Level Design and Low-Level Design, then build it as a multi-container application using Docker Compose (Nginx, FastAPI, Redis, PostgreSQL).
- Write k6 load tests to simulate realistic traffic and identify a real performance bottleneck under stress.
- Apply a design-level fix (Redis caching) and measure the improvement with before/after metrics that prove your fix worked.
- Secret Mission: Added connection pooling and query optimization for compounding performance gains across multiple system layers.
Ready to quiz yourself?