Build a Scalable URL Shortener

Build a distributed URL shortener with load balancing, caching, and scaling.

Introduction

30 Second Summary

Every time you click a shortened link, a whole distributed system springs into action. Your request passes through load balancers, app servers, caches, and databases before you even see the destination page.

In this project, you will build a fully working distributed URL shortener on your own machine using Docker Compose to orchestrate five interconnected services that mirror real production architecture.

What You'll Build

A local distributed system with an Nginx load balancer, three Node.js app servers, a Redis cache, and a PostgreSQL database, all orchestrated with Docker Compose.

By the end of this project, you'll have:

  • A working Docker Compose stack with five services that you can start, stop, and scale with a single command.
  • Hands-on experience with load balancing by watching requests distribute across multiple app servers in real time.
  • A measurable caching layer that proves why caching matters through visible cache hit and miss response headers.
  • Secret Mission: A Redis-based rate limiter that protects your API from abuse.

Are there any prerequisites?

You need Docker Desktop installed on your machine. Docker Desktop includes Docker Compose, so no separate install is needed.

Basic familiarity with HTTP, APIs, and JavaScript will help you follow along. All services run inside Docker containers, so you do not need Node.js or any database installed locally.

Set Up the Project Structure

Every distributed system starts with a blueprint. Before writing any application code, you need to define the infrastructure that your services will run on.

In this step, you will create the project folder structure and write a compose.yaml file that defines your entire stack. This single file tells Docker Compose how to spin up six services that talk to each other by name.

In this step, get ready to:
  • Create the project directory with app/ and nginx/ subdirectories.
  • Define all six services in compose.yaml so they can communicate by name.
  • Confirm Docker Compose is available in your project directory.
Create the project directory

You need a dedicated folder on your Desktop to hold the project files. This folder will contain two subdirectories: one for the Node.js app code, and one for the Nginx load balancer configuration.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir url-shortener
cd url-shortener
  • Create the two subdirectories by running:
mkdir app
mkdir nginx
  • Confirm the directories exist by running:
ls

You should see app and nginx listed in the output.

Don't see both folders?

Make sure you ran each mkdir command from inside the url-shortener folder. Run pwd (macOS/Linux) or Get-Location (Windows PowerShell) to check your current directory.

help me troubleshoot why my project directories are not showing up

Define your services in compose.yaml

A compose.yaml file is the blueprint for a multi-container application. It tells Docker Compose which services to start, what images they use, and how they connect to each other.

Your stack will have six services: an Nginx load balancer, three Node.js app replicas, a Redis cache, and a PostgreSQL database.

  • Open the url-shortener folder in VS Code.
  • Create a new file called compose.yaml in the project root (not inside app/ or nginx/).
  • Paste the following into compose.yaml:
services:
  # Load balancer - routes incoming traffic to app servers
  nginx:
    image: nginx:1.30.2-alpine
    ports:
      - "80:80"
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - app-1
      - app-2
      - app-3

  # App server replica 1
  app-1:
    build: ./app
    environment:
      - INSTANCE_ID=app-1
      - DATABASE_URL=postgres://shortener:shortener@postgres:5432/shortener
      - REDIS_URL=redis://redis:6379
    depends_on:
      - postgres
      - redis

  # App server replica 2
  app-2:
    build: ./app
    environment:
      - INSTANCE_ID=app-2
      - DATABASE_URL=postgres://shortener:shortener@postgres:5432/shortener
      - REDIS_URL=redis://redis:6379
    depends_on:
      - postgres
      - redis

  # App server replica 3
  app-3:
    build: ./app
    environment:
      - INSTANCE_ID=app-3
      - DATABASE_URL=postgres://shortener:shortener@postgres:5432/shortener
      - REDIS_URL=redis://redis:6379
    depends_on:
      - postgres
      - redis

  # Cache layer - stores frequently accessed URLs for fast reads
  redis:
    image: redis:8.6.4-alpine
    ports:
      - "6379:6379"

  # Database - permanent storage for URL mappings
  postgres:
    image: postgres:17-alpine
    environment:
      - POSTGRES_USER=shortener
      - POSTGRES_PASSWORD=shortener
      - POSTGRES_DB=shortener
    ports:
      - "5432:5432"
    volumes:
      - pgdata:/var/lib/postgresql/data

# Persistent storage so database data survives container restarts
volumes:
  pgdata:

What does this file do?

This file defines six services that Docker Compose will start together on the same network:

  • nginx pulls the official Nginx image and maps port 80 on your machine to port 80 inside the container. It mounts your custom config file as read-only.
  • app-1, app-2, app-3 are three identical app servers built from the ./app directory. Each gets a unique INSTANCE_ID so you can tell which one handled a request.
  • redis runs the Redis cache. The app servers connect to it using the service name redis as the hostname.
  • postgres runs the PostgreSQL database. The pgdata volume keeps data intact even if the container restarts.

The depends_on field tells Docker Compose which services to start first. Nginx waits for the app servers, and the app servers wait for the database and cache.

How do services talk to each other?

Docker Compose puts all services on the same internal network automatically. Each service can reach another by using the service name as a hostname. That is why DATABASE_URL uses postgres as the host and REDIS_URL uses redis. No IP addresses needed.

  • Save compose.yaml.

Seeing YAML syntax errors in your editor?

YAML is whitespace-sensitive. Make sure you are using spaces (not tabs) for indentation. Each level of nesting uses exactly two spaces. Check that no lines have trailing spaces or mixed indentation.

help me fix YAML formatting issues in my compose.yaml

✔️ Awesome, I've got everything!

Great. Make sure you have saved compose.yaml before moving on.

ⓧ I'd like to double check the full code

Here is the complete compose.yaml without comments for a clean comparison:

services:
  nginx:
    image: nginx:1.30.2-alpine
    ports:
      - "80:80"
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - app-1
      - app-2
      - app-3

  app-1:
    build: ./app
    environment:
      - INSTANCE_ID=app-1
      - DATABASE_URL=postgres://shortener:shortener@postgres:5432/shortener
      - REDIS_URL=redis://redis:6379
    depends_on:
      - postgres
      - redis

  app-2:
    build: ./app
    environment:
      - INSTANCE_ID=app-2
      - DATABASE_URL=postgres://shortener:shortener@postgres:5432/shortener
      - REDIS_URL=redis://redis:6379
    depends_on:
      - postgres
      - redis

  app-3:
    build: ./app
    environment:
      - INSTANCE_ID=app-3
      - DATABASE_URL=postgres://shortener:shortener@postgres:5432/shortener
      - REDIS_URL=redis://redis:6379
    depends_on:
      - postgres
      - redis

  redis:
    image: redis:8.6.4-alpine
    ports:
      - "6379:6379"

  postgres:
    image: postgres:17-alpine
    environment:
      - POSTGRES_USER=shortener
      - POSTGRES_PASSWORD=shortener
      - POSTGRES_DB=shortener
    ports:
      - "5432:5432"
    volumes:
      - pgdata:/var/lib/postgresql/data

volumes:
  pgdata:
Confirm Docker Compose is ready
  • Confirm Docker Compose is available from your project directory by running:
docker compose version

You should see a version number printed, such as Docker Compose version v2.x.x. This confirms that Docker can read your project structure and is ready to build the stack.

Command not found?

If docker compose version is not recognized, make sure Docker Desktop is running. Look for the Docker whale icon in your system tray (Windows) or menu bar (macOS). Start Docker Desktop if it is not already running, then try again.

help me get docker compose working in my terminal

Your project skeleton is ready. Next up, you will write the Node.js application that actually shortens URLs and serves redirects.

Build the Node.js URL Shortener API

Your project structure and Docker Compose configuration are ready. Now it's time to build the actual application that powers your URL shortener.

Each of the three app services in your compose.yaml will run the same Node.js server. This server needs to handle two core operations: creating short URLs and redirecting visitors to the original destination.

In this step, get ready to:
  • Write the Express API with endpoints for shortening and redirecting URLs.
  • Create a Dockerfile to containerize the Node.js app.
  • Define the project dependencies in package.json.
Write the URL shortener API

The heart of your system is a Node.js server that talks to both PostgreSQL (for permanent storage) and Redis (for caching). You'll write this in a single file called server.js.

  • In your code editor, open the app/ folder inside your project directory.
  • Create a new file called server.js inside app/.
  • Add the imports and connection setup by pasting this code:
const express = require('express');
const { Pool } = require('pg');
const Redis = require('ioredis');
const crypto = require('crypto');

const app = express();
app.use(express.json());

// Identifies which container handled the request
const INSTANCE_ID = process.env.INSTANCE_ID || 'unknown';

// Connect to PostgreSQL using the URL from environment variables
const pool = new Pool({
  connectionString: process.env.DATABASE_URL,
});

// Connect to Redis using the URL from environment variables
const redis = new Redis(process.env.REDIS_URL);

What does this code do?

This block sets up your server's foundations. It imports four libraries: Express for HTTP routing, pg for PostgreSQL queries, ioredis for Redis caching, and crypto for generating random short codes.

INSTANCE_ID reads the container's name from an environment variable. Remember how each app service in your compose.yaml sets INSTANCE_ID=app-1, INSTANCE_ID=app-2, etc.? This is how each container identifies itself in response headers.

  • Below the Redis connection, add the database initialization and code generation functions:
// Creates the urls table if it doesn't exist yet
async function initDB() {
  const client = await pool.connect();
  try {
    await client.query(`
      CREATE TABLE IF NOT EXISTS urls (
        id SERIAL PRIMARY KEY,
        code VARCHAR(8) UNIQUE NOT NULL,
        original_url TEXT NOT NULL,
        created_at TIMESTAMP DEFAULT NOW()
      )
    `);
  } finally {
    client.release();
  }
}

// Generates a random 8-character hex code for short URLs
function generateCode() {
  return crypto.randomBytes(4).toString('hex');
}

What does this code do?

initDB() runs when the server starts. It creates a urls table in PostgreSQL with columns for a unique short code, the original URL, and a timestamp. The IF NOT EXISTS clause means restarting the container won't break anything.

generateCode() creates 4 random bytes and converts them to an 8-character hex string like a3f2b1c9. This becomes the short code in your URL (e.g. http://localhost/a3f2b1c9).

  • Below the generateCode function, add the POST endpoint that creates short URLs:
// Accepts a URL and returns a shortened version
app.post('/shorten', async (req, res) => {
  const { url } = req.body;
  if (!url) {
    return res.status(400).json({ error: 'url is required' });
  }

  const code = generateCode();
  try {
    await pool.query(
      'INSERT INTO urls (code, original_url) VALUES ($1, $2)',
      [code, url]
    );
    // Header reveals which container processed this request
    res.set('X-Served-By', INSTANCE_ID);
    res.json({ short_url: `http://localhost/${code}`, code });
  } catch (err) {
    res.status(500).json({ error: 'Failed to create short URL' });
  }
});

What does this code do?

This route handles POST /shorten requests. It takes a URL from the request body, generates a random short code, and stores the mapping in PostgreSQL.

The X-Served-By header is the key to observing load balancing later. When you send multiple requests, you'll see this header rotate between app-1, app-2, and app-3 as Nginx distributes traffic.

  • Below the POST route, add the GET endpoint for redirects, plus the health check and server startup:
// Looks up a short code and redirects to the original URL
app.get('/:code', async (req, res) => {
  const { code } = req.params;

  // Check Redis cache first
  const cached = await redis.get(`url:${code}`);
  if (cached) {
    res.set('X-Cache', 'HIT');
    res.set('X-Served-By', INSTANCE_ID);
    return res.redirect(301, cached);
  }

  // Cache miss - query PostgreSQL
  const result = await pool.query(
    'SELECT original_url FROM urls WHERE code = $1',
    [code]
  );

  if (result.rows.length === 0) {
    return res.status(404).json({ error: 'Short URL not found' });
  }

  // Store in Redis with a 1-hour TTL for future requests
  const originalUrl = result.rows[0].original_url;
  await redis.set(`url:${code}`, originalUrl, 'EX', 3600);

  res.set('X-Cache', 'MISS');
  res.set('X-Served-By', INSTANCE_ID);
  res.redirect(301, originalUrl);
});

// Health check endpoint (defined before /:code would shadow it)
app.get('/health', (req, res) => {
  res.set('X-Served-By', INSTANCE_ID);
  res.json({ status: 'ok', instance: INSTANCE_ID });
});

// Initialize the database table, then start listening
initDB().then(() => {
  app.listen(3000, () => {
    console.log(`${INSTANCE_ID} listening on port 3000`);
  });
});

What does this code do?

The GET /:code route implements the cache-aside pattern. It checks Redis first. If the URL is cached, it responds immediately with an X-Cache: HIT header. If not, it queries PostgreSQL, stores the result in Redis (with a 1-hour TTL), and responds with X-Cache: MISS.

The /health route lets you check if a container is alive. It's defined before the /:code catch-all route so Express doesn't accidentally match health as a short code.

  • Save app/server.js.

Not sure if you have everything?

Use the double-check tab below to compare your file against the full version. Common mistakes: missing commas between function arguments, mismatched backticks around the SQL query, or forgetting the async keyword before route callbacks.

help me find syntax errors in my server.js

✔️ Awesome, I've got everything!

Great. Double check you've saved app/server.js before moving on.

ⓧ I'd like to double check the full code

const express = require('express');
const { Pool } = require('pg');
const Redis = require('ioredis');
const crypto = require('crypto');

const app = express();
app.use(express.json());

const INSTANCE_ID = process.env.INSTANCE_ID || 'unknown';

const pool = new Pool({
  connectionString: process.env.DATABASE_URL,
});

const redis = new Redis(process.env.REDIS_URL);

async function initDB() {
  const client = await pool.connect();
  try {
    await client.query(`
      CREATE TABLE IF NOT EXISTS urls (
        id SERIAL PRIMARY KEY,
        code VARCHAR(8) UNIQUE NOT NULL,
        original_url TEXT NOT NULL,
        created_at TIMESTAMP DEFAULT NOW()
      )
    `);
  } finally {
    client.release();
  }
}

function generateCode() {
  return crypto.randomBytes(4).toString('hex');
}

app.post('/shorten', async (req, res) => {
  const { url } = req.body;
  if (!url) {
    return res.status(400).json({ error: 'url is required' });
  }

  const code = generateCode();
  try {
    await pool.query(
      'INSERT INTO urls (code, original_url) VALUES ($1, $2)',
      [code, url]
    );
    res.set('X-Served-By', INSTANCE_ID);
    res.json({ short_url: `http://localhost/${code}`, code });
  } catch (err) {
    res.status(500).json({ error: 'Failed to create short URL' });
  }
});

app.get('/:code', async (req, res) => {
  const { code } = req.params;

  const cached = await redis.get(`url:${code}`);
  if (cached) {
    res.set('X-Cache', 'HIT');
    res.set('X-Served-By', INSTANCE_ID);
    return res.redirect(301, cached);
  }

  const result = await pool.query(
    'SELECT original_url FROM urls WHERE code = $1',
    [code]
  );

  if (result.rows.length === 0) {
    return res.status(404).json({ error: 'Short URL not found' });
  }

  const originalUrl = result.rows[0].original_url;
  await redis.set(`url:${code}`, originalUrl, 'EX', 3600);

  res.set('X-Cache', 'MISS');
  res.set('X-Served-By', INSTANCE_ID);
  res.redirect(301, originalUrl);
});

app.get('/health', (req, res) => {
  res.set('X-Served-By', INSTANCE_ID);
  res.json({ status: 'ok', instance: INSTANCE_ID });
});

initDB().then(() => {
  app.listen(3000, () => {
    console.log(`${INSTANCE_ID} listening on port 3000`);
  });
});
Create the Dockerfile

Docker needs instructions to turn your Node.js code into a runnable container image. The Dockerfile tells Docker which base image to use, how to install dependencies, and what command to run at startup.

  • Create a new file called Dockerfile (no extension) inside the app/ folder.
  • Paste the following container build instructions:
FROM node:24-alpine

WORKDIR /app

# Copy dependency manifests first for better Docker layer caching
COPY package.json package-lock.json* ./
RUN npm install --production

# Copy the application code
COPY server.js ./

EXPOSE 3000
CMD ["node", "server.js"]

What does this Dockerfile do?

FROM node:24-alpine starts from a lightweight Node.js image. Alpine-based images are much smaller than full images, so your containers start faster.

The two-step COPY pattern (dependencies first, then code) is a Docker best practice. Docker caches each layer, so if only server.js changes, Docker reuses the cached npm install layer instead of reinstalling everything.

  • Save app/Dockerfile.

Watch out for the filename

The file must be named exactly Dockerfile with a capital D and no file extension. If you accidentally name it Dockerfile.txt or dockerfile, Docker Compose will not find it when building the image.

help me fix my Dockerfile naming issue

✔️ Awesome, I've got everything!

Great. Make sure Dockerfile is saved inside the app/ folder, not the project root.

ⓧ I'd like to double check the full code

FROM node:24-alpine

WORKDIR /app

COPY package.json package-lock.json* ./
RUN npm install --production

COPY server.js ./

EXPOSE 3000
CMD ["node", "server.js"]
Define the project dependencies

The Dockerfile runs npm install, which reads from a package.json file to know what libraries to download. You need to list the three libraries your server imports.

  • Create a new file called package.json inside the app/ folder.
  • Paste the following dependency manifest:
{
  "name": "url-shortener",
  "version": "1.0.0",
  "description": "Scalable URL shortener for system design learning",
  "main": "server.js",
  "scripts": {
    "start": "node server.js"
  },
  "dependencies": {
    "express": "^4.21.2",
    "ioredis": "^5.6.1",
    "pg": "^8.16.0"
  }
}

What are these dependencies?

  • express is the web framework that handles HTTP routing.
  • ioredis is a Redis client for Node.js. Your server uses it to read and write cache entries.
  • pg is the PostgreSQL client. Your server uses it to store and retrieve URL mappings.
  • Save app/package.json.
  • Confirm all three files exist by running this in your terminal:
ls app/

You should see Dockerfile, package.json, and server.js listed in the output.

Missing a file?

If ls app/ doesn't show all three files, make sure you saved each file inside the app/ subfolder, not in the project root. A common mistake is saving server.js or Dockerfile one level too high.

help me find where my files got saved

✔️ Awesome, I've got everything!

Great. All three files are in place and ready to be built into a Docker image.

ⓧ I'd like to double check the full code

{
  "name": "url-shortener",
  "version": "1.0.0",
  "description": "Scalable URL shortener for system design learning",
  "main": "server.js",
  "scripts": {
    "start": "node server.js"
  },
  "dependencies": {
    "express": "^4.21.2",
    "ioredis": "^5.6.1",
    "pg": "^8.16.0"
  }
}

Your application code is written and containerized. Next up, you'll configure Nginx as a load balancer so incoming requests get distributed across all three app containers.

Configure Nginx as a Load Balancer

Your app servers are built and ready to handle requests. But in a real distributed system, clients never talk directly to app servers.

Instead, a load balancer sits in front and distributes incoming requests across multiple servers. In this step, you'll configure Nginx to spread traffic evenly across your three app replicas using round-robin distribution.

In this step, get ready to:
  • Write an Nginx configuration that load-balances across three backend servers.
  • Add debugging headers so you can see which server handled each request.
  • Confirm how the config file reaches the Nginx container via Docker Compose.
Create the Nginx configuration file

Nginx needs a configuration file that tells it where to send incoming traffic. You'll define an upstream block listing your three app servers, and a server block that proxies all requests to that group.

  • Create the nginx directory inside your project by running this command:
mkdir nginx
  • Confirm the directory exists by running:
ls

You should see app, compose.yaml, and nginx in the output.

  • Create a new file called nginx.conf inside the nginx/ directory and add the following configuration:
events {
    worker_connections 1024;
}

http {
    upstream app_servers {
        server app-1:3000;
        server app-2:3000;
        server app-3:3000;
    }

    server {
        listen 80;

        location / {
            proxy_pass http://app_servers;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            add_header X-Upstream-Addr $upstream_addr always;
        }
    }
}

What does this configuration do?

  • upstream app_servers defines a group of backend servers. Nginx will distribute requests across app-1, app-2, and app-3 on port 3000. These names match the service names in your compose.yaml.
  • Round-robin is the default load balancing method. No extra directive is needed. Nginx rotates through servers in order: request 1 goes to app-1, request 2 to app-2, request 3 to app-3, then back to app-1.
  • proxy_pass http://app_servers forwards every incoming request to the upstream group.
  • add_header X-Upstream-Addr $upstream_addr always attaches a response header showing which backend IP handled the request. This is your debugging tool for confirming load balancing works.

Why Nginx over Traefik?

Traefik is more Docker-native (it auto-discovers containers), but Nginx's explicit upstream configuration is the canonical example used in system design interviews.

When an interviewer asks you to design a load balancer layer, they expect the Nginx upstream pattern. Building it yourself means you can whiteboard it from memory.

  • Save nginx/nginx.conf.
  • Confirm the file was created by running:
ls nginx/

You should see nginx.conf in the output.

Don't see nginx.conf?

  • Make sure you created the file inside the nginx/ directory, not in the project root.
  • Check that your terminal is in the url-shortener project folder (the one containing compose.yaml).

Help me troubleshoot my nginx.conf file location.

Understand how the config reaches the container

You might wonder how this config file gets inside the Nginx container. The answer is already in your compose.yaml from the previous step.

  • Switch back to compose.yaml and find the nginx service definition near the top:
  nginx:
    image: nginx:1.30.2-alpine
    ports:
      - "80:80"
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - app-1
      - app-2
      - app-3

What does the volume mount do?

The volumes line mounts your local nginx/nginx.conf directly into the container at /etc/nginx/nginx.conf. The :ro suffix means read-only, so the container cannot modify your config.

This approach means you do not need a separate Nginx Dockerfile. Docker Compose pulls the official nginx:1.30.2-alpine image and injects your custom config at startup.

✔️ Awesome, I've got everything!

Great. Make sure you have saved nginx/nginx.conf.

ⓧ I'd like to double check the full code

Here is the complete nginx/nginx.conf file:

events {
    worker_connections 1024;
}

http {
    upstream app_servers {
        server app-1:3000;
        server app-2:3000;
        server app-3:3000;
    }

    server {
        listen 80;

        location / {
            proxy_pass http://app_servers;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            add_header X-Upstream-Addr $upstream_addr always;
        }
    }
}

Your load balancer is configured and ready to distribute traffic. Next up, you'll launch the entire stack and watch requests rotate across all three app servers in real time.

Launch the Stack and Observe Load Balancing

You have all the pieces of a distributed system ready to go. An Nginx load balancer, three identical app servers, a Redis cache, and a PostgreSQL database are all defined in your Compose file.

Now it's time to bring them all up at once and prove that load balancing actually works. You'll send requests through the front door (Nginx on port 80) and watch response headers rotate between your three app servers.

In this step, get ready to:
  • Build and start the entire Docker Compose stack.
  • Send POST requests and observe round-robin distribution across app servers.
  • Send GET requests for short codes and confirm redirects work with proper headers.
Build and start the Docker Compose stack

Docker Compose reads your compose.yaml and builds all the images, creates a shared network, and starts every container in the correct order (databases first, then app servers, then Nginx).

  • In your terminal, make sure you are inside the url-shortener project folder (where compose.yaml lives).
  • Build the images and start all services in detached mode by running this command:
docker compose up --build -d

What does this command do?

  • --build tells Docker to rebuild images from your Dockerfiles (so any code changes get picked up).
  • -d runs containers in detached mode, so they stay running in the background and you get your terminal back.

The first run takes a minute or two as Docker pulls base images and installs npm packages. Subsequent runs are much faster thanks to layer caching.

  • Verify all containers are running by running:
docker compose ps

You should see 6 containers listed, all with a status of Up: nginx, app-1, app-2, app-3, redis, and postgres.

Seeing containers exit or fail to start?

  • Check that Docker Desktop is running. If it's not, start it and try the command again.
  • If a container shows Exit status, check its logs by running docker compose logs <service-name> (replace with the failing service name).
  • If port 80 is already in use, you'll see a bind error. Stop whatever else is using port 80 and try again.

Help me troubleshoot why my Docker Compose containers won't start.

Send POST requests and observe round-robin distribution

Round-robin load balancing means Nginx sends each incoming request to the next server in the list. Your first request goes to app-1, the second to app-2, the third to app-3, then back to app-1.

You'll prove this is happening by reading the X-Served-By header your app sets on every response.

Windows PowerShell users

In Windows PowerShell, curl is an alias for a different command. Use curl.exe instead to run the real curl utility. Replace curl with curl.exe in all the commands below.

  • Send your first POST request to shorten a URL by running:
curl -i -X POST http://localhost/shorten -H "Content-Type: application/json" -d "{\"url\":\"https://example.com\"}"

What does this command do?

  • -i includes the response headers in the output so you can see X-Served-By.
  • -X POST sets the HTTP method to POST.
  • -d sends the JSON body containing the URL you want to shorten.

In the response headers, look for the X-Served-By line. It should show one of your instance names (e.g. app-1).

  • Run the same command two more times. Watch X-Served-By rotate from app-1 to app-2 to app-3.

Each response body also includes a code field (like a1b2c3d4). Copy one of these codes for the next substep.

Why does this prove load balancing?

Every request hits the same Nginx entry point on port 80. Nginx then forwards it to the next server in the upstream list. The fact that X-Served-By changes with each request proves that different containers are handling different requests.

This is the core of horizontal scaling. No single server handles all the traffic.

Seeing the same server every time?

  • Make sure you're hitting http://localhost/shorten (port 80, through Nginx) and not http://localhost:3000/shorten (directly to one container).
  • Check that all three app containers are running with docker compose ps. If one is down, Nginx will skip it.

Help me figure out why my requests aren't being distributed across servers.

Send GET requests and confirm redirects with headers

The other half of your URL shortener is the redirect. When someone visits a short URL, the app looks up the original URL and sends a 301 redirect. Nginx also adds the X-Upstream-Addr header showing which backend handled the request.

  • Send a GET request using one of the short codes you received from the POST responses (replace your-code with your actual code):
curl -i http://localhost/[[SHORT_CODE="your-code"]]

You should see a 301 Moved Permanently response with a Location header pointing to your original URL.

  • Look at the response headers for two key values:
  • X-Served-By shows which app container processed the request (set by your Node.js code).
  • X-Upstream-Addr shows the IP address of the upstream server that Nginx forwarded to (set by your Nginx config).
  • Run the same GET request two more times. Watch both headers change as Nginx distributes requests across all three app servers.

What is the X-Upstream-Addr header?

This header comes from the Nginx $upstream_addr variable you configured. It shows the raw IP and port of the container Nginx chose.

In production, you wouldn't expose internal IPs to clients. Here it's useful for learning and debugging.

Getting a 404 response?

  • Double check that you copied the code correctly from your POST response. The code is the random 8-character hex string in the JSON body.
  • Make sure the URL is http://localhost/your-code with no extra path or trailing slash.

Help me debug why my GET request returns 404.

Your distributed system is live and load balancing is confirmed. Next up, you'll add the caching layer and measure how much faster cached reads are compared to database lookups.

Implement and Measure Caching

In the previous step, you confirmed that requests are distributed across all three app servers. But every single request still queries PostgreSQL directly.

A URL shortener is a read-heavy system. The same short codes get looked up far more often than new ones get created. Caching solves this by storing frequently accessed data in memory so the database doesn't get hammered on every read. Your code already implements this pattern. In this step, you'll observe it in action and measure the performance difference.

In this step, get ready to:
  • Walk through the cache-aside pattern in your server code.
  • Measure cache hits vs misses using response headers.
  • Confirm the cache is shared across all app instances.
Understand the cache-aside pattern

The cache-aside pattern is a strategy where the application checks a fast in-memory cache before querying the slower database. Your GET /:code endpoint in app/server.js already implements this.

  • In your code editor, open app/server.js and find the app.get('/:code', ...) route handler.

Here is the section that implements the pattern:

app.get('/:code', async (req, res) => {
  const { code } = req.params;

  // Step 1: Check Redis cache first
  const cached = await redis.get(`url:${code}`);
  if (cached) {
    res.set('X-Cache', 'HIT');
    res.set('X-Served-By', INSTANCE_ID);
    return res.redirect(301, cached);
  }

  // Step 2: Cache miss - query PostgreSQL
  const result = await pool.query(
    'SELECT original_url FROM urls WHERE code = $1',
    [code]
  );

  if (result.rows.length === 0) {
    return res.status(404).json({ error: 'Short URL not found' });
  }

  // Step 3: Populate cache with a 1-hour TTL
  const originalUrl = result.rows[0].original_url;
  await redis.set(`url:${code}`, originalUrl, 'EX', 3600);

  res.set('X-Cache', 'MISS');
  res.set('X-Served-By', INSTANCE_ID);
  res.redirect(301, originalUrl);
});

What is the cache-aside pattern?

The pattern follows three steps on every read request:

  • Check the cache (Redis) for the key url:{code}. If found, return immediately. This is a cache HIT.
  • If not found (cache MISS), query the database (PostgreSQL) for the original URL.
  • Store the result in Redis with a TTL of 3600 seconds (1 hour) so future requests skip the database entirely.

The X-Cache response header tells you whether each request was served from cache or from the database. This is how you'll measure the pattern in action.

Test cache hits and misses

Now you'll send requests and watch the X-Cache header flip from MISS to HIT on repeated lookups of the same short code.

  • Create a new short URL by running this command:
curl -s -X POST http://localhost/shorten -H "Content-Type: application/json" -d "{\"url\": \"https://www.example.com\"}"

What does this command do?

This sends a POST request to create a new short URL mapping. The response JSON includes a code field. You'll use that code in the next requests.

  • Copy the code value from the JSON response (it looks like a1b2c3d4).
  • Send the first GET request using your code. This is the first lookup, so it must hit the database:
curl -i http://localhost/[[SHORT_CODE="your short code from the POST response"]]
  • Look at the response headers. You should see X-Cache: MISS because Redis didn't have this URL cached yet.
  • Send the same request again:
curl -i http://localhost/[[SHORT_CODE="your short code from the POST response"]]
  • This time you should see X-Cache: HIT. Redis returned the cached URL without touching PostgreSQL.

What should I see?

The first request shows X-Cache: MISS because the code wasn't in Redis yet. Every subsequent request for the same code shows X-Cache: HIT because the first request populated the cache.

Now measure the response time difference between a cache miss and a cache hit.

  • Create another short URL so you get a fresh code with no cache entry:
curl -s -X POST http://localhost/shorten -H "Content-Type: application/json" -d "{\"url\": \"https://www.wikipedia.org\"}"
  • Measure the first request (cache miss) by running:
curl -o /dev/null -s -w "Cache: %header{X-Cache}\nTime: %{time_total}s\n" http://localhost/[[SHORT_CODE2="your new short code"]]
  • Measure the second request (cache hit) by running the same command again:
curl -o /dev/null -s -w "Cache: %header{X-Cache}\nTime: %{time_total}s\n" http://localhost/[[SHORT_CODE2="your new short code"]]
  • Compare the two times. The cache HIT should be noticeably faster because it skips the database query entirely.

Why is the cache hit faster?

Redis stores data in memory. Reading from memory takes microseconds. PostgreSQL stores data on disk and must parse SQL, plan the query, and fetch rows. Even locally, you'll see the cache hit complete in a fraction of the time.

Seeing X-Cache: MISS on every request?

Make sure you're using the exact same short code for repeated requests. A different code means a different cache key.

Check that your Redis container is running by running docker compose ps and confirming the redis service shows as running.

help me debug why my Redis cache isn't returning HITs

Confirm the cache is shared across all instances

The power of this architecture is that Redis is a separate, centralized service. All three app instances read from and write to the same Redis. A URL cached by app-1 is immediately available to app-2 and app-3.

  • Send several requests to the same short code and observe which instance handles each one:
curl -s -o /dev/null -w "Served-By: %header{X-Served-By} | Cache: %header{X-Cache}\n" http://localhost/[[SHORT_CODE="your short code"]]
curl -s -o /dev/null -w "Served-By: %header{X-Served-By} | Cache: %header{X-Cache}\n" http://localhost/[[SHORT_CODE="your short code"]]
curl -s -o /dev/null -w "Served-By: %header{X-Served-By} | Cache: %header{X-Cache}\n" http://localhost/[[SHORT_CODE="your short code"]]

What should I see?

The X-Served-By header rotates between app-1, app-2, and app-3 (round-robin load balancing). But X-Cache shows HIT on every single request regardless of which instance handles it.

This proves the cache is centralized. One instance wrote the cache entry. All instances can read it. If you added app-4 and app-5 to your system, they would immediately benefit from the existing cached data without any duplication.

Why this architecture scales horizontally

Because the app servers are stateless (they store nothing locally), you can add or remove instances without affecting the cache. Redis holds the shared state. This is the core principle of horizontal scaling: scale compute independently from data.

Seeing mixed HITs and MISSes across instances?

If one instance shows MISS while others show HIT for the same code, the Redis connection may have failed for that container. Check the logs for that specific app instance:

docker compose logs app-1

Look for Redis connection errors. All three app services should connect to the same redis://redis:6379 URL defined in compose.yaml.

help me debug why one of my app instances isn't reading from the shared Redis cache

✔️ Awesome, I've got everything!

Great. Make sure your containers are still running by checking docker compose ps.

ⓧ I'd like to double check the full code

Here is the complete app/server.js for reference:

const express = require('express');
const { Pool } = require('pg');
const Redis = require('ioredis');
const crypto = require('crypto');

const app = express();
app.use(express.json());

const INSTANCE_ID = process.env.INSTANCE_ID || 'unknown';

const pool = new Pool({
  connectionString: process.env.DATABASE_URL,
});

const redis = new Redis(process.env.REDIS_URL);

async function initDB() {
  const client = await pool.connect();
  try {
    await client.query(`
      CREATE TABLE IF NOT EXISTS urls (
        id SERIAL PRIMARY KEY,
        code VARCHAR(8) UNIQUE NOT NULL,
        original_url TEXT NOT NULL,
        created_at TIMESTAMP DEFAULT NOW()
      )
    `);
  } finally {
    client.release();
  }
}

function generateCode() {
  return crypto.randomBytes(4).toString('hex');
}

app.post('/shorten', async (req, res) => {
  const { url } = req.body;
  if (!url) {
    return res.status(400).json({ error: 'url is required' });
  }

  const code = generateCode();
  try {
    await pool.query(
      'INSERT INTO urls (code, original_url) VALUES ($1, $2)',
      [code, url]
    );
    res.set('X-Served-By', INSTANCE_ID);
    res.json({ short_url: `http://localhost/${code}`, code });
  } catch (err) {
    res.status(500).json({ error: 'Failed to create short URL' });
  }
});

app.get('/:code', async (req, res) => {
  const { code } = req.params;

  const cached = await redis.get(`url:${code}`);
  if (cached) {
    res.set('X-Cache', 'HIT');
    res.set('X-Served-By', INSTANCE_ID);
    return res.redirect(301, cached);
  }

  const result = await pool.query(
    'SELECT original_url FROM urls WHERE code = $1',
    [code]
  );

  if (result.rows.length === 0) {
    return res.status(404).json({ error: 'Short URL not found' });
  }

  const originalUrl = result.rows[0].original_url;
  await redis.set(`url:${code}`, originalUrl, 'EX', 3600);

  res.set('X-Cache', 'MISS');
  res.set('X-Served-By', INSTANCE_ID);
  res.redirect(301, originalUrl);
});

app.get('/health', (req, res) => {
  res.set('X-Served-By', INSTANCE_ID);
  res.json({ status: 'ok', instance: INSTANCE_ID });
});

initDB().then(() => {
  app.listen(3000, () => {
    console.log(`${INSTANCE_ID} listening on port 3000`);
  });
});

You've now proven that your distributed system handles reads efficiently. The cache-aside pattern eliminates redundant database queries, and the centralized Redis instance ensures all app servers benefit from cached data. Next up, you'll take on a bonus challenge to protect your API from abuse.

Secret mission

Add a Redis-Based Rate Limiter

Your URL shortener has no protection against abuse. A single user could overwhelm your system with thousands of requests per second. In this mission, you'll implement a distributed rate limiter using Redis that tracks requests per IP and returns HTTP 429 when the limit is exceeded.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally using Docker containers, so there are no ongoing cloud costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Docker containers (nginx, app-1, app-2, app-3, redis, postgres).
  • Docker volume (pgdata) storing PostgreSQL data.
  • Docker images built and pulled for this project.
  • Local project directory (url-shortener) on your Desktop.

Keep everything running

No action needed. Choose this if you're still actively experimenting with the system, scaling replicas, or exploring the caching and rate limiting behavior.

  • Your containers continue running in the background and are accessible at http://localhost.
  • Docker images remain cached on your machine but take no active resources when containers are stopped.

Pause - I'll come back to this later

Shut down running containers to free up memory and CPU, but keep your project files and data volume intact so you can resume later.

  • Navigate to your project directory and stop the containers by running:
cd ~/Desktop/url-shortener
docker compose down

This stops and removes the containers and network, but preserves the pgdata volume with your stored URLs.

  • When you're ready to resume, run docker compose up --build -d from the same directory to start everything again.

Delete - I don't want to use this again

Remove all project resources and start fresh. This deletes containers, networks, volumes, and images associated with this project.

  • Navigate to your project directory and tear down the entire stack by running:
cd ~/Desktop/url-shortener
docker compose down --volumes --rmi all

What does this command do?

  • docker compose down stops and removes all containers and the network.
  • --volumes removes the pgdata volume containing your PostgreSQL data.
  • --rmi all removes all Docker images used by the services (nginx, node, redis, postgres).
  • Delete the local project directory by running:
cd ~/Desktop
rm -rf url-shortener
  • On Windows, use this command instead:
cd ~\Desktop
rmdir /s /q url-shortener
  • Confirm the folder is gone by running ls (macOS/Linux) or dir (Windows) in your Desktop directory.

Nice Work!

Nice Work!

Nice work! You just built a production-grade distributed system from scratch using Docker Compose, complete with load balancing, caching, and horizontal scaling.

You've learned how to:

  • Orchestrate a multi-service distributed system with Docker Compose, wiring together Nginx, Node.js, Redis, and PostgreSQL into a single stack.
  • Configure Nginx as a reverse proxy and load balancer with round-robin distribution, and verify request routing through response headers.
  • Implement the cache-aside pattern with Redis and measure cache hit rates, proving why caching matters for read-heavy systems at scale.
  • (Secret Mission) Build a distributed rate limiter using Redis sliding window counters to protect your API from abuse.

Ready to quiz yourself?