Build a Vercel Clone on AWS

Build a 3-service deployment pipeline from GitHub repo to live site using S3 and SQS.

Introduction

30 Second Summary

Every time you push code to GitHub and see it appear as a live website moments later, there is a complex pipeline working behind the scenes. Multiple services coordinate to clone your code, build it, and serve it to the world.

In this project, you will build a mini Vercel clone consisting of three microservices that work together as a deployment pipeline. You'll take a GitHub repo URL, process it through AWS services, and serve the result as a live site accessible via a custom subdomain.

What You'll Build

A deployment pipeline with three Node.js microservices that clone a GitHub repository, build it, and serve the output through subdomain-based routing using AWS primitives.

By the end of this project, you'll have:

  • An Upload Service that accepts a GitHub repo URL, clones it, and stores source files in S3.
  • A Deploy Service that polls SQS for build jobs, builds the project, and uploads artifacts back to S3.
  • A Request Handler that serves deployed sites via wildcard subdomain routing from S3.
  • Secret Mission: Add deployment status tracking with Redis so the system reports real-time build states.

Are there any prerequisites?

This project assumes strong Node.js and Express knowledge, familiarity with Docker, and Git installed on your machine. You should be comfortable creating APIs and working with npm packages.

Set Up AWS and Scaffold the Project

Before you can build a deployment pipeline, you need the infrastructure it runs on. AWS provides the storage and messaging primitives your three services will use to communicate.

In this step, you will create an AWS account, install the AWS CLI, and provision an S3 bucket and an SQS queue. Then you will scaffold a monorepo with three service directories and install their dependencies.

In this step, get ready to:
  • Create an AWS account and configure the CLI with access keys.
  • Provision an S3 bucket and an SQS queue.
  • Scaffold the monorepo with three service directories and install dependencies.
Create an AWS account and install the AWS CLI

The AWS CLI lets you manage cloud resources from your terminal instead of clicking through the console. You need an account, an IAM user with access keys, and the CLI installed locally.

  • Go to https://aws.amazon.com/free/ and create a free AWS account (or sign in if you already have one).
  • In the AWS Console, navigate to IAM and create a new user with programmatic access.
  • On the user's security credentials page, click Create access key.
  • Copy the Access Key ID and Secret Access Key somewhere safe. You will need them in the next step.

Why create an IAM user?

Your root account has unlimited permissions, which is risky for day-to-day work. An IAM user with scoped permissions limits the blast radius if credentials leak.

Now install the AWS CLI v2. Check whether it is already installed by running this command in your terminal:

aws --version

✔️ I see aws-cli/2.x

Your AWS CLI v2 is ready. Move on to configuring it below.

ⓧ I see aws-cli/1.x

You have version 1 installed, which is missing features this project needs. Uninstall version 1 first, then install version 2 using the instructions in the next tab.

Follow the official migration guide at docs.aws.amazon.com/cli/latest/userguide/cliv2-migration-instructions.html to remove v1, then install v2 using the steps below.

macOS

  • Download and run the official installer by running these commands:
curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
sudo installer -pkg ./AWSCLIV2.pkg -target /
  • Verify the installation by running aws --version. You should see a version starting with aws-cli/2.

Windows

  • Download and run the MSI installer from https://awscli.amazonaws.com/AWSCLIV2.msi.
  • Follow the on-screen prompts to complete the install.
  • Close and reopen your terminal, then run aws --version to confirm it shows aws-cli/2.

Linux

  • Install the CLI by running these commands:
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
  • Verify the installation by running aws --version. You should see a version starting with aws-cli/2.

ⓧ Command not found

The AWS CLI is not installed yet. Install it using the instructions for your operating system below.

macOS

  • Download and run the official installer by running these commands:
curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
sudo installer -pkg ./AWSCLIV2.pkg -target /
  • Verify the installation by running aws --version. You should see a version starting with aws-cli/2.

Windows

  • Download and run the MSI installer from https://awscli.amazonaws.com/AWSCLIV2.msi.
  • Follow the on-screen prompts to complete the install.
  • Close and reopen your terminal, then run aws --version to confirm it shows aws-cli/2.

Linux

  • Install the CLI by running these commands:
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
  • Verify the installation by running aws --version. You should see a version starting with aws-cli/2.

Now configure the CLI with your IAM user credentials.

  • Run the configure command:
aws configure

What does this command do?

The aws configure command prompts you for four values: Access Key ID, Secret Access Key, default region, and output format. It saves these to ~/.aws/credentials and ~/.aws/config so every future CLI call uses them automatically.

  • When prompted, paste your Access Key ID.
  • Paste your Secret Access Key.
  • Enter us-east-1 for the default region.
  • Press Enter to accept the default output format (json).

Seeing an invalid credentials error?

Double-check that you copied the full Access Key ID and Secret Access Key without trailing spaces. If your key was created for a different region, the CLI still works because IAM credentials are global.

Help me troubleshoot my AWS CLI configuration.

Create the S3 bucket and SQS queue

Your pipeline needs two AWS resources: an S3 bucket to store source and built files, and an SQS queue to pass build jobs between services.

  • Create the S3 bucket by running this command:
aws s3api create-bucket --bucket vercel-clone-outputs --region us-east-1

What does this command do?

This creates a new S3 bucket named vercel-clone-outputs in the us-east-1 region. S3 bucket names are globally unique. If this name is already taken, append your initials or a short random string (e.g. vercel-clone-outputs-abc).

  • Verify the bucket exists by running:
aws s3 ls
  • You should see vercel-clone-outputs in the list.
  • Create the SQS queue by running this command:
aws sqs create-queue --queue-name vercel-deploy-queue

What does this command do?

This creates a standard SQS queue named vercel-deploy-queue. The command returns a JSON object with a QueueUrl field. Copy this URL and save it somewhere. You will need it in the next step when configuring your services.

  • Copy the QueueUrl from the output. It looks like https://sqs.us-east-1.amazonaws.com/123456789012/vercel-deploy-queue.
  • Verify the queue exists by running:
aws sqs list-queues
  • You should see your queue URL in the output.

Queue creation failed?

Make sure your IAM user has SQS permissions. If you see an access denied error, attach the AmazonSQSFullAccess managed policy to your IAM user from the AWS Console.

Help me fix my SQS permissions.

Scaffold the monorepo and install dependencies

Your project is a monorepo with three independent services, each in its own directory with its own package.json.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create the project folder and all three service directories by running these commands:
mkdir vercel-clone
cd vercel-clone
mkdir upload-service deploy-service request-handler
  • Confirm the directories exist by running:
ls
  • You should see deploy-service, request-handler, and upload-service listed.

Now create the package.json for each service. Start with the upload service.

  • Create upload-service/package.json by opening the file in VS Code and pasting the following:
{
  "name": "upload-service",
  "version": "1.0.0",
  "type": "module",
  "scripts": {
    "start": "node index.js"
  },
  "dependencies": {
    "@aws-sdk/client-s3": "^3.1063.0",
    "@aws-sdk/client-sqs": "^3.1062.0",
    "cors": "^2.8.5",
    "express": "^4.19.2",
    "simple-git": "^3.36.0"
  }
}

What are these dependencies?

The @aws-sdk/client-s3 and @aws-sdk/client-sqs packages let you call S3 and SQS from Node.js. The simple-git package clones GitHub repos programmatically. The express and cors packages create the HTTP server.

Setting "type": "module" enables ES module syntax (import/export) across all files in the service.

  • Create deploy-service/package.json with the following content:
{
  "name": "deploy-service",
  "version": "1.0.0",
  "type": "module",
  "scripts": {
    "start": "node index.js"
  },
  "dependencies": {
    "@aws-sdk/client-s3": "^3.1063.0",
    "@aws-sdk/client-sqs": "^3.1062.0"
  }
}
  • Create request-handler/package.json with the following content:
{
  "name": "request-handler",
  "version": "1.0.0",
  "type": "module",
  "scripts": {
    "start": "node index.js"
  },
  "dependencies": {
    "@aws-sdk/client-s3": "^3.1063.0",
    "express": "^4.19.2",
    "mime-types": "^2.1.35"
  }
}

Now install the dependencies for each service.

  • Install the upload service dependencies by running:
cd upload-service && npm install && cd ..
  • Install the deploy service dependencies by running:
cd deploy-service && npm install && cd ..
  • Install the request handler dependencies by running:
cd request-handler && npm install && cd ..
  • Each command takes a few seconds. When they finish, each service directory contains a node_modules folder with its packages.

npm install failing?

Make sure you are inside the vercel-clone folder (not inside one of the service subdirectories). Run pwd to check your current location.

Help me fix my npm install error.

  • Open the project in VS Code by pressing Cmd+Space (macOS) or the Windows key (Windows), typing VS Code, and pressing Enter.
  • In VS Code, click File in the top menu bar.
  • Click Open Folder.
  • Select the vercel-clone folder on your Desktop, then click Open.

In the Explorer sidebar, you should see three folders: deploy-service, request-handler, and upload-service. Each contains a package.json and a node_modules folder.

✔️ Awesome, I've got everything!

Great. Make sure you have your SQS queue URL saved somewhere handy. You will paste it into your service code in the next step.

ⓧ I'd like to double check the full code

Here are the three package.json files for comparison.

{
  "name": "upload-service",
  "version": "1.0.0",
  "type": "module",
  "scripts": {
    "start": "node index.js"
  },
  "dependencies": {
    "@aws-sdk/client-s3": "^3.1063.0",
    "@aws-sdk/client-sqs": "^3.1062.0",
    "cors": "^2.8.5",
    "express": "^4.19.2",
    "simple-git": "^3.36.0"
  }
}
{
  "name": "deploy-service",
  "version": "1.0.0",
  "type": "module",
  "scripts": {
    "start": "node index.js"
  },
  "dependencies": {
    "@aws-sdk/client-s3": "^3.1063.0",
    "@aws-sdk/client-sqs": "^3.1062.0"
  }
}
{
  "name": "request-handler",
  "version": "1.0.0",
  "type": "module",
  "scripts": {
    "start": "node index.js"
  },
  "dependencies": {
    "@aws-sdk/client-s3": "^3.1063.0",
    "express": "^4.19.2",
    "mime-types": "^2.1.35"
  }
}

Your environment is ready. Next up, you will build the Upload Service that accepts a GitHub URL, clones the repo, and pushes the files to S3.

Build the Upload Service

Your AWS infrastructure is ready. You have an S3 bucket waiting to store files and an SQS queue ready to pass messages between services.

Now you need a service that accepts a GitHub URL, pulls down the source code, and feeds it into your pipeline. This is the Upload Service. It clones a repository, pushes every file to S3, then drops a message into SQS so the Deploy Service knows there is work to do.

In this step, get ready to:
  • Create an Express server with a POST /deploy endpoint that clones a GitHub repo.
  • Upload all cloned source files to S3 under a unique ID prefix.
  • Send a deployment message to SQS and verify the full pipeline handoff.
Set up the Express server and helper utilities

The Upload Service needs to accept HTTP requests, interact with AWS, and work with the file system. You will create the main server file with all the setup code and two helper functions that the endpoint will use.

  • Create a new file called index.js inside your upload-service/ directory and add the following setup code:
import express from "express";
import cors from "cors";
import { simpleGit } from "simple-git";
import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
import { SQSClient, SendMessageCommand } from "@aws-sdk/client-sqs";
import { randomBytes } from "crypto";
import path from "path";
import fs from "fs";

// Set up Express with CORS and JSON body parsing
const app = express();
app.use(cors());
app.use(express.json());

// Initialize AWS SDK clients for S3 and SQS
const s3Client = new S3Client({ region: "us-east-1" });
const sqsClient = new SQSClient({ region: "us-east-1" });

// Constants for your AWS resources
const S3_BUCKET = "vercel-clone-outputs";
const SQS_QUEUE_URL = "your-sqs-queue-url-here";

// Generate a random 12-character hex string as a deployment ID
function generateId() {
  return randomBytes(6).toString("hex");
}

// Recursively collect all file paths in a directory, skipping node_modules and .git
async function getAllFiles(dirPath) {
  const files = [];
  const entries = fs.readdirSync(dirPath, { withFileTypes: true });
  for (const entry of entries) {
    const fullPath = path.join(dirPath, entry.name);
    if (entry.isDirectory()) {
      if (entry.name === "node_modules" || entry.name === ".git") continue;
      files.push(...(await getAllFiles(fullPath)));
    } else {
      files.push(fullPath);
    }
  }
  return files;
}
  • Replace the value of SQS_QUEUE_URL with the actual queue URL you noted in Step 1.

What does this code do?

The imports bring in Express for the HTTP server, simple-git for cloning repos, and the AWS SDK v3 clients for S3 and SQS. The crypto module provides randomBytes to generate unique deployment IDs.

The getAllFiles helper walks a directory tree recursively and returns every file path. It skips node_modules and .git folders because those are large, irrelevant to deployment, and would waste S3 storage.

Why event-driven architecture instead of a monolith?

You could build a single service that clones, builds, and serves all in one process. The problem is that builds are slow and unpredictable. If a build takes 60 seconds, the HTTP request hangs for 60 seconds.

By splitting into separate services connected by a message queue, the Upload Service responds instantly while the Deploy Service handles the heavy lifting in the background. This is the same pattern Vercel uses at scale.

  • Save upload-service/index.js.

Seeing a red underline on imports?

Make sure your upload-service/package.json has "type": "module" set. Without this, Node.js treats the file as CommonJS and does not understand import statements.

Also confirm you ran npm install inside the upload-service/ directory in Step 1. The imports rely on packages listed in package.json.

Help me fix import errors in my upload service

Add the POST /deploy endpoint with S3 upload

This is the core of the Upload Service. When a request arrives with a GitHub URL, the endpoint clones the repo, uploads every file to S3 under a unique prefix, notifies SQS, and returns the deployment ID to the caller.

  • Below the getAllFiles function in upload-service/index.js, add the POST endpoint and server listener:
app.post("/deploy", async (req, res) => {
  const { repoUrl } = req.body;
  if (!repoUrl) {
    return res.status(400).json({ error: "repoUrl is required" });
  }

  const id = generateId();
  const outputDir = path.join("/tmp", `source-${id}`);

  try {
    // Clone the GitHub repository to a temp directory
    const git = simpleGit();
    await git.clone(repoUrl, outputDir);

    // Upload every file to S3 under the source/{id}/ prefix
    const files = await getAllFiles(outputDir);
    for (const file of files) {
      const relativePath = path.relative(outputDir, file);
      const fileContent = fs.readFileSync(file);

      await s3Client.send(
        new PutObjectCommand({
          Bucket: S3_BUCKET,
          Key: `source/${id}/${relativePath}`,
          Body: fileContent,
        })
      );
    }

    // Notify the Deploy Service by sending the deployment ID to SQS
    await sqsClient.send(
      new SendMessageCommand({
        QueueUrl: SQS_QUEUE_URL,
        MessageBody: JSON.stringify({ id }),
      })
    );

    // Clean up the cloned repo from disk
    fs.rmSync(outputDir, { recursive: true, force: true });

    res.json({ id, status: "queued" });
  } catch (error) {
    console.error("Deploy error:", error);
    res.status(500).json({ error: "Failed to process repository" });
  }
});

app.listen(3000, () => {
  console.log("Upload service running on port 3000");
});

What does this endpoint do?

  • Validates that a repoUrl was provided in the JSON body. Returns a 400 error if missing.
  • Generates a unique 12-character hex ID using randomBytes. This ID becomes the deployment's identity across all three services.
  • Clones the repository into a temporary directory using simple-git.
  • Uploads every file to S3 under the key pattern source/{id}/{relativePath}. This preserves the folder structure.
  • Sends a JSON message containing the deployment ID to the SQS queue. The Deploy Service will pick this up later.
  • Cleans up the cloned files from the temp directory to free disk space.
  • Save upload-service/index.js.

Getting credential errors when saving?

If you see an error about missing AWS credentials when you later run the service, double-check that aws configure completed successfully in Step 1. The SDK reads credentials from ~/.aws/credentials automatically.

Help me fix AWS credential issues in my upload service

✔️ Awesome, I've got everything!

Great. Double check you have saved upload-service/index.js before moving on to testing.

ⓧ I'd like to double check the full code

Here is the complete upload-service/index.js file:

import express from "express";
import cors from "cors";
import { simpleGit } from "simple-git";
import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
import { SQSClient, SendMessageCommand } from "@aws-sdk/client-sqs";
import { randomBytes } from "crypto";
import path from "path";
import fs from "fs";

const app = express();
app.use(cors());
app.use(express.json());

const s3Client = new S3Client({ region: "us-east-1" });
const sqsClient = new SQSClient({ region: "us-east-1" });

const S3_BUCKET = "vercel-clone-outputs";
const SQS_QUEUE_URL = "your-sqs-queue-url-here";

function generateId() {
  return randomBytes(6).toString("hex");
}

async function getAllFiles(dirPath) {
  const files = [];
  const entries = fs.readdirSync(dirPath, { withFileTypes: true });
  for (const entry of entries) {
    const fullPath = path.join(dirPath, entry.name);
    if (entry.isDirectory()) {
      if (entry.name === "node_modules" || entry.name === ".git") continue;
      files.push(...(await getAllFiles(fullPath)));
    } else {
      files.push(fullPath);
    }
  }
  return files;
}

app.post("/deploy", async (req, res) => {
  const { repoUrl } = req.body;
  if (!repoUrl) {
    return res.status(400).json({ error: "repoUrl is required" });
  }

  const id = generateId();
  const outputDir = path.join("/tmp", `source-${id}`);

  try {
    const git = simpleGit();
    await git.clone(repoUrl, outputDir);

    const files = await getAllFiles(outputDir);
    for (const file of files) {
      const relativePath = path.relative(outputDir, file);
      const fileContent = fs.readFileSync(file);

      await s3Client.send(
        new PutObjectCommand({
          Bucket: S3_BUCKET,
          Key: `source/${id}/${relativePath}`,
          Body: fileContent,
        })
      );
    }

    await sqsClient.send(
      new SendMessageCommand({
        QueueUrl: SQS_QUEUE_URL,
        MessageBody: JSON.stringify({ id }),
      })
    );

    fs.rmSync(outputDir, { recursive: true, force: true });

    res.json({ id, status: "queued" });
  } catch (error) {
    console.error("Deploy error:", error);
    res.status(500).json({ error: "Failed to process repository" });
  }
});

app.listen(3000, () => {
  console.log("Upload service running on port 3000");
});
Test the deployment pipeline handoff

With the code in place, it is time to start the server and trigger a deployment. You will send a real GitHub repository URL to the endpoint and verify that files land in S3 and a message appears in SQS.

  • Start the Upload Service by running this command from the upload-service/ directory:
npm start
  • You should see Upload service running on port 3000 in your terminal.
  • Open a second terminal. Send a test deployment request by running:
curl -X POST http://localhost:3000/deploy -H "Content-Type: application/json" -d '{"repoUrl": "https://github.com/vercel/next-learn"}'

What should I see?

The curl command should return a JSON response like {"id":"a1b2c3d4e5f6","status":"queued"}. The ID will be different each time since it is randomly generated. This confirms the endpoint accepted the request and queued the deployment.

The clone and upload may take 10-30 seconds depending on the repository size and your internet speed.

  • Verify that files were uploaded to S3 by running this command (replace the ID with the one from your response):
aws s3 ls s3://vercel-clone-outputs/source/ --recursive | head -20
  • You should see a list of file paths starting with source/{your-id}/. This confirms the repository was cloned and uploaded.
  • Verify that the SQS message was sent by running:
aws sqs get-queue-attributes --queue-url [[SQS_URL="your SQS queue URL"]] --attribute-names ApproximateNumberOfMessages
  • You should see "ApproximateNumberOfMessages": "1" in the output. This confirms the Upload Service successfully notified the queue.

Getting a connection refused error from curl?

Make sure the Upload Service is still running in the first terminal. If it crashed, check the error output for clues. Common issues include a typo in the SQS queue URL or missing AWS credentials.

If the clone step fails, confirm the GitHub URL is publicly accessible. Private repositories require authentication that this service does not handle.

Help me debug my upload service test failure

Your Upload Service is live and successfully handing off work to the queue. Next up, you will build the Deploy Service that picks up these messages, builds the project, and uploads the finished artifacts back to S3.

Build the Deploy Service

Your Upload Service is live and queuing deployment jobs. Source files land in S3, and a message appears in your SQS queue. But nothing happens next because no worker is listening.

The Deploy Service is the worker that picks up those queued messages, downloads the source code, builds the project, and pushes the finished artifacts back to S3. This is the core of your pipeline.

In this step, get ready to:
  • Create a polling worker that listens for SQS messages.
  • Download source files from S3 and run the project build.
  • Upload the built output back to S3 with correct content types.
Create the SQS polling worker

The Deploy Service does not use an HTTP server. Instead, it continuously polls the SQS queue for new messages using long polling. When a message arrives, the worker processes it and deletes it from the queue.

  • Switch to your terminal and navigate into the deploy-service directory by running:
cd deploy-service
  • Install the mime-types package (used later for setting content-type headers) alongside your existing dependencies by running:
npm install mime-types
  • Create a new file called index.js inside the deploy-service folder.
  • Add the imports and client configuration at the top of the file by pasting this code:
import {
  S3Client,
  ListObjectsV2Command,
  GetObjectCommand,
  PutObjectCommand,
} from "@aws-sdk/client-s3";
import {
  SQSClient,
  ReceiveMessageCommand,
  DeleteMessageCommand,
} from "@aws-sdk/client-sqs";
import { execSync } from "child_process";
import path from "path";
import fs from "fs";
import { Readable } from "stream";
import { lookup } from "mime-types";

const s3Client = new S3Client({ region: "us-east-1" });
const sqsClient = new SQSClient({ region: "us-east-1" });

const S3_BUCKET = "vercel-clone-outputs";
const SQS_QUEUE_URL = "your-sqs-queue-url-here";

What does this code do?

This sets up the AWS SDK clients for both S3 and SQS. The S3 imports cover listing objects, downloading files, and uploading files. The SQS imports handle receiving and deleting messages from the queue.

The execSync import from child_process lets you run shell commands (like npm install) from within Node.js. The lookup function from mime-types determines the correct content-type header for each file extension.

  • Now add the polling loop at the very bottom of the file. This is the main function that drives the worker:
async function pollQueue() {
  console.log("Deploy service polling for messages...");

  while (true) {
    try {
      const response = await sqsClient.send(
        new ReceiveMessageCommand({
          QueueUrl: SQS_QUEUE_URL,
          MaxNumberOfMessages: 1,
          WaitTimeSeconds: 10,
        })
      );

      if (response.Messages && response.Messages.length > 0) {
        for (const message of response.Messages) {
          await processMessage(message);

          await sqsClient.send(
            new DeleteMessageCommand({
              QueueUrl: SQS_QUEUE_URL,
              ReceiptHandle: message.ReceiptHandle,
            })
          );
        }
      }
    } catch (error) {
      console.error("Error polling queue:", error);
    }
  }
}

pollQueue();

What does this code do?

The pollQueue function runs an infinite loop. Each iteration sends a ReceiveMessageCommand with WaitTimeSeconds: 10. This is called long polling. Instead of hammering the API every millisecond, the request waits up to 10 seconds for a message to arrive before returning empty.

When a message is received, the worker calls processMessage (you will write this next), then deletes the message from the queue with DeleteMessageCommand. Deleting the message prevents it from being processed again.

Why long polling instead of short polling?

Short polling returns immediately even when the queue is empty, burning through API calls. Long polling reduces empty responses and keeps your usage well within the SQS free tier of 1 million requests per month.

Download source files from S3 and run the build

The worker needs to pull source code from S3, write it to disk, then run the project's build command. You will add three functions between your config constants and the pollQueue function.

  • Add the streamToBuffer helper and the downloadSourceFromS3 function directly below your SQS_QUEUE_URL constant:
async function streamToBuffer(stream) {
  const chunks = [];
  for await (const chunk of stream) {
    chunks.push(chunk);
  }
  return Buffer.concat(chunks);
}

async function downloadSourceFromS3(id, outputDir) {
  const prefix = `source/${id}/`;
  const listResponse = await s3Client.send(
    new ListObjectsV2Command({ Bucket: S3_BUCKET, Prefix: prefix })
  );

  if (!listResponse.Contents) return;

  for (const object of listResponse.Contents) {
    const relativePath = object.Key.replace(prefix, "");
    const localPath = path.join(outputDir, relativePath);

    fs.mkdirSync(path.dirname(localPath), { recursive: true });

    const getResponse = await s3Client.send(
      new GetObjectCommand({ Bucket: S3_BUCKET, Key: object.Key })
    );

    const body = await streamToBuffer(getResponse.Body);
    fs.writeFileSync(localPath, body);
  }
}

What does this code do?

The streamToBuffer helper converts an S3 response stream into a Buffer you can write to disk. The AWS SDK returns file bodies as readable streams, not raw buffers.

The downloadSourceFromS3 function lists all objects under the source/{id}/ prefix using ListObjectsV2Command. For each file, it recreates the directory structure locally and writes the file content to /tmp/build-{id}/.

  • Now add the processMessage function below downloadSourceFromS3. This function orchestrates the full build lifecycle:
async function processMessage(message) {
  const { id } = JSON.parse(message.Body);
  console.log(`Processing deployment: ${id}`);

  const workDir = path.join("/tmp", `build-${id}`);
  fs.mkdirSync(workDir, { recursive: true });

  try {
    await downloadSourceFromS3(id, workDir);

    console.log(`Running build for ${id}...`);
    execSync("npm install && npm run build", {
      cwd: workDir,
      stdio: "inherit",
    });

    const distDir = fs.existsSync(path.join(workDir, "dist"))
      ? path.join(workDir, "dist")
      : path.join(workDir, "build");

    await uploadBuildToS3(id, distDir);
    console.log(`Deployment ${id} completed successfully`);
  } finally {
    fs.rmSync(workDir, { recursive: true, force: true });
  }
}

What does this code do?

The processMessage function extracts the deployment ID from the SQS message body, downloads the source from S3, then runs npm install && npm run build using execSync. The cwd option sets the working directory to the downloaded source.

After the build completes, it checks for either a dist/ folder (Vite projects) or a build/ folder (Create React App projects). The finally block ensures the temp directory is always cleaned up, even if the build fails.

Upload built files to S3

The last piece is uploading the compiled output back to S3 so the Request Handler can serve it. Each file needs the correct content-type header so browsers render CSS, JavaScript, and HTML properly.

  • Add the getAllFiles utility and uploadBuildToS3 function between downloadSourceFromS3 and processMessage:
async function uploadBuildToS3(id, buildDir) {
  const files = getAllFiles(buildDir);
  for (const file of files) {
    const relativePath = path.relative(buildDir, file);
    const fileContent = fs.readFileSync(file);
    const contentType = lookup(relativePath) || "application/octet-stream";

    await s3Client.send(
      new PutObjectCommand({
        Bucket: S3_BUCKET,
        Key: `dist/${id}/${relativePath}`,
        Body: fileContent,
        ContentType: contentType,
      })
    );
  }
}

function getAllFiles(dirPath) {
  const files = [];
  const entries = fs.readdirSync(dirPath, { withFileTypes: true });
  for (const entry of entries) {
    const fullPath = path.join(dirPath, entry.name);
    if (entry.isDirectory()) {
      files.push(...getAllFiles(fullPath));
    } else {
      files.push(fullPath);
    }
  }
  return files;
}

What does this code do?

The uploadBuildToS3 function iterates over every file in the build output folder and uploads it to S3 under the dist/{id}/ prefix. The lookup function from mime-types determines the correct ContentType based on the file extension. Without this, browsers would not know how to interpret CSS or JavaScript files.

The getAllFiles helper recursively collects every file path in a directory tree. Unlike the Upload Service version, this one does not skip node_modules because the build output should only contain production-ready files.

  • Save deploy-service/index.js.
  • Update the SQS_QUEUE_URL constant to use your actual queue URL (the same one you used in the Upload Service).

Not sure where your queue URL is?

Run aws sqs list-queues in a terminal to see all your queue URLs. Copy the one you created earlier and paste it into the SQS_QUEUE_URL constant.

Help me find my SQS queue URL

  • Install the project dependencies by running:
npm install
  • Start the Deploy Service by running:
npm start
  • You should see Deploy service polling for messages... printed to the terminal.

Seeing a module not found error?

Make sure you ran npm install mime-types earlier in this step. Also confirm your package.json has "type": "module" since this project uses ES module imports.

Help me debug my Deploy Service startup error

Now test the full flow. Open a second terminal and trigger a deployment through the Upload Service.

  • In a new terminal window, send a test deployment request by running:
curl -X POST http://localhost:3000/deploy \
  -H "Content-Type: application/json" \
  -d '{"repoUrl": "https://github.com/vercel/next-learn"}'
  • Switch back to the terminal running the Deploy Service. You should see log output showing the deployment ID being processed, followed by npm install and build output.
  • After the build completes, verify the output landed in S3 by running:
aws s3 ls s3://vercel-clone-outputs/dist/ --recursive | head -20
  • You should see files like index.html and asset files under a dist/{id}/ prefix.

Build failing or no output in S3?

Make sure the GitHub repo you are deploying has a build script in its package.json. Repos without a build step will fail at npm run build. Try a simple Vite or Create React App repository.

If you see S3 permission errors, confirm your AWS CLI credentials have s3:PutObject and s3:GetObject permissions on your bucket.

Help me debug why my deploy service build is failing

✔️ Awesome, I've got everything!

Great. Double check you have saved deploy-service/index.js and that the service is polling without errors.

ⓧ I'd like to double check the full code

Here is the complete deploy-service/index.js for comparison:

import {
  S3Client,
  ListObjectsV2Command,
  GetObjectCommand,
  PutObjectCommand,
} from "@aws-sdk/client-s3";
import {
  SQSClient,
  ReceiveMessageCommand,
  DeleteMessageCommand,
} from "@aws-sdk/client-sqs";
import { execSync } from "child_process";
import path from "path";
import fs from "fs";
import { Readable } from "stream";
import { lookup } from "mime-types";

const s3Client = new S3Client({ region: "us-east-1" });
const sqsClient = new SQSClient({ region: "us-east-1" });

const S3_BUCKET = "vercel-clone-outputs";
const SQS_QUEUE_URL = "your-sqs-queue-url-here";

async function streamToBuffer(stream) {
  const chunks = [];
  for await (const chunk of stream) {
    chunks.push(chunk);
  }
  return Buffer.concat(chunks);
}

async function downloadSourceFromS3(id, outputDir) {
  const prefix = `source/${id}/`;
  const listResponse = await s3Client.send(
    new ListObjectsV2Command({ Bucket: S3_BUCKET, Prefix: prefix })
  );

  if (!listResponse.Contents) return;

  for (const object of listResponse.Contents) {
    const relativePath = object.Key.replace(prefix, "");
    const localPath = path.join(outputDir, relativePath);

    fs.mkdirSync(path.dirname(localPath), { recursive: true });

    const getResponse = await s3Client.send(
      new GetObjectCommand({ Bucket: S3_BUCKET, Key: object.Key })
    );

    const body = await streamToBuffer(getResponse.Body);
    fs.writeFileSync(localPath, body);
  }
}

async function uploadBuildToS3(id, buildDir) {
  const files = getAllFiles(buildDir);
  for (const file of files) {
    const relativePath = path.relative(buildDir, file);
    const fileContent = fs.readFileSync(file);
    const contentType = lookup(relativePath) || "application/octet-stream";

    await s3Client.send(
      new PutObjectCommand({
        Bucket: S3_BUCKET,
        Key: `dist/${id}/${relativePath}`,
        Body: fileContent,
        ContentType: contentType,
      })
    );
  }
}

function getAllFiles(dirPath) {
  const files = [];
  const entries = fs.readdirSync(dirPath, { withFileTypes: true });
  for (const entry of entries) {
    const fullPath = path.join(dirPath, entry.name);
    if (entry.isDirectory()) {
      files.push(...getAllFiles(fullPath));
    } else {
      files.push(fullPath);
    }
  }
  return files;
}

async function processMessage(message) {
  const { id } = JSON.parse(message.Body);
  console.log(`Processing deployment: ${id}`);

  const workDir = path.join("/tmp", `build-${id}`);
  fs.mkdirSync(workDir, { recursive: true });

  try {
    await downloadSourceFromS3(id, workDir);

    console.log(`Running build for ${id}...`);
    execSync("npm install && npm run build", {
      cwd: workDir,
      stdio: "inherit",
    });

    const distDir = fs.existsSync(path.join(workDir, "dist"))
      ? path.join(workDir, "dist")
      : path.join(workDir, "build");

    await uploadBuildToS3(id, distDir);
    console.log(`Deployment ${id} completed successfully`);
  } finally {
    fs.rmSync(workDir, { recursive: true, force: true });
  }
}

async function pollQueue() {
  console.log("Deploy service polling for messages...");

  while (true) {
    try {
      const response = await sqsClient.send(
        new ReceiveMessageCommand({
          QueueUrl: SQS_QUEUE_URL,
          MaxNumberOfMessages: 1,
          WaitTimeSeconds: 10,
        })
      );

      if (response.Messages && response.Messages.length > 0) {
        for (const message of response.Messages) {
          await processMessage(message);

          await sqsClient.send(
            new DeleteMessageCommand({
              QueueUrl: SQS_QUEUE_URL,
              ReceiptHandle: message.ReceiptHandle,
            })
          );
        }
      }
    } catch (error) {
      console.error("Error polling queue:", error);
    }
  }
}

pollQueue();

Your Deploy Service is processing builds and storing artifacts in S3. Next up, you will build the Request Handler that serves these deployed sites through subdomain-based routing.

Build the Request Handler

Your deploy service now processes build jobs and uploads finished artifacts to S3. The final piece of the puzzle is serving those artifacts to a browser.

The request handler is a reverse proxy that intercepts incoming HTTP requests, identifies which deployment the user wants by reading the subdomain, and fetches the matching files from S3. When someone visits http://abc123.localhost:8000, the handler extracts abc123 and serves that deployment's built files directly from your bucket.

In this step, get ready to:
  • Create an Express server that extracts the subdomain from incoming requests.
  • Fetch deployed files from S3 using the subdomain as a deployment ID.
  • Test the full pipeline from GitHub repo to live site.
Create the Express server with subdomain extraction

The request handler needs to turn a hostname like abc123.localhost into a deployment ID. Express gives you req.hostname which returns the full hostname without the port. Splitting on the dot gives you the subdomain as the first element.

  • Switch to the request-handler/ directory and install its dependencies by running these commands:
cd request-handler
npm install
  • Confirm the packages installed by running:
ls node_modules

You should see folders for @aws-sdk, express, and mime-types listed in the output.

Seeing install errors?

Make sure you are inside the request-handler/ directory, not the project root. The package.json must be in the current folder for npm install to find it.

Help me troubleshoot npm install errors

  • Create the index.js file inside request-handler/ and add the server setup and utility function:
import express from "express";
import { S3Client, GetObjectCommand } from "@aws-sdk/client-s3";
import { lookup } from "mime-types";

const app = express();
const s3Client = new S3Client({ region: "us-east-1" });

const S3_BUCKET = "vercel-clone-outputs";

// Converts S3 response stream into a Buffer for Express to send
async function streamToBuffer(stream) {
  const chunks = [];
  for await (const chunk of stream) {
    chunks.push(chunk);
  }
  return Buffer.concat(chunks);
}

What does this code do?

  • Imports express for the HTTP server, GetObjectCommand to read files from S3, and lookup from mime-types to determine correct Content-Type headers.
  • Creates an S3 client pointed at us-east-1 and references the same bucket your other services use.
  • The streamToBuffer helper converts the S3 response stream into a Buffer so Express can send it to the browser.
  • Save index.js.
Fetch files from S3 using subdomain routing

Now you need the route handler that ties subdomain extraction to S3 fetching. When a request arrives at abc123.localhost:8000/assets/style.css, the handler builds the S3 key dist/abc123/assets/style.css and streams the file back with the correct MIME type.

  • Add the route handler and server listen call below your existing code in index.js:
app.get("/*", async (req, res) => {
  // Extract subdomain from hostname (e.g. abc123 from abc123.localhost)
  const hostname = req.hostname;
  const subdomain = hostname.split(".")[0];

  // Guard against bare localhost requests with no subdomain
  if (subdomain === "localhost" || !subdomain) {
    return res.status(400).send("Please use a subdomain: http://{id}.localhost:8000");
  }

  // Default root path to index.html, build the S3 key
  let filePath = req.path === "/" ? "/index.html" : req.path;
  const key = `dist/${subdomain}${filePath}`;

  try {
    const response = await s3Client.send(
      new GetObjectCommand({ Bucket: S3_BUCKET, Key: key })
    );

    const body = await streamToBuffer(response.Body);
    const contentType = lookup(filePath) || "application/octet-stream";

    res.set("Content-Type", contentType);
    res.send(body);
  } catch (error) {
    // SPA fallback: serve index.html for unknown paths
    if (error.name === "NoSuchKey") {
      const indexKey = `dist/${subdomain}/index.html`;
      try {
        const fallback = await s3Client.send(
          new GetObjectCommand({ Bucket: S3_BUCKET, Key: indexKey })
        );
        const body = await streamToBuffer(fallback.Body);
        res.set("Content-Type", "text/html");
        res.send(body);
      } catch {
        res.status(404).send("Deployment not found");
      }
    } else {
      res.status(500).send("Internal server error");
    }
  }
});

app.listen(8000, () => {
  console.log("Request handler running on port 8000");
});

What does this route do?

  • Extracts the subdomain from req.hostname by splitting on the dot and taking the first segment. For abc123.localhost, this gives abc123.
  • Guards against bare localhost requests with no subdomain.
  • Defaults the root path to /index.html and constructs the S3 key as dist/{subdomain}/{path}.
  • Uses lookup from mime-types to set the correct Content-Type header so CSS and JS files load properly in the browser.
  • Falls back to serving index.html when a file is not found. This supports single-page applications that use client-side routing.

Why the SPA fallback?

React apps use client-side routing. When a user navigates to a path like /about, there is no about.html file in the build output. The app needs index.html to load, and then the JavaScript router handles the path.

The NoSuchKey catch block serves index.html as a fallback for any path that does not match a real file in S3. This is the same pattern Vercel and Netlify use.

  • Save index.js.
  • Start the request handler by running:
node index.js

You should see Request handler running on port 8000 in the terminal.

Port 8000 already in use?

If another process is using port 8000, stop it first. On macOS or Linux, run lsof -i :8000 to find the process, then kill its PID.

Help me fix the port conflict

✔️ Awesome, I've got everything!

Great. Double check you have saved request-handler/index.js and the server is running on port 8000.

ⓧ I'd like to double check the full code

import express from "express";
import { S3Client, GetObjectCommand } from "@aws-sdk/client-s3";
import { lookup } from "mime-types";

const app = express();
const s3Client = new S3Client({ region: "us-east-1" });

const S3_BUCKET = "vercel-clone-outputs";

async function streamToBuffer(stream) {
  const chunks = [];
  for await (const chunk of stream) {
    chunks.push(chunk);
  }
  return Buffer.concat(chunks);
}

app.get("/*", async (req, res) => {
  const hostname = req.hostname;
  const subdomain = hostname.split(".")[0];

  if (subdomain === "localhost" || !subdomain) {
    return res.status(400).send("Please use a subdomain: http://{id}.localhost:8000");
  }

  let filePath = req.path === "/" ? "/index.html" : req.path;
  const key = `dist/${subdomain}${filePath}`;

  try {
    const response = await s3Client.send(
      new GetObjectCommand({ Bucket: S3_BUCKET, Key: key })
    );

    const body = await streamToBuffer(response.Body);
    const contentType = lookup(filePath) || "application/octet-stream";

    res.set("Content-Type", contentType);
    res.send(body);
  } catch (error) {
    if (error.name === "NoSuchKey") {
      const indexKey = `dist/${subdomain}/index.html`;
      try {
        const fallback = await s3Client.send(
          new GetObjectCommand({ Bucket: S3_BUCKET, Key: indexKey })
        );
        const body = await streamToBuffer(fallback.Body);
        res.set("Content-Type", "text/html");
        res.send(body);
      } catch {
        res.status(404).send("Deployment not found");
      }
    } else {
      res.status(500).send("Internal server error");
    }
  }
});

app.listen(8000, () => {
  console.log("Request handler running on port 8000");
});

How does subdomain routing work on localhost?

Modern browsers (Chrome, Firefox) automatically resolve any subdomain of localhost to 127.0.0.1. This means abc123.localhost hits your local machine without any /etc/hosts editing.

Express sees the full hostname including the subdomain via req.hostname. Your handler splits it and uses the first segment as the deployment ID.

Test the full deployment pipeline

All three services are now complete. Time to run the entire pipeline end-to-end and see a deployed site in your browser.

  • Make sure all three services are running in separate terminal windows: the upload service on port 3000, the deploy service polling SQS, and the request handler on port 8000.
  • Trigger a deployment by sending a POST request with a public React repository URL:
curl -X POST http://localhost:3000/deploy \
  -H "Content-Type: application/json" \
  -d '{"repoUrl": "[[REPO_URL=\"https://github.com/your-username/your-react-repo\"]]"}'

Use any public React repo

You can use any public GitHub repository that has a package.json with a build script. A simple Vite React template or Create React App project works well.

  • Note the id value returned in the response. It will look something like a1b2c3d4e5f6.
  • Watch the deploy service terminal. You should see it pick up the message, run the build, and upload the dist files.
  • Once the deploy service logs that the deployment completed, open your browser and visit:
http://[[DEPLOY_ID="your-deployment-id"]].localhost:8000

Replace the deployment ID placeholder with the actual ID from your curl response. You should see the deployed React app running in your browser.

Seeing 'Deployment not found'?

  • Check that the deploy service finished building. Look for the Deployment {id} completed successfully log line.
  • Verify that S3 has the built files by running aws s3 ls s3://vercel-clone-outputs/dist/ and confirming your deployment ID folder exists.
  • Make sure you are using the exact ID from the curl response in the subdomain.

Help me debug why my deployment isn't showing up

Your deployment pipeline is fully operational. You can now take any public GitHub repo with a React app, POST its URL to the upload service, and have it live on a unique subdomain within minutes. Next up, the Secret Mission adds real-time status tracking so you always know whether a deployment is queued, building, or live.

Secret mission

Add Deployment Status Tracking with Redis

Your pipeline works, but it's a black box. Once a deploy request goes in, there's no way to know if it's queued, building, deployed, or failed. Add Redis-backed status tracking so your system reports real-time deployment state across all three services.

Clean Up Your Resources

Clean Up Your Resources

This project uses a mix of AWS cloud resources and local Docker containers. S3 and SQS both fall within AWS Free Tier limits for this project's usage, so there are no ongoing costs from leaving them idle.

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • S3 bucket (vercel-clone-outputs) storing source and built files.
  • SQS queue (free tier: 1 million requests/month).
  • Redis Docker container (local, port 6379).
  • Local project files (upload-service/, deploy-service/, request-handler/).

Keep everything running

No action needed. Choose this if you are still actively building or want to experiment further with your deployment pipeline.

  • Your S3 bucket and SQS queue remain available at no cost within Free Tier limits.
  • Your Redis container stays running locally until you stop Docker or restart your machine.
  • Monitor usage in the AWS Billing dashboard to stay within Free Tier.

Pause - I'll come back to this later

Shut down running processes to free up local memory. Your AWS resources stay in place at minimal cost.

  • Stop your three Node.js services (upload-service, deploy-service, request-handler) by pressing Ctrl+C in each terminal window.
  • Stop the Redis container by running:
docker stop $(docker ps -q --filter ancestor=redis:7)
  • S3 and SQS incur no cost at rest within Free Tier. You can restart everything later by running your services again and starting Redis with docker start.

Delete - I don't want to use this again

Remove all project resources and start fresh. This permanently deletes your S3 data and SQS queue.

  • Delete your S3 bucket and all its contents by running:
aws s3 rb s3://vercel-clone-outputs --force
  • Delete your SQS queue by running:
aws sqs delete-queue --queue-url [[SQS_QUEUE_URL="your SQS queue URL from earlier"]]
  • Stop and remove the Redis container by running:
docker stop $(docker ps -q --filter ancestor=redis:7)
docker rm $(docker ps -aq --filter ancestor=redis:7)
  • Delete your local project files by running:
cd ~/Desktop
rm -rf upload-service deploy-service request-handler

That's a Wrap!

That's a Wrap!

Nice work! You just built a deployment pipeline that mirrors how platforms like Vercel work under the hood.

You've learned how to:

  • Built a 3-service microservice pipeline that takes a GitHub URL, clones the repo, builds it, and serves the result as a live site.
  • Used AWS S3 as a durable artifact store for both source code and built output, and AWS SQS as a fault-tolerant job queue to decouple your services.
  • Built a reverse proxy that serves deployed sites via wildcard subdomain routing from S3.
  • Secret Mission: Added real-time deployment status tracking with Redis, giving your pipeline queued/building/deployed/failed state reporting.

Ready to quiz yourself?