Async Media Pipeline with AI Tagging

Introduction

30 Second Summary

Every time you upload a photo to Instagram or Shopify, the app doesn't process it while you wait. It queues the work in the background so you get an instant response.

In this project, you will build an asynchronous media processing pipeline that resizes, compresses, and AI-tags uploaded images using background workers. You will containerize the entire stack and deploy it to AWS.

What You'll Build

An Express API that accepts image uploads, queues processing jobs in Redis via BullMQ, and uses background workers to resize images with Sharp and auto-tag them using Hugging Face AI models.

By the end of this project, you'll have:

• A producer/worker pipeline where your Express API queues jobs and BullMQ workers process images asynchronously with AI-powered tagging.
• A Prisma database schema tracking each image's journey through the pipeline, including AI-generated tags and on-demand captions.
• A containerized deployment on AWS EC2 with Docker Compose orchestrating your API, PostgreSQL database, Redis queue, and background workers.
• Secret Mission: Add a Bull Board dashboard for real-time job monitoring so you can watch jobs flow through your pipeline.

Are there any prerequisites?

You need Node.js and Docker installed on your computer. You will also create a free AWS account and a free Hugging Face account during the project.

Set Up the Project and Docker Services

Your media processing pipeline has several services that need to work together. An Express API accepts uploads, PostgreSQL stores file metadata, Redis powers the job queue, and a background worker handles image processing and AI tagging.

Before writing any application code, you need a solid foundation. In this step, you'll initialize the project, install all dependencies, and spin up PostgreSQL and Redis using Docker Compose.

In this step, get ready to:

• Initialize a TypeScript project and install all runtime and dev dependencies.
• Create a Docker Compose file for PostgreSQL and Redis.
• Start the database services and initialize Prisma ORM.

Create the project and install dependencies

The project uses TypeScript with ES modules for type safety across the entire stack. You'll create the project folder, configure the module system, and install every package the pipeline needs.

• Open your terminal.
• Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
• Type Terminal (macOS) or PowerShell (Windows) and press Enter to open it.
• Move to your Desktop by running this command:

cd ~/Desktop

• Create your project folder, move into it, and initialize a Node.js project by running these commands:

mkdir media-pipeline
cd media-pipeline
npm init -y

What do these commands do?

• mkdir media-pipeline creates a new folder for your project.
• cd media-pipeline moves your terminal into that folder.
• npm init -y creates a package.json file with default settings. This file tracks your project's dependencies and scripts.

• Confirm the project was initialized by running:

ls

You should see package.json listed in the output.

Don't see package.json?

• Make sure you ran cd media-pipeline before npm init -y. You need to be inside the project folder.
• Verify Node.js is installed by running node --version. You need 20 or higher.

Why can't I initialize my project?

Now open the project in VS Code so you can edit the configuration files.

• Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
• Type Visual Studio Code and press Enter to open it.
• In VS Code, click File in the top menu bar.
• Click Open Folder.
• Select the media-pipeline folder on your Desktop, then click Open.

The npm init command created a default package.json with a placeholder test script. You need to configure it for ES modules and add the scripts that will start your server and worker later.

• In the VS Code Explorer sidebar, click package.json to open it.
• Find the "scripts" section. Replace it with the scripts below, and add "type": "module" right above the "scripts" line:

"type": "module",
"scripts": {
  "dev:server": "tsx src/server.ts",
  "dev:worker": "tsx src/worker.ts",
  "dev": "concurrently 'npm run dev:server' 'npm run dev:worker'"
},

What does this configuration do?

• "type": "module" tells Node.js to use ES module syntax (import/export) instead of the older CommonJS (require()). Prisma 7 and BullMQ both require ESM.
• dev:server runs the Express API, dev:worker runs the background job processor, and dev runs both at the same time.

• Leave the rest of the fields that npm init created (name, version, description, etc.) as they are.
• Save package.json.

Next, create the TypeScript configuration. This file tells TypeScript how to compile your code.

• Create a new file called tsconfig.json in the root of your media-pipeline folder by clicking the new file icon in the VS Code Explorer sidebar.
• Paste the following configuration into the file:

{
  "compilerOptions": {
    "module": "ESNext",
    "moduleResolution": "bundler",
    "target": "ES2023",
    "strict": true,
    "esModuleInterop": true,
    "ignoreDeprecations": "6.0"
  }
}

What do these compiler options do?

• module: "ESNext" and moduleResolution: "bundler" configure TypeScript to use modern ES module resolution. This matches the "type": "module" setting you added to package.json.
• target: "ES2023" tells TypeScript to output modern JavaScript that Node.js 20 understands.
• strict: true enables all strict type-checking options. This catches bugs before your code runs.
• esModuleInterop: true allows you to import CommonJS packages (like Express) using the standard import syntax.

• Save tsconfig.json.

Now install all the runtime packages your pipeline needs. This single command installs the web framework, file upload handler, job queue, image processor, database client, and AI SDK.

• Install the runtime dependencies by running this command in your terminal:

npm install express@5.2.1 multer@2.1.1 bullmq@5.76.2 ioredis@5.10.1 sharp@0.34.5 @prisma/client@7.8.0 @prisma/adapter-pg@7.8.0 pg@8.13.1 dotenv@16.5.0 @huggingface/inference@3.13.2

What are all these packages?

• express is the web framework for your API.
• multer handles file uploads from the browser.
• bullmq and ioredis power the Redis-backed job queue.
• sharp resizes, compresses, and converts images to WebP.
• @prisma/client and @prisma/adapter-pg connect your app to PostgreSQL through the Prisma ORM.
• @huggingface/inference is the official SDK for calling Hugging Face AI models.

• Install the development dependencies by running:

npm install --save-dev prisma@7.8.0 typescript@5.8.3 tsx@4.19.2 @types/node@22.15.3 @types/express@5.0.0 @types/pg@8.11.6 concurrently@9.1.2

What are dev dependencies?

Dev dependencies are packages you only need during development, not in production. prisma generates the database client, tsx runs TypeScript files directly without a separate compile step, and concurrently runs the server and worker at the same time.

• Verify the install succeeded by checking that node_modules exists:

ls

You should see node_modules, package.json, package-lock.json, and tsconfig.json in the output.

Seeing install errors?

• If you see EACCES permission denied, do not use sudo. Instead, fix your npm permissions by following the official npm guide.
• If sharp fails to install, make sure you are running Node.js 20.3.0 or higher. Check with node --version.

Help me fix my npm install errors.

✔️ Awesome, I've got everything!

Great. Double check you've saved both package.json and tsconfig.json before moving on.

ⓧ I'd like to double check the full code

Here is the full tsconfig.json for comparison:

{
  "compilerOptions": {
    "module": "ESNext",
    "moduleResolution": "bundler",
    "target": "ES2023",
    "strict": true,
    "esModuleInterop": true,
    "ignoreDeprecations": "6.0"
  }
}

Your package.json should include these fields (along with the defaults from npm init and the dependencies added by npm install):

"type": "module",
"scripts": {
  "dev:server": "tsx src/server.ts",
  "dev:worker": "tsx src/worker.ts",
  "dev": "concurrently 'npm run dev:server' 'npm run dev:worker'"
}

Create the Docker Compose file for PostgreSQL and Redis

Your pipeline stores file metadata in PostgreSQL and queues processing jobs in Redis. Instead of installing these databases directly on your machine, you'll run them as Docker containers. This keeps your setup clean and reproducible.

• Create a new file called docker-compose.dev.yml in the root of your media-pipeline folder.
• Paste the following configuration into the file:

services:
  # PostgreSQL database for storing image metadata
  postgres:
    image: postgres:16-alpine
    ports:
      - "5433:5432"
    environment:
      POSTGRES_DB: imageprocessor
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
    volumes:
      - pgdata_dev:/var/lib/postgresql/data

  # Redis for the BullMQ job queue
  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    volumes:
      - redisdata_dev:/data

volumes:
  pgdata_dev:
  redisdata_dev:

What does this Docker Compose file do?

• The postgres service runs PostgreSQL 16 on port 5433 (mapped from the container's internal port 5432). Port 5433 avoids conflicts if you already have PostgreSQL installed locally.
• The redis service runs Redis 7 on port 6379. BullMQ uses Redis as its backing store for the job queue.
• The volumes section ensures your data persists even if you stop and restart the containers.

• Save docker-compose.dev.yml.

✔️ Awesome, I've got everything!

Great. Make sure you've saved the file before moving on.

ⓧ I'd like to double check the full code

services:
  postgres:
    image: postgres:16-alpine
    ports:
      - "5433:5432"
    environment:
      POSTGRES_DB: imageprocessor
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
    volumes:
      - pgdata_dev:/var/lib/postgresql/data

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    volumes:
      - redisdata_dev:/data

volumes:
  pgdata_dev:
  redisdata_dev:

Start the services, initialize Prisma, and verify

With the Docker Compose file in place, you can start both databases with a single command. Then you'll initialize Prisma, which is the ORM that connects your TypeScript code to PostgreSQL.

• Start the PostgreSQL and Redis containers by running this command in your terminal:

docker compose -f docker-compose.dev.yml up -d

What does this command do?

• -f docker-compose.dev.yml tells Docker Compose to use your development file (instead of the default docker-compose.yml, which you'll create later for production).
• -d runs the containers in detached mode so they run in the background and free up your terminal.

• Verify both containers are running by running:

docker compose -f docker-compose.dev.yml ps

You should see two services listed, both with a status of running. One is postgres and the other is redis.

Containers not running?

• Make sure Docker Desktop is running. Look for the Docker whale icon in your system tray or menu bar.
• If port 5433 is already in use, check for an existing PostgreSQL process. On macOS, run lsof -i :5433. On Windows, run netstat -ano | findstr :5433.
• If you see a YAML syntax error, make sure your indentation in docker-compose.dev.yml uses spaces, not tabs.

Help me get my Docker containers running.

Now that PostgreSQL is running, you can initialize Prisma. Prisma generates a type-safe database client from your schema definition.

• Scaffold the Prisma files by running:

npx prisma init --datasource-provider postgresql --output ../generated/prisma

What does this command do?

This creates a prisma/ directory with a schema.prisma file. It also creates a .env file with a placeholder database URL.

The --output ../generated/prisma flag tells Prisma to generate its client into a generated/prisma/ folder. This is the new pattern in Prisma 7.

The command created a .env file, but it has a placeholder database URL. You need to update it with the correct connection string for your Docker PostgreSQL container.

• Open .env in VS Code.
• Replace the entire contents with the following:

DATABASE_URL="postgresql://postgres:postgres@localhost:5433/imageprocessor?schema=public"
REDIS_URL="redis://localhost:6379"
HF_TOKEN="your-hugging-face-token-here"
PORT=3000

What are these environment variables?

• DATABASE_URL points to your Docker PostgreSQL container on port 5433, using the database name and credentials from your Docker Compose file.
• REDIS_URL points to your Docker Redis container on port 6379.
• HF_TOKEN is your Hugging Face API token for AI tagging and captioning.

• Replace your-hugging-face-token-here with your actual Hugging Face token. If you don't have one yet, follow the steps below.

Get your Hugging Face token

• Go to huggingface.co/settings/tokens and sign in (or create a free account).
• Click Create new token.
• Give it a name like media-pipeline and select Read access.
• Copy the token and paste it into your .env file as the HF_TOKEN value.

• Save .env.

Prisma 7 requires a configuration file that tells it where to find the schema and how to connect to the database.

• Create a new file called prisma.config.ts in the root of your media-pipeline folder (not inside the prisma/ directory).
• Paste the following configuration:

import "dotenv/config";
import { defineConfig, env } from "prisma/config";

// Tell Prisma where the schema lives and how to connect
export default defineConfig({
  schema: "prisma/schema.prisma",
  migrations: {
    path: "prisma/migrations",
  },
  datasource: {
    url: env("DATABASE_URL"),
  },
});

What does this file do?

This is a Prisma 7 requirement. It defines three things: the path to your schema file, the directory for database migrations, and the database connection URL (pulled from your .env file).

• Save prisma.config.ts.
• Confirm your project structure by running this in the terminal:

ls

You should see docker-compose.dev.yml, node_modules, package.json, prisma/, prisma.config.ts, tsconfig.json, and .env in the output.

Missing prisma/ folder or .env?

• If the prisma/ folder is missing, rerun npx prisma init --datasource-provider postgresql --output ../generated/prisma. Make sure you run it from inside the media-pipeline folder.
• The .env file is hidden by default. Run ls -a to see hidden files, or toggle hidden files in VS Code's Explorer.

Help me verify my project setup.

✔️ Awesome, I've got everything!

Great. Make sure you've saved all files before moving on.

ⓧ I'd like to double check the full code

Here is the full .env file:

DATABASE_URL="postgresql://postgres:postgres@localhost:5433/imageprocessor?schema=public"
REDIS_URL="redis://localhost:6379"
HF_TOKEN="your-hugging-face-token-here"
PORT=3000

Here is the full prisma.config.ts file:

import "dotenv/config";
import { defineConfig, env } from "prisma/config";

export default defineConfig({
  schema: "prisma/schema.prisma",
  migrations: {
    path: "prisma/migrations",
  },
  datasource: {
    url: env("DATABASE_URL"),
  },
});

Your foundation is solid. You have a TypeScript project, all your dependencies installed, and PostgreSQL and Redis running in Docker containers. Next up, you'll define the database schema that tracks every image through the processing pipeline.

Define the Database Schema with Prisma

Your Docker services are running and Prisma is initialized. Now it is time to define the shape of your data.

Every image that flows through your pipeline needs metadata tracked in the database. You need fields for the original upload info, the processing status, the processed output, and the AI-generated tags and captions.

In this step, get ready to:

• Define a Prisma schema with a processing status enum and an image file model.
• Run a database migration to create the tables in PostgreSQL.
• Create a reusable Prisma client module for your application code.

Create the Prisma schema

The Prisma schema is a single file that defines your database models. Prisma uses it to generate type-safe queries and to create the actual tables in PostgreSQL through migrations.

When you initialized Prisma in the previous step, it created a basic prisma/schema.prisma file with just the generator and datasource blocks. You need to add an enum and a model to track images through the pipeline.

• Open prisma/schema.prisma in your editor and replace its entire contents with the following code:

// Configure the Prisma client generator
generator client {
  provider = "prisma-client"
  output   = "../generated/prisma"
}

// Use PostgreSQL as the database
datasource db {
  provider = "postgresql"
}

// Track where each image is in the processing pipeline
enum ProcessingStatus {
  PENDING
  PROCESSING
  COMPLETED
  FAILED
}

// Store metadata for each uploaded image
model ImageFile {
  id                String           @id @default(uuid())
  originalName      String
  fileName          String           @unique
  mimeType          String
  size              Int
  status            ProcessingStatus @default(PENDING)
  processedFileName String?          // Set after processing completes
  processedSize     Int?             // Size of the processed WebP file
  tags              String[]         @default([])  // AI-generated classification labels
  caption           String?          // AI-generated caption (on-demand)
  createdAt         DateTime         @default(now())
  updatedAt         DateTime         @updatedAt
}

What does this schema do?

The ProcessingStatus enum defines the four states an image can be in as it moves through your pipeline: PENDING (just uploaded), PROCESSING (worker picked it up), COMPLETED, or FAILED.

The ImageFile model stores everything about each image. Fields with ? are optional because they only get filled in after the worker processes the image or the user requests an AI caption.

The tags field is a PostgreSQL string array that stores AI-generated classification labels. The caption field stores an on-demand AI-generated description of the image.

• Save prisma/schema.prisma.

You will not see a visible change yet. The next substep runs the migration to create actual database tables from this schema.

✔️ Awesome, I've got everything!

Double check you have saved your file before moving on.

ⓧ I'd like to double check the full code

generator client {
  provider = "prisma-client"
  output   = "../generated/prisma"
}

datasource db {
  provider = "postgresql"
}

enum ProcessingStatus {
  PENDING
  PROCESSING
  COMPLETED
  FAILED
}

model ImageFile {
  id                String           @id @default(uuid())
  originalName      String
  fileName          String           @unique
  mimeType          String
  size              Int
  status            ProcessingStatus @default(PENDING)
  processedFileName String?
  processedSize     Int?
  tags              String[]         @default([])
  caption           String?
  createdAt         DateTime         @default(now())
  updatedAt         DateTime         @updatedAt
}

Run the migration and generate the Prisma Client

A database migration translates your Prisma schema into SQL commands that create the actual tables and columns in PostgreSQL. Prisma tracks each migration so you can evolve your schema over time without losing data.

• Create and apply the initial migration by running this command in your terminal:

npx prisma migrate dev --name init

What does this command do?

This command reads your schema, generates a SQL migration file in prisma/migrations/, and runs it against your PostgreSQL database. The --name init flag gives the migration a descriptive name.

• You should see output that includes Your database is now in sync with your schema and a line showing the migration name.

Migration failed?

Make sure your Docker containers are still running. Run docker compose -f docker-compose.dev.yml ps to check. If they are stopped, start them again with docker compose -f docker-compose.dev.yml up -d.

Check that your DATABASE_URL in .env uses port 5433 (not the default 5432) since that is what your Docker Compose file maps to.

Help me debug my Prisma migration error

• Generate the type-safe Prisma Client by running:

npx prisma generate

• You should see output confirming the client was generated to the generated/prisma directory.

Why generate separately?

The prisma generate command creates a type-safe client based on your schema. Every time you change schema.prisma, you need to regenerate the client so your TypeScript code stays in sync with the database.

Now you need a reusable module that creates a single Prisma client instance for your application. Prisma 7 requires a driver adapter to connect to the database. You will use the PrismaPg adapter for PostgreSQL.

• Create the src/lib/ directory by running this command:

mkdir -p src/lib

• Create a new file called src/lib/prisma.ts and add the following code:

import "dotenv/config";
import { PrismaPg } from "@prisma/adapter-pg";
import { PrismaClient } from "../../generated/prisma/client.js";

// Read the database connection string from .env
const connectionString = process.env.DATABASE_URL!;

// Create the PostgreSQL adapter and Prisma client
const adapter = new PrismaPg({ connectionString });
const prisma = new PrismaClient({ adapter });

export { prisma };

What does this code do?

This file creates a singleton Prisma client that the rest of your application imports. The PrismaPg adapter connects Prisma to PostgreSQL using the DATABASE_URL from your .env file.

The import path ../../generated/prisma/client.js points to the client you just generated. The .js extension is required because your project uses ESM modules.

• Save src/lib/prisma.ts.
• Confirm the file exists by running:

ls src/lib/

• You should see prisma.ts in the output.

Seeing an import error in your editor?

If your editor underlines the import from ../../generated/prisma/client.js, make sure you ran npx prisma generate successfully. The generated/prisma/ folder needs to exist for the import to resolve.

Help me fix my Prisma client import error

✔️ Awesome, I've got everything!

Double check you have saved both files before moving on.

ⓧ I'd like to double check the full code

generator client {
  provider = "prisma-client"
  output   = "../generated/prisma"
}

datasource db {
  provider = "postgresql"
}

enum ProcessingStatus {
  PENDING
  PROCESSING
  COMPLETED
  FAILED
}

model ImageFile {
  id                String           @id @default(uuid())
  originalName      String
  fileName          String           @unique
  mimeType          String
  size              Int
  status            ProcessingStatus @default(PENDING)
  processedFileName String?
  processedSize     Int?
  tags              String[]         @default([])
  caption           String?
  createdAt         DateTime         @default(now())
  updatedAt         DateTime         @updatedAt
}

import "dotenv/config";
import { PrismaPg } from "@prisma/adapter-pg";
import { PrismaClient } from "../../generated/prisma/client.js";

const connectionString = process.env.DATABASE_URL!;

const adapter = new PrismaPg({ connectionString });
const prisma = new PrismaClient({ adapter });

export { prisma };

Your database is ready and your Prisma client is configured. Next up, you will build the Express API that accepts image uploads and queues them for processing.

Build the Express API (the Producer)

In Step 2, you defined the database schema and ran your first migration. Now it's time to build the API that accepts image uploads and kicks off the processing pipeline.

When a user uploads an image, your server needs to respond immediately. Instead of processing the image during the request (which would block the server), the API saves the file, records it in the database, and drops a job into a Redis queue. A separate worker picks up the job later. This is the producer side of the producer/worker pattern.

In this step, get ready to:

• Set up Express with Multer for file uploads.
• Add listing, download, and AI caption endpoints.
• Create the Redis connection, job queue, and AI client modules.

Set up Express with Multer for file uploads

Your API needs to accept image files from users. Express handles HTTP requests, and Multer is a middleware that parses multipart form data (the format browsers use to send files).

• Create the uploads/ directory where Multer will save incoming files, by running this command:

mkdir -p uploads

• Confirm the directory exists by running:

ls

You should see uploads listed alongside src, prisma, and your other project files.

Now you'll create the main server file. It sets up Express, configures Multer, and defines the upload route.

• Create a new file called src/server.ts and add the following imports, Express setup, and Multer configuration:

import "dotenv/config";
import express from "express";
import multer from "multer";
import path from "path";
import fs from "fs";
import { fileURLToPath } from "url";
import { prisma } from "./lib/prisma.js";
import { imageQueue } from "./lib/queue.js";
import { hf } from "./lib/ai.js";

// ESM doesn't have __dirname by default, so we derive it
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);

const app = express();
const PORT = process.env.PORT || 3000;

// Parse JSON bodies and serve static files
app.use(express.json());
app.use(express.static(path.join(__dirname, "../public")));
app.use("/uploads", express.static(path.join(process.cwd(), "uploads")));

// Configure Multer to save files to uploads/ with unique names
const storage = multer.diskStorage({
  destination: (req, file, cb) => {
    cb(null, "uploads/");
  },
  filename: (req, file, cb) => {
    const uniqueSuffix = Date.now() + "-" + Math.round(Math.random() * 1e9);
    const ext = path.extname(file.originalname);
    cb(null, uniqueSuffix + ext);
  },
});

// Only allow image files, max 10MB
const upload = multer({
  storage,
  limits: { fileSize: 10 * 1024 * 1024 },
  fileFilter: (req, file, cb) => {
    if (file.mimetype.startsWith("image/")) {
      cb(null, true);
    } else {
      cb(new Error("Only image files are allowed"));
    }
  },
});

What does this code do?

The imports reference three files inside src/lib/ that you haven't created yet. You'll build those in the third substep. The server won't run until all three exist.

• fileURLToPath recreates __dirname because ESM modules don't provide it automatically like CommonJS does.
• Multer's diskStorage saves each upload to the uploads/ folder with a timestamp-based unique name to prevent collisions.
• The fileFilter rejects anything that isn't an image, and the size limit caps uploads at 10 MB.

Now add the upload route. This is the heart of the producer pattern.

• In src/server.ts, add the following POST /api/upload route below the Multer configuration:

// Upload route: save metadata to DB, then queue a processing job
app.post("/api/upload", upload.single("image"), async (req, res) => {
  if (!req.file) {
    res.status(400).json({ error: "No image file uploaded" });
    return;
  }

  // Save file metadata to PostgreSQL with PENDING status
  const imageRecord = await prisma.imageFile.create({
    data: {
      originalName: req.file.originalname,
      fileName: req.file.filename,
      mimeType: req.file.mimetype,
      size: req.file.size,
      status: "PENDING",
    },
  });

  // Queue a background job instead of processing inline
  await imageQueue.add("process-image", {
    imageId: imageRecord.id,
    filePath: req.file.path,
  });

  // Respond immediately — don't wait for processing
  res.status(201).json(imageRecord);
});

Why queue the job instead of processing it here?

Image resizing, format conversion, and AI tagging are CPU-intensive. If you did all that inside the request handler, the user would wait 5-10 seconds for a response. Other users' requests would queue up behind it.

By saving the metadata and immediately queuing a job, the API responds in milliseconds. The heavy work happens in a background worker (which you'll build in the next step). This is how services like Cloudinary and Shopify handle media uploads at scale.

• Save src/server.ts.

You won't see a change yet. The server can't start until you create the lib files in substep 3.

Add listing, download, and caption endpoints

The upload route handles incoming images. Now you need endpoints so the frontend can list images, download processed results, and request AI captions.

• In src/server.ts, add the following three GET routes below the upload route:

// List all images, newest first
app.get("/api/images", async (req, res) => {
  const images = await prisma.imageFile.findMany({
    orderBy: { createdAt: "desc" },
  });
  res.json(images);
});

// Get a single image record by ID
app.get("/api/images/:id", async (req, res) => {
  const image = await prisma.imageFile.findUnique({
    where: { id: req.params.id },
  });
  if (!image) {
    res.status(404).json({ error: "Image not found" });
    return;
  }
  res.json(image);
});

// Download the processed (WebP) version of an image
app.get("/api/images/:id/download", async (req, res) => {
  const image = await prisma.imageFile.findUnique({
    where: { id: req.params.id },
  });
  if (!image) {
    res.status(404).json({ error: "Image not found" });
    return;
  }
  if (image.status !== "COMPLETED" || !image.processedFileName) {
    res.status(400).json({ error: "Image processing not complete" });
    return;
  }
  const filePath = path.join("processed", image.processedFileName);
  res.download(filePath, image.processedFileName);
});

What do these routes do?

• GET /api/images returns every image record from the database, newest first. The frontend polls this to refresh the gallery.
• GET /api/images/:id returns a single image record by its UUID.
• GET /api/images/:id/download checks that processing is complete before serving the file. If the worker hasn't finished yet, it returns an error instead of a broken download.

• Save src/server.ts.

The last endpoint uses the Hugging Face Inference API to generate a natural-language caption for an image on demand.

• In src/server.ts, add the caption route and the app.listen call below the download route:

// Generate an AI caption for an image using Hugging Face
app.post("/api/images/:id/caption", async (req, res) => {
  const image = await prisma.imageFile.findUnique({
    where: { id: req.params.id },
  });
  if (!image) {
    res.status(404).json({ error: "Image not found" });
    return;
  }
  // Return existing caption if already generated
  if (image.caption) {
    res.json({ caption: image.caption });
    return;
  }

  // Read the original upload and convert to a Blob for the HF API
  const imageBuffer = fs.readFileSync(path.join("uploads", image.fileName));
  const blob = new Blob([imageBuffer]);

  const result = await hf.imageToText({
    data: blob,
    model: "nlpconnect/vit-gpt2-image-captioning",
  });

  const caption = result.generated_text || "No caption generated";

  // Persist the caption so we don't call the API again for this image
  await prisma.imageFile.update({
    where: { id: image.id },
    data: { caption },
  });

  res.json({ caption });
});

// Start the server
app.listen(PORT, () => {
  console.log(`Server running on http://localhost:${PORT}`);
});

What does the caption route do?

• It first checks if a caption already exists. If so, it returns it immediately without calling the API. This avoids burning rate-limited API calls.
• hf.imageToText() sends the image to the nlpconnect/vit-gpt2-image-captioning model, which returns a sentence describing the image (e.g. "a dog sitting on a bench in a park").
• The caption is saved to the database so future requests for the same image skip the API call entirely.

Why is captioning on-demand instead of automatic?

Auto-tagging runs in the background worker because it adds minimal extra time to the processing job. Caption generation is different. It's a separate API call that would double the AI requests for every upload. Making it on-demand (triggered by a button click) conserves your free-tier rate limits.

• Save src/server.ts.

You won't see a change yet. The server still needs the three lib modules you'll create next.

Before moving on, double check your complete src/server.ts file matches the reference below.

✔️ Awesome, I've got everything!

Great. Make sure you've saved the file before continuing.

ⓧ I'd like to double check the full code

import "dotenv/config";
import express from "express";
import multer from "multer";
import path from "path";
import fs from "fs";
import { fileURLToPath } from "url";
import { prisma } from "./lib/prisma.js";
import { imageQueue } from "./lib/queue.js";
import { hf } from "./lib/ai.js";

const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);

const app = express();
const PORT = process.env.PORT || 3000;

app.use(express.json());
app.use(express.static(path.join(__dirname, "../public")));
app.use("/uploads", express.static(path.join(process.cwd(), "uploads")));

const storage = multer.diskStorage({
  destination: (req, file, cb) => {
    cb(null, "uploads/");
  },
  filename: (req, file, cb) => {
    const uniqueSuffix = Date.now() + "-" + Math.round(Math.random() * 1e9);
    const ext = path.extname(file.originalname);
    cb(null, uniqueSuffix + ext);
  },
});

const upload = multer({
  storage,
  limits: { fileSize: 10 * 1024 * 1024 },
  fileFilter: (req, file, cb) => {
    if (file.mimetype.startsWith("image/")) {
      cb(null, true);
    } else {
      cb(new Error("Only image files are allowed"));
    }
  },
});

app.post("/api/upload", upload.single("image"), async (req, res) => {
  if (!req.file) {
    res.status(400).json({ error: "No image file uploaded" });
    return;
  }

  const imageRecord = await prisma.imageFile.create({
    data: {
      originalName: req.file.originalname,
      fileName: req.file.filename,
      mimeType: req.file.mimetype,
      size: req.file.size,
      status: "PENDING",
    },
  });

  await imageQueue.add("process-image", {
    imageId: imageRecord.id,
    filePath: req.file.path,
  });

  res.status(201).json(imageRecord);
});

app.get("/api/images", async (req, res) => {
  const images = await prisma.imageFile.findMany({
    orderBy: { createdAt: "desc" },
  });
  res.json(images);
});

app.get("/api/images/:id", async (req, res) => {
  const image = await prisma.imageFile.findUnique({
    where: { id: req.params.id },
  });
  if (!image) {
    res.status(404).json({ error: "Image not found" });
    return;
  }
  res.json(image);
});

app.get("/api/images/:id/download", async (req, res) => {
  const image = await prisma.imageFile.findUnique({
    where: { id: req.params.id },
  });
  if (!image) {
    res.status(404).json({ error: "Image not found" });
    return;
  }
  if (image.status !== "COMPLETED" || !image.processedFileName) {
    res.status(400).json({ error: "Image processing not complete" });
    return;
  }
  const filePath = path.join("processed", image.processedFileName);
  res.download(filePath, image.processedFileName);
});

app.post("/api/images/:id/caption", async (req, res) => {
  const image = await prisma.imageFile.findUnique({
    where: { id: req.params.id },
  });
  if (!image) {
    res.status(404).json({ error: "Image not found" });
    return;
  }
  if (image.caption) {
    res.json({ caption: image.caption });
    return;
  }

  const imageBuffer = fs.readFileSync(path.join("uploads", image.fileName));
  const blob = new Blob([imageBuffer]);

  const result = await hf.imageToText({
    data: blob,
    model: "nlpconnect/vit-gpt2-image-captioning",
  });

  const caption = result.generated_text || "No caption generated";

  await prisma.imageFile.update({
    where: { id: image.id },
    data: { caption },
  });

  res.json({ caption });
});

app.listen(PORT, () => {
  console.log(`Server running on http://localhost:${PORT}`);
});

Set up the Redis connection and BullMQ queue

Your server imports three modules from src/lib/ that don't exist yet. Each one wraps a third-party service into a clean, reusable export.

BullMQ uses Redis as its backing store for job data. The Redis connection must be configured with a specific setting that BullMQ requires.

• Create a new file called src/lib/redis.ts and add the following code:

import IORedis from "ioredis";

// Read the Redis URL from environment variables, default to localhost
const redisUrl = process.env.REDIS_URL || "redis://localhost:6379";

// maxRetriesPerRequest: null is required by BullMQ
const connection = new IORedis(redisUrl, {
  maxRetriesPerRequest: null,
});

export { connection };

Why maxRetriesPerRequest: null?

BullMQ uses Redis blocking commands (like BLPOP) to wait for new jobs. By default, IORedis retries failed requests a limited number of times and then throws an error. Setting maxRetriesPerRequest: null disables that limit so the blocking commands can wait indefinitely. Without this, BullMQ throws errors at startup.

• Save src/lib/redis.ts.

Next, create the job queue that sits on top of this Redis connection.

• Create a new file called src/lib/queue.ts and add the following code:

import { Queue } from "bullmq";
import { connection } from "./redis.js";

// Create a named queue with retry and cleanup settings
const imageQueue = new Queue("image-processing", {
  connection,
  defaultJobOptions: {
    attempts: 3,
    backoff: {
      type: "exponential",
      delay: 1000,
    },
    removeOnComplete: { count: 50 },
    removeOnFail: { count: 100 },
  },
});

export { imageQueue };

What does this queue configuration do?

• attempts: 3 means each job gets 3 tries before being marked as failed.
• backoff: { type: "exponential", delay: 1000 } waits 1 second before the first retry, 2 seconds before the second, and 4 seconds before the third. This gives transient errors time to resolve.
• removeOnComplete and removeOnFail keep the last 50 completed jobs and 100 failed jobs in Redis for inspection, then automatically clean up older ones.

• Save src/lib/queue.ts.

The last module wraps the Hugging Face Inference API client. The server uses it for on-demand caption generation, and the worker (built in the next step) uses it for auto-tagging.

• Create a new file called src/lib/ai.ts and add the following code:

import { InferenceClient } from "@huggingface/inference";

// Initialize the Hugging Face client with your API token
const hfToken = process.env.HF_TOKEN || "";
const hf = new InferenceClient(hfToken);

export { hf };

What is InferenceClient?

InferenceClient is the official Hugging Face TypeScript SDK for calling AI models. It provides typed methods like imageClassification() and imageToText() so you don't have to build raw HTTP requests.

• Save src/lib/ai.ts.

All three lib files are in place. Your server can now start successfully.

• Start the server by running this command:

npx tsx src/server.ts

You should see Server running on http://localhost:3000 in your terminal.

• Press Ctrl+C to stop the server.

Server not starting?

• If you see Cannot find module './lib/redis.js' (or similar for queue or ai), make sure all three files are saved inside the src/lib/ folder.
• If you see a Redis connection error, confirm your Docker containers are still running with docker compose -f docker-compose.dev.yml ps. Restart them if needed with docker compose -f docker-compose.dev.yml up -d.
• If you see EADDRINUSE, port 3000 is already in use. Stop whatever is running on that port and try again.

Help me debug my Express server startup error.

Your Express API is complete. It accepts uploads, queues processing jobs, and exposes endpoints for listing images, downloading results, and generating AI captions. Next up, you'll build the background worker that actually processes those queued jobs.

Build the Background Worker with AI Tagging

Your Express API queues jobs in Redis the moment a user uploads an image. Right now, those jobs sit in the queue with nobody picking them up.

In this step, you will build the consumer side of the pipeline. A BullMQ Worker process will listen for queued jobs, use Sharp to resize and compress images, then call the Hugging Face Inference API to automatically tag each image with AI-generated labels.

In this step, get ready to:

• Create a background worker that processes image jobs from the Redis queue.
• Integrate AI-powered image classification to auto-tag every upload.
• Test the full pipeline end to end by uploading an image and verifying the output.

Create the worker file with imports and directory setup

The worker runs as a separate process from your Express server. It connects to the same Redis queue and waits for jobs to appear.

• Create a new file called src/worker.ts inside your src folder.
• Add the imports and directory setup by pasting this code:

import "dotenv/config";
import { Worker } from "bullmq";
import sharp from "sharp";
import path from "path";
import fs from "fs";
import { prisma } from "./lib/prisma.js";
import { connection } from "./lib/redis.js";
import { hf } from "./lib/ai.js";

// Create the processed/ directory if it doesn't exist yet
if (!fs.existsSync("processed")) {
  fs.mkdirSync("processed", { recursive: true });
}

What does this code do?

The imports pull in everything the worker needs: BullMQ's Worker class to consume jobs, sharp for image processing, and the shared library modules you created earlier (prisma, connection, hf).

The fs.mkdirSync call ensures the processed/ directory exists before the worker tries to save output files there. Without it, the very first job would crash.

You won't see a change yet. The next chunk adds the core processing logic.

Add the Worker processor with Sharp and AI classification

This is the heart of the pipeline. When a job arrives, the worker resizes and compresses the image with Sharp, then sends it to the Hugging Face Inference API for automatic classification.

• In src/worker.ts, add the following code below the directory setup block:

const worker = new Worker(
  "image-processing",
  async (job) => {
    const { imageId, filePath } = job.data;

    // Mark the image as PROCESSING in the database
    await prisma.imageFile.update({
      where: { id: imageId },
      data: { status: "PROCESSING" },
    });

    // Resize, compress, and convert the image to WebP
    const processedFileName = `processed-${Date.now()}.webp`;
    const outputPath = path.join("processed", processedFileName);

    await sharp(filePath)
      .resize(800, null, { withoutEnlargement: true })
      .webp({ quality: 80 })
      .toFile(outputPath);

    const stats = fs.statSync(outputPath);

    // Call the Hugging Face image classification API for auto-tagging
    let tags: string[] = [];
    try {
      const imageBuffer = fs.readFileSync(filePath);
      const blob = new Blob([imageBuffer]);
      const classifications = await hf.imageClassification({
        data: blob,
        model: "google/vit-base-patch16-224",
      });
      tags = classifications.slice(0, 5).map((c) => c.label);
    } catch (err) {
      console.warn(`AI tagging failed for job ${job.id}:`, (err as Error).message);
    }

    // Save the processed file info and AI tags to the database
    await prisma.imageFile.update({
      where: { id: imageId },
      data: {
        status: "COMPLETED",
        processedFileName,
        processedSize: stats.size,
        tags,
      },
    });

    return { processedFileName, processedSize: stats.size, tags };
  },
  {
    connection,
    concurrency: 2,
  }
);

What does this code do?

The Worker listens on the image-processing queue. Every time the server adds a job, this processor function runs automatically.

• It first updates the database status to PROCESSING so the UI can show the image is being worked on.
• Sharp resizes the image to a max width of 800 pixels, compresses it, and converts it to WebP format. The withoutEnlargement: true option prevents small images from being scaled up.
• The AI classification call sends the original image to the google/vit-base-patch16-224 model on Hugging Face. It returns labels like "golden retriever", "lakeside", or "sports car".
• The try/catch around the AI call is intentional. If Hugging Face is rate-limited or unreachable, the image still gets processed and marked COMPLETED with empty tags. AI failure should never block image processing.
• The concurrency: 2 option lets the worker handle two jobs in parallel.

Why wrap AI tagging in try/catch?

The Hugging Face free tier enforces rate limits in 5-minute windows. If the API is temporarily unavailable, the image processing itself should still succeed. Tags are a nice-to-have enhancement, not a hard requirement for the pipeline to function.

You won't see a change yet. The next chunk adds the event handlers that log results and handle failures.

Add event handlers and the startup log

• In src/worker.ts, add the following code below the Worker constructor:

// Log when a job completes successfully
worker.on("completed", (job) => {
  console.log(`Job ${job.id} completed`);
});

// Log failures and update the database status to FAILED
worker.on("failed", async (job, err) => {
  console.error(`Job ${job?.id} failed:`, err.message);
  if (job) {
    await prisma.imageFile.update({
      where: { id: job.data.imageId },
      data: { status: "FAILED" },
    });
  }
});

console.log("Worker started, waiting for jobs...");

What does this code do?

• The completed event fires after a job's processor function returns successfully. It logs the job ID so you can track progress in the terminal.
• The failed event fires when all retry attempts are exhausted. It marks the image as FAILED in the database so the UI can show the error state.
• The final console.log confirms the worker process started and is listening for jobs.

• Save src/worker.ts.

Seeing red squiggles in your editor?

• Make sure all import paths end with .js (e.g., ./lib/prisma.js). ESM resolution requires the file extension even for TypeScript files.
• Confirm that src/lib/ai.ts, src/lib/redis.ts, and src/lib/prisma.ts all exist from the previous steps.
• help me fix import errors in my worker.ts file

✔️ Awesome, I've got everything!

Double check you have saved src/worker.ts.

ⓧ I'd like to double check the full code

Here is the complete src/worker.ts file for comparison.

import "dotenv/config";
import { Worker } from "bullmq";
import sharp from "sharp";
import path from "path";
import fs from "fs";
import { prisma } from "./lib/prisma.js";
import { connection } from "./lib/redis.js";
import { hf } from "./lib/ai.js";

if (!fs.existsSync("processed")) {
  fs.mkdirSync("processed", { recursive: true });
}

const worker = new Worker(
  "image-processing",
  async (job) => {
    const { imageId, filePath } = job.data;

    await prisma.imageFile.update({
      where: { id: imageId },
      data: { status: "PROCESSING" },
    });

    const processedFileName = `processed-${Date.now()}.webp`;
    const outputPath = path.join("processed", processedFileName);

    await sharp(filePath)
      .resize(800, null, { withoutEnlargement: true })
      .webp({ quality: 80 })
      .toFile(outputPath);

    const stats = fs.statSync(outputPath);

    let tags: string[] = [];
    try {
      const imageBuffer = fs.readFileSync(filePath);
      const blob = new Blob([imageBuffer]);
      const classifications = await hf.imageClassification({
        data: blob,
        model: "google/vit-base-patch16-224",
      });
      tags = classifications.slice(0, 5).map((c) => c.label);
    } catch (err) {
      console.warn(`AI tagging failed for job ${job.id}:`, (err as Error).message);
    }

    await prisma.imageFile.update({
      where: { id: imageId },
      data: {
        status: "COMPLETED",
        processedFileName,
        processedSize: stats.size,
        tags,
      },
    });

    return { processedFileName, processedSize: stats.size, tags };
  },
  {
    connection,
    concurrency: 2,
  }
);

worker.on("completed", (job) => {
  console.log(`Job ${job.id} completed`);
});

worker.on("failed", async (job, err) => {
  console.error(`Job ${job?.id} failed:`, err.message);
  if (job) {
    await prisma.imageFile.update({
      where: { id: job.data.imageId },
      data: { status: "FAILED" },
    });
  }
});

console.log("Worker started, waiting for jobs...");

Run the full pipeline end to end

Your package.json already has a dev script that uses concurrently to start the Express server and the BullMQ worker at the same time. Time to fire it up and test.

• Make sure your Docker containers for PostgreSQL and Redis are still running. Confirm by running:

docker compose -f docker-compose.dev.yml ps

You should see both postgres and redis services listed with a status of Up.

Containers not running?

• Start them again by running docker compose -f docker-compose.dev.yml up -d.
• Make sure Docker Desktop is open and running on your machine.

• Start both the server and worker by running:

npm run dev

You should see two lines in your terminal: Server running on http://localhost:3000 and Worker started, waiting for jobs....

What is concurrently doing?

The dev script runs npm run dev:server and npm run dev:worker in parallel. This mirrors a production setup where the API and worker are separate processes that share the same Redis queue.

• Open a second terminal window. Test the upload endpoint by sending an image with curl:

curl -X POST -F "image=@/path/to/any-image.jpg" http://localhost:3000/api/upload

Pick any image on your computer

Replace /path/to/any-image.jpg with the actual path to a JPEG or PNG file on your machine. For example, ~/Desktop/photo.jpg. Any image under 10 MB works.

The curl command should immediately return a JSON response with "status": "PENDING". That instant response is the whole point of asynchronous processing.

• Switch back to the terminal running npm run dev.

Within a few seconds, you should see the worker log a message like Job <id> completed. That means the image was resized, converted to WebP, and tagged by the AI model.

• Verify the processed file was created by running:

ls processed/

You should see a file named something like processed-1746188400000.webp.

• Check the database to confirm AI tags were stored by running this in your second terminal:

curl http://localhost:3000/api/images | npx -y json

You should see a JSON array with your image record. The status field should read COMPLETED, and the tags field should contain up to 5 classification labels from the AI model.

Tags array is empty?

• Check that your HF_TOKEN in the .env file is a real Hugging Face access token, not the placeholder value.
• Look at the terminal running your worker. If you see AI tagging failed warnings, the Hugging Face API may be rate-limiting you. Wait a few minutes and try again.
• The image still processes and reaches COMPLETED status even if tagging fails. This is the expected behavior.
• help me debug why my Hugging Face AI tags are empty

Seeing a connection error when npm run dev starts?

• The worker needs both PostgreSQL and Redis running. Run docker compose -f docker-compose.dev.yml ps to confirm both containers are up.
• If the worker crashes with ECONNREFUSED on port 6379, Redis is not running. Restart the containers.
• help me fix connection errors when starting my worker

Your pipeline is alive. Images flow from the API to Redis to the worker and back to the database, complete with AI-generated tags. Next up, you will build a web UI so users can see all of this in action without touching curl.

Build the Web UI with Real-Time Status and AI Features

Your backend pipeline is fully functional. The Express API accepts uploads, BullMQ queues jobs, and the worker processes images with AI tagging.

Right now, the only way to interact with the pipeline is through raw API calls. In this step, you will build a web UI with drag-and-drop uploads, real-time status polling, AI tag display, and on-demand caption generation.

In this step, get ready to:

• Create an HTML page with a drag-and-drop upload zone and image gallery.
• Add JavaScript to handle uploads, render pipeline status, display AI tags, and generate captions on demand.

Create the HTML page with upload form and gallery

Your Express server already serves static files from the public/ directory. You need to create the HTML page that users will see when they visit http://localhost:3000.

• Create a public folder in your project root (the same level as src/) by running this command:

mkdir -p public

• Create a new file called index.html inside the public/ folder, then paste the following code into it:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Media Processing Pipeline</title>
  <style>
    /* Reset all default margins and paddings */
    * { margin: 0; padding: 0; box-sizing: border-box; }
    body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #0f172a; color: #e2e8f0; padding: 2rem; }
    h1 { text-align: center; margin-bottom: 0.5rem; font-size: 1.8rem; }
    .subtitle { text-align: center; color: #64748b; margin-bottom: 2rem; font-size: 0.95rem; }

    /* Drag-and-drop upload zone */
    .upload-zone {
      border: 2px dashed #475569; border-radius: 12px; padding: 3rem;
      text-align: center; cursor: pointer; transition: all 0.2s;
      max-width: 600px; margin: 0 auto 2rem;
    }
    .upload-zone:hover, .upload-zone.dragover { border-color: #3b82f6; background: #1e293b; }
    .upload-zone p { font-size: 1.1rem; color: #94a3b8; }
    .upload-zone input { display: none; }

    /* Gallery grid for image cards */
    .gallery { display: grid; grid-template-columns: repeat(auto-fill, minmax(300px, 1fr)); gap: 1.5rem; max-width: 1200px; margin: 0 auto; }
    .card { background: #1e293b; border-radius: 12px; padding: 1.2rem; border: 1px solid #334155; }
    .card img { width: 100%; height: 180px; object-fit: cover; border-radius: 8px; margin-bottom: 0.8rem; }
    .card h3 { font-size: 0.9rem; margin-bottom: 0.4rem; word-break: break-all; }
    .card .meta { font-size: 0.8rem; color: #94a3b8; margin-bottom: 0.5rem; }

    /* Color-coded status pills */
    .status { display: inline-block; padding: 0.2rem 0.6rem; border-radius: 999px; font-size: 0.75rem; font-weight: 600; }
    .status.PENDING { background: #fbbf24; color: #1e293b; }
    .status.PROCESSING { background: #3b82f6; color: white; }
    .status.COMPLETED { background: #22c55e; color: white; }
    .status.FAILED { background: #ef4444; color: white; }

    /* AI tag pills and caption text */
    .tags { display: flex; flex-wrap: wrap; gap: 0.3rem; margin-top: 0.5rem; margin-bottom: 0.5rem; }
    .tag { background: #334155; color: #93c5fd; padding: 0.15rem 0.5rem; border-radius: 999px; font-size: 0.7rem; }
    .caption-text { font-style: italic; color: #a5b4fc; font-size: 0.85rem; margin-top: 0.5rem; padding: 0.5rem; background: #0f172a; border-radius: 6px; }

    /* Action buttons */
    .download-btn, .caption-btn {
      display: inline-block; margin-top: 0.5rem; padding: 0.4rem 1rem;
      background: #3b82f6; color: white; border-radius: 6px;
      text-decoration: none; font-size: 0.85rem; border: none; cursor: pointer;
    }
    .caption-btn { background: #8b5cf6; margin-left: 0.5rem; }
    .download-btn:hover { background: #2563eb; }
    .caption-btn:hover { background: #7c3aed; }
    .caption-btn:disabled { background: #475569; cursor: not-allowed; }
    .savings { color: #22c55e; font-weight: 600; }
  </style>
</head>
<body>
  <h1>Media Processing Pipeline</h1>
  <p class="subtitle">Upload images to process through the async pipeline (resize, compress, convert to WebP, AI tagging)</p>

  <!-- Drag-and-drop upload area -->
  <div class="upload-zone" id="uploadZone">
    <p>Drag and drop images here, or click to select</p>
    <input type="file" id="fileInput" accept="image/*">
  </div>

  <!-- Gallery grid populated by app.js -->
  <div class="gallery" id="gallery"></div>

  <script src="app.js"></script>
</body>
</html>

What does this HTML do?

The page has three main sections:

• An upload zone (#uploadZone) with a hidden file input. Clicking the zone or dragging a file onto it triggers the upload.
• A gallery grid (#gallery) that will be populated dynamically by JavaScript. Each image gets a card with its status, AI tags, and action buttons.
• Color-coded status pills show exactly where each image is in the pipeline: yellow for PENDING, blue for PROCESSING, green for COMPLETED, red for FAILED.

• Save public/index.html.
• Make sure your dev server is running with npm run dev, then open http://localhost:3000 in your browser.
• You should see a dark page with the title "Media Processing Pipeline" and a dashed upload zone in the center.

Seeing a blank page or "Cannot GET /"?

• Make sure the file is saved as public/index.html (not src/public/index.html). The public/ folder should be at the project root, next to src/.
• Check your terminal for errors. If you see Cannot find module, restart the dev server with npm run dev.
• Still stuck? Help me debug why my index.html page is not loading at localhost:3000.

Add JavaScript to handle uploads, display AI tags, and generate captions

The HTML page is static right now. You need client-side JavaScript to handle file uploads, poll the API for status updates, and render each image's journey through the pipeline.

• Create a new file called app.js inside the public/ folder.
• Add the event listeners and upload function by pasting this code into public/app.js:

// Cache DOM references for the upload zone, file input, and gallery
const uploadZone = document.getElementById("uploadZone");
const fileInput = document.getElementById("fileInput");
const gallery = document.getElementById("gallery");

// Clicking the upload zone opens the hidden file input
uploadZone.addEventListener("click", () => fileInput.click());

// Highlight the zone when a file is dragged over it
uploadZone.addEventListener("dragover", (e) => {
  e.preventDefault();
  uploadZone.classList.add("dragover");
});

// Remove the highlight when the file leaves the zone
uploadZone.addEventListener("dragleave", () => {
  uploadZone.classList.remove("dragover");
});

// Handle the dropped file
uploadZone.addEventListener("drop", (e) => {
  e.preventDefault();
  uploadZone.classList.remove("dragover");
  const files = e.dataTransfer.files;
  if (files.length > 0) {
    uploadFile(files[0]);
  }
});

// Handle file selection via the file input dialog
fileInput.addEventListener("change", () => {
  if (fileInput.files.length > 0) {
    uploadFile(fileInput.files[0]);
    fileInput.value = "";
  }
});

// Send the selected file to the API using FormData
async function uploadFile(file) {
  const formData = new FormData();
  formData.append("image", file);

  uploadZone.querySelector("p").textContent = "Uploading...";

  try {
    const response = await fetch("/api/upload", {
      method: "POST",
      body: formData,
    });
    if (!response.ok) {
      const error = await response.json();
      throw new Error(error.error || "Upload failed");
    }
    uploadZone.querySelector("p").textContent =
      "Queued for processing! Drag and drop more images, or click to select";
    loadImages();
  } catch (err) {
    uploadZone.querySelector("p").textContent = "Upload failed: " + err.message;
    setTimeout(() => {
      uploadZone.querySelector("p").textContent =
        "Drag and drop images here, or click to select";
    }, 3000);
  }
}

What does this code do?

• Five event listeners handle the drag-and-drop interaction: click opens the file picker, dragover highlights the zone, dragleave removes the highlight, drop captures the file, and change handles the file input dialog.
• The uploadFile() function wraps the file in a FormData object and sends it to POST /api/upload. On success, it refreshes the gallery. On failure, it shows the error for 3 seconds.

• Save public/app.js.
• You won't see a change in the browser yet. The next chunk adds the gallery rendering that makes uploaded images visible.

• Add the helper functions and caption generation logic below the uploadFile() function in public/app.js:

// Convert byte counts to human-readable sizes
function formatSize(bytes) {
  if (bytes < 1024) return bytes + " B";
  if (bytes < 1024 * 1024) return (bytes / 1024).toFixed(1) + " KB";
  return (bytes / (1024 * 1024)).toFixed(1) + " MB";
}

// Call the caption API and replace the button with the generated text
async function generateCaption(imageId, buttonEl) {
  buttonEl.disabled = true;
  buttonEl.textContent = "Generating...";

  try {
    const response = await fetch(`/api/images/${imageId}/caption`, {
      method: "POST",
    });
    const data = await response.json();
    const card = buttonEl.closest(".card");
    buttonEl.remove();
    const captionEl = document.createElement("p");
    captionEl.className = "caption-text";
    captionEl.textContent = '"' + data.caption + '"';
    card.appendChild(captionEl);
  } catch (err) {
    buttonEl.disabled = false;
    buttonEl.textContent = "Generate Caption";
  }
}

What does this code do?

• formatSize() converts raw byte counts (like 1048576) into readable strings (like "1.0 MB") for the UI.
• generateCaption() calls POST /api/images/:id/caption which triggers the Hugging Face image-to-text model on the server. While waiting, the button shows "Generating..." and is disabled to prevent duplicate requests.

• Save public/app.js.
• You won't see a change yet. The next chunk adds the rendering function that displays the gallery.

• Add the image rendering and polling logic below the generateCaption() function in public/app.js:

// Build the gallery HTML from the image records
function renderImages(images) {
  gallery.innerHTML = images
    .map((img) => {
      // Start with the status pill and original file size
      let details = `<span class="status ${img.status}">${img.status}</span>`;
      details += `<p class="meta">Original: ${formatSize(img.size)}</p>`;

      // Show AI-generated tags as colored pills
      if (img.tags && img.tags.length > 0) {
        details += `<div class="tags">${img.tags.map((t) => `<span class="tag">${t}</span>`).join("")}</div>`;
      }

      // For completed images, show compression savings and action buttons
      if (img.status === "COMPLETED" && img.processedSize) {
        const saved = Math.round((1 - img.processedSize / img.size) * 100);
        details += `<p class="meta">Compressed: ${formatSize(img.processedSize)} <span class="savings">(${saved}% smaller)</span></p>`;
        details += `<a class="download-btn" href="/api/images/${img.id}/download">Download WebP</a>`;
        if (img.caption) {
          details += `<p class="caption-text">"${img.caption}"</p>`;
        } else {
          details += `<button class="caption-btn" onclick="generateCaption('${img.id}', this)">Generate Caption</button>`;
        }
      }

      return `
        <div class="card">
          <img src="/uploads/${img.fileName}" alt="${img.originalName}">
          <h3>${img.originalName}</h3>
          ${details}
        </div>
      `;
    })
    .join("");
}

// Fetch all images from the API and render them
async function loadImages() {
  try {
    const response = await fetch("/api/images");
    const images = await response.json();
    renderImages(images);
  } catch (err) {
    console.error("Failed to load images:", err);
  }
}

// Load images immediately, then poll every 3 seconds for status updates
loadImages();
setInterval(loadImages, 3000);

What does this code do?

• renderImages() maps each image record into an HTML card. Each card shows the original image, a color-coded status pill, AI-generated tags as small rounded pills, compression savings, a download link, and a "Generate Caption" button.
• loadImages() fetches all image records from GET /api/images and passes them to the render function.
• setInterval(loadImages, 3000) polls the API every 3 seconds. This is how status transitions (PENDING to PROCESSING to COMPLETED) appear in real time without the user refreshing the page.

• Save public/app.js.
• Refresh http://localhost:3000 in your browser. The upload zone should still be visible.
• Drag and drop any image file onto the upload zone (or click to select one).
• Watch the gallery below. You should see a card appear with a yellow PENDING pill, which changes to blue PROCESSING, then green COMPLETED within a few seconds.
• Once the status turns green, you should see AI-generated tag pills (like "golden retriever" or "sports car"), the compression savings, a blue Download WebP link, and a purple Generate Caption button.
• Click the Generate Caption button. After a few seconds, an italic caption should appear describing the image content.

Upload not working or gallery empty?

• Open your browser's developer console (press F12, then click the Console tab). Look for red error messages that indicate which request is failing.
• If you see Failed to load images, make sure both the server and worker are running. Stop your terminal and restart with npm run dev.
• If images stay stuck on PENDING, the worker may not be connected to Redis. Check the terminal output for worker errors.
• If tags are empty but the status is COMPLETED, your HF_TOKEN may be missing or invalid. Check your .env file. The worker logs a warning when AI tagging fails.
• My web UI is not displaying images or tags correctly after uploading. Help me debug.

✔️ Awesome, I've got everything!

Make sure you have saved both public/index.html and public/app.js.

ⓧ I'd like to double check the full code

Here is the complete public/index.html:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Media Processing Pipeline</title>
  <style>
    * { margin: 0; padding: 0; box-sizing: border-box; }
    body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #0f172a; color: #e2e8f0; padding: 2rem; }
    h1 { text-align: center; margin-bottom: 0.5rem; font-size: 1.8rem; }
    .subtitle { text-align: center; color: #64748b; margin-bottom: 2rem; font-size: 0.95rem; }
    .upload-zone {
      border: 2px dashed #475569; border-radius: 12px; padding: 3rem;
      text-align: center; cursor: pointer; transition: all 0.2s;
      max-width: 600px; margin: 0 auto 2rem;
    }
    .upload-zone:hover, .upload-zone.dragover { border-color: #3b82f6; background: #1e293b; }
    .upload-zone p { font-size: 1.1rem; color: #94a3b8; }
    .upload-zone input { display: none; }
    .gallery { display: grid; grid-template-columns: repeat(auto-fill, minmax(300px, 1fr)); gap: 1.5rem; max-width: 1200px; margin: 0 auto; }
    .card { background: #1e293b; border-radius: 12px; padding: 1.2rem; border: 1px solid #334155; }
    .card img { width: 100%; height: 180px; object-fit: cover; border-radius: 8px; margin-bottom: 0.8rem; }
    .card h3 { font-size: 0.9rem; margin-bottom: 0.4rem; word-break: break-all; }
    .card .meta { font-size: 0.8rem; color: #94a3b8; margin-bottom: 0.5rem; }
    .status { display: inline-block; padding: 0.2rem 0.6rem; border-radius: 999px; font-size: 0.75rem; font-weight: 600; }
    .status.PENDING { background: #fbbf24; color: #1e293b; }
    .status.PROCESSING { background: #3b82f6; color: white; }
    .status.COMPLETED { background: #22c55e; color: white; }
    .status.FAILED { background: #ef4444; color: white; }
    .tags { display: flex; flex-wrap: wrap; gap: 0.3rem; margin-top: 0.5rem; margin-bottom: 0.5rem; }
    .tag { background: #334155; color: #93c5fd; padding: 0.15rem 0.5rem; border-radius: 999px; font-size: 0.7rem; }
    .caption-text { font-style: italic; color: #a5b4fc; font-size: 0.85rem; margin-top: 0.5rem; padding: 0.5rem; background: #0f172a; border-radius: 6px; }
    .download-btn, .caption-btn {
      display: inline-block; margin-top: 0.5rem; padding: 0.4rem 1rem;
      background: #3b82f6; color: white; border-radius: 6px;
      text-decoration: none; font-size: 0.85rem; border: none; cursor: pointer;
    }
    .caption-btn { background: #8b5cf6; margin-left: 0.5rem; }
    .download-btn:hover { background: #2563eb; }
    .caption-btn:hover { background: #7c3aed; }
    .caption-btn:disabled { background: #475569; cursor: not-allowed; }
    .savings { color: #22c55e; font-weight: 600; }
  </style>
</head>
<body>
  <h1>Media Processing Pipeline</h1>
  <p class="subtitle">Upload images to process through the async pipeline (resize, compress, convert to WebP, AI tagging)</p>
  <div class="upload-zone" id="uploadZone">
    <p>Drag and drop images here, or click to select</p>
    <input type="file" id="fileInput" accept="image/*">
  </div>
  <div class="gallery" id="gallery"></div>
  <script src="app.js"></script>
</body>
</html>

Here is the complete public/app.js:

const uploadZone = document.getElementById("uploadZone");
const fileInput = document.getElementById("fileInput");
const gallery = document.getElementById("gallery");

uploadZone.addEventListener("click", () => fileInput.click());

uploadZone.addEventListener("dragover", (e) => {
  e.preventDefault();
  uploadZone.classList.add("dragover");
});

uploadZone.addEventListener("dragleave", () => {
  uploadZone.classList.remove("dragover");
});

uploadZone.addEventListener("drop", (e) => {
  e.preventDefault();
  uploadZone.classList.remove("dragover");
  const files = e.dataTransfer.files;
  if (files.length > 0) {
    uploadFile(files[0]);
  }
});

fileInput.addEventListener("change", () => {
  if (fileInput.files.length > 0) {
    uploadFile(fileInput.files[0]);
    fileInput.value = "";
  }
});

async function uploadFile(file) {
  const formData = new FormData();
  formData.append("image", file);

  uploadZone.querySelector("p").textContent = "Uploading...";

  try {
    const response = await fetch("/api/upload", {
      method: "POST",
      body: formData,
    });
    if (!response.ok) {
      const error = await response.json();
      throw new Error(error.error || "Upload failed");
    }
    uploadZone.querySelector("p").textContent =
      "Queued for processing! Drag and drop more images, or click to select";
    loadImages();
  } catch (err) {
    uploadZone.querySelector("p").textContent = "Upload failed: " + err.message;
    setTimeout(() => {
      uploadZone.querySelector("p").textContent =
        "Drag and drop images here, or click to select";
    }, 3000);
  }
}

function formatSize(bytes) {
  if (bytes < 1024) return bytes + " B";
  if (bytes < 1024 * 1024) return (bytes / 1024).toFixed(1) + " KB";
  return (bytes / (1024 * 1024)).toFixed(1) + " MB";
}

async function generateCaption(imageId, buttonEl) {
  buttonEl.disabled = true;
  buttonEl.textContent = "Generating...";

  try {
    const response = await fetch(`/api/images/${imageId}/caption`, {
      method: "POST",
    });
    const data = await response.json();
    const card = buttonEl.closest(".card");
    buttonEl.remove();
    const captionEl = document.createElement("p");
    captionEl.className = "caption-text";
    captionEl.textContent = '"' + data.caption + '"';
    card.appendChild(captionEl);
  } catch (err) {
    buttonEl.disabled = false;
    buttonEl.textContent = "Generate Caption";
  }
}

function renderImages(images) {
  gallery.innerHTML = images
    .map((img) => {
      let details = `<span class="status ${img.status}">${img.status}</span>`;
      details += `<p class="meta">Original: ${formatSize(img.size)}</p>`;

      if (img.tags && img.tags.length > 0) {
        details += `<div class="tags">${img.tags.map((t) => `<span class="tag">${t}</span>`).join("")}</div>`;
      }

      if (img.status === "COMPLETED" && img.processedSize) {
        const saved = Math.round((1 - img.processedSize / img.size) * 100);
        details += `<p class="meta">Compressed: ${formatSize(img.processedSize)} <span class="savings">(${saved}% smaller)</span></p>`;
        details += `<a class="download-btn" href="/api/images/${img.id}/download">Download WebP</a>`;
        if (img.caption) {
          details += `<p class="caption-text">"${img.caption}"</p>`;
        } else {
          details += `<button class="caption-btn" onclick="generateCaption('${img.id}', this)">Generate Caption</button>`;
        }
      }

      return `
        <div class="card">
          <img src="/uploads/${img.fileName}" alt="${img.originalName}">
          <h3>${img.originalName}</h3>
          ${details}
        </div>
      `;
    })
    .join("");
}

async function loadImages() {
  try {
    const response = await fetch("/api/images");
    const images = await response.json();
    renderImages(images);
  } catch (err) {
    console.error("Failed to load images:", err);
  }
}

loadImages();
setInterval(loadImages, 3000);

Your pipeline now has a real frontend. Users can drag and drop images, watch them flow through the queue in real time, see AI-generated tags, and request captions on demand. Next up, you will containerize the entire stack with Docker so it can run anywhere.

Containerize the Pipeline with Docker

Your pipeline works locally with separate terminal windows running the server and worker. In production, you need a way to package everything together so the entire stack starts with a single command on any machine.

In this step, you will containerize the full pipeline using Docker. You will write a multi-stage Dockerfile to build a lightweight image, create a production Docker Compose file to orchestrate all three services (app, PostgreSQL, Redis), and verify the full pipeline runs inside containers.

In this step, get ready to:

• Write a multi-stage Dockerfile that builds and runs the pipeline as a non-root user.
• Create a production Docker Compose file with app, PostgreSQL, and Redis services.
• Build the Docker image and test the full pipeline locally in containers.

Write the Dockerfile

A Dockerfile is a recipe that tells Docker how to build your application into an image. You will use a multi-stage build, which creates a lightweight production image by separating the "build" step from the "run" step.

Why multi-stage builds?

A single-stage Dockerfile would include all the build tools and dev dependencies in the final image. That bloats the image size and increases the attack surface.

With multi-stage builds, the first stage ("builder") installs everything and generates the Prisma client. The second stage ("runner") copies only what the app needs at runtime. The final image is smaller and more secure.

Before writing the Dockerfile, you need a .dockerignore file. This tells Docker which files to exclude from the build context, keeping the image clean and the build fast.

• Create a new file called .dockerignore in the root of your project folder and add the following:

node_modules
uploads
processed
generated
.env
*.pem
.git

What does this file do?

• node_modules is excluded because the Dockerfile runs npm ci to install fresh dependencies inside the container.
• uploads, processed, and generated are runtime artifacts that should not be baked into the image.
• .env and *.pem contain secrets that must never end up in a Docker image.

• Save .dockerignore.

• Now create a file called Dockerfile (no file extension) in the root of your project folder and add the following:

# Stage 1: Build stage - install deps and generate Prisma client
FROM node:20-alpine AS builder

WORKDIR /app

# Copy package files and install dependencies
COPY package.json package-lock.json ./
RUN npm ci

# Copy Prisma schema and config, then generate the client
COPY prisma ./prisma
COPY prisma.config.ts ./
RUN npx prisma generate

# Copy all remaining source files
COPY . .

# Stage 2: Production stage - only copy what the app needs to run
FROM node:20-alpine AS runner

WORKDIR /app

# Create a non-root user for security
RUN addgroup -S appgroup && adduser -S appuser -G appgroup

# Copy runtime files from the builder stage
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/generated ./generated
COPY --from=builder /app/package.json ./
COPY --from=builder /app/prisma ./prisma
COPY --from=builder /app/prisma.config.ts ./
COPY --from=builder /app/src ./src
COPY --from=builder /app/public ./public
COPY --from=builder /app/tsconfig.json ./
COPY --from=builder /app/start.sh ./

# Create directories for uploaded and processed files
RUN mkdir -p uploads processed && chown -R appuser:appgroup /app

# Switch to the non-root user
USER appuser

EXPOSE 3000

# Run the startup script
CMD ["sh", "start.sh"]

What does this Dockerfile do?

Stage 1 (builder):

• Starts from node:20-alpine, a minimal Node.js image.
• Copies package.json and package-lock.json first and runs npm ci so Docker can cache the dependency layer.
• Copies the Prisma schema and config, then runs npx prisma generate to create the type-safe client.

Stage 2 (runner):

• Creates a non-root user called appuser so the container does not run as root (a security best practice).
• Copies only the files needed at runtime from the builder stage.
• Creates the uploads/ and processed/ directories that the server and worker write to.

• Save Dockerfile.

You will not see a change yet. The Dockerfile is just a build recipe that Docker will use in substep 3.

Seeing a "file not found" error later when building?

• Make sure the Dockerfile is in the root of your project folder (the same level as package.json), not inside src/ or another subdirectory.
• Check that the file is named exactly Dockerfile with a capital D and no file extension.
• Make sure .dockerignore is also in the project root.

Still stuck? Ask for help.

✔️ Awesome, I've got everything!

Great. Make sure you have saved both .dockerignore and Dockerfile.

ⓧ I'd like to double check the full code

Here is the complete .dockerignore:

node_modules
uploads
processed
generated
.env
*.pem
.git

Here is the complete Dockerfile:

FROM node:20-alpine AS builder

WORKDIR /app

COPY package.json package-lock.json ./
RUN npm ci

COPY prisma ./prisma
COPY prisma.config.ts ./
RUN npx prisma generate

COPY . .

FROM node:20-alpine AS runner

WORKDIR /app

RUN addgroup -S appgroup && adduser -S appuser -G appgroup

COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/generated ./generated
COPY --from=builder /app/package.json ./
COPY --from=builder /app/prisma ./prisma
COPY --from=builder /app/prisma.config.ts ./
COPY --from=builder /app/src ./src
COPY --from=builder /app/public ./public
COPY --from=builder /app/tsconfig.json ./
COPY --from=builder /app/start.sh ./

RUN mkdir -p uploads processed && chown -R appuser:appgroup /app

USER appuser

EXPOSE 3000

CMD ["sh", "start.sh"]

Create the production Docker Compose file and startup script

The Dockerfile packages your app into a single image. Now you need a way to start your app, PostgreSQL, and Redis together as a coordinated stack.

You already have docker-compose.dev.yml for local development. Now you will create a production docker-compose.yml that also includes your app container alongside the database and cache.

• First, create a startup script that the Docker container runs when it starts. Create a file called start.sh in the root of your project folder and add the following:

#!/bin/sh
# Run database migrations to ensure the schema is up to date
npx prisma migrate deploy
# Start the background worker process
npx tsx src/worker.ts &
# Start the Express server in the foreground
npx tsx src/server.ts

What does this script do?

• npx prisma migrate deploy applies any pending database migrations. This is the production-safe version of prisma migrate dev.
• The & at the end of the worker command starts it as a background process, so the script can continue to the next line.
• The server runs in the foreground. If the server exits, the container stops.

• Save start.sh.

• Now create a file called docker-compose.yml in the root of your project folder and add the following:

services:
  # Your app container - runs the Express API and BullMQ worker
  app:
    build: .
    ports:
      - "3000:3000"
    environment:
      # Use Docker network hostnames (postgres, redis) instead of localhost
      DATABASE_URL: postgresql://postgres:postgres@postgres:5432/imageprocessor?schema=public
      REDIS_URL: redis://redis:6379
      # Reads HF_TOKEN from the .env file on the host machine
      HF_TOKEN: ${HF_TOKEN}
      PORT: 3000
    depends_on:
      - postgres
      - redis
    restart: unless-stopped

  # PostgreSQL database service
  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: imageprocessor
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: unless-stopped

  # Redis cache and job queue service
  redis:
    image: redis:7-alpine
    volumes:
      - redisdata:/data
    restart: unless-stopped

# Named volumes persist data across container restarts
volumes:
  pgdata:
  redisdata:

How is this different from the dev Compose file?

• The dev file only runs PostgreSQL and Redis. This production file also builds and runs your app as a container.
• The DATABASE_URL uses postgres:5432 instead of localhost:5433. Inside the Docker network, containers reach each other by service name.
• The ${HF_TOKEN} syntax tells Docker Compose to read the value from the .env file on the host machine. Your existing .env already has this value set.
• depends_on ensures PostgreSQL and Redis start before the app container.

• Save docker-compose.yml.

You will not see a visible change yet. The next substep builds the image and runs the full stack.

YAML indentation errors later?

• YAML is whitespace-sensitive. Make sure you use spaces, not tabs.
• Each nested level should be indented by exactly 2 spaces.
• Check that docker-compose.yml is a separate file from docker-compose.dev.yml. Do not overwrite your dev file.

Need help debugging your Compose file?

✔️ Awesome, I've got everything!

Make sure you have saved both start.sh and docker-compose.yml.

ⓧ I'd like to double check the full code

Here is the complete start.sh:

#!/bin/sh
npx prisma migrate deploy
npx tsx src/worker.ts &
npx tsx src/server.ts

Here is the complete docker-compose.yml:

services:
  app:
    build: .
    ports:
      - "3000:3000"
    environment:
      DATABASE_URL: postgresql://postgres:postgres@postgres:5432/imageprocessor?schema=public
      REDIS_URL: redis://redis:6379
      HF_TOKEN: ${HF_TOKEN}
      PORT: 3000
    depends_on:
      - postgres
      - redis
    restart: unless-stopped

  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: imageprocessor
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: unless-stopped

  redis:
    image: redis:7-alpine
    volumes:
      - redisdata:/data
    restart: unless-stopped

volumes:
  pgdata:
  redisdata:

Build and test locally

All four Docker files are in place. Now you will build the image, start the full stack inside containers, and verify the pipeline works end to end.

Before starting the production stack, you need to stop the dev containers to avoid port conflicts.

• If your dev server is still running from the previous step, press Ctrl+C in that terminal to stop it.
• Stop the dev Docker containers by running this command:

docker compose -f docker-compose.dev.yml down

• Build the Docker image and start all three services by running this command:

docker compose up --build

What does this command do?

• --build tells Docker Compose to build the image from the Dockerfile before starting the containers.
• Docker Compose automatically creates a shared network so the app, postgres, and redis containers can talk to each other by name.

The first build takes a few minutes because Docker downloads the base images and installs all npm packages. Subsequent builds are much faster thanks to Docker layer caching.

Wait for the terminal to show Server running on http://localhost:3000 and Worker started, waiting for jobs....

• Open your browser and go to http://localhost:3000.
• You should see the Media Processing Pipeline UI with the drag-and-drop upload zone.
• Upload an image by dragging it onto the upload zone or clicking to select a file.
• Watch the status badge cycle from PENDING (yellow) to PROCESSING (blue) to COMPLETED (green).
• Once the image is completed, you should see AI-generated tags appear as small blue pills below the image.
• Click the Generate Caption button on a completed image to test the on-demand AI captioning.
• Click Download WebP to verify the processed image downloads correctly.

Container fails to start or page won't load?

• If you see port is already allocated, your dev containers might still be running. Run docker compose -f docker-compose.dev.yml down first.
• If you see COPY failed: file not found during the build, check that start.sh exists in the project root alongside package.json.
• If the app starts but the database connection fails, make sure the DATABASE_URL in docker-compose.yml uses postgres:5432 (the Docker service name), not localhost:5433 (the dev setting).
• If AI tagging returns empty tags, your HF_TOKEN might not be set. Check that your .env file has a valid token. Docker Compose reads ${HF_TOKEN} from this file.

Still having trouble with your Docker setup?

• Once you have confirmed the pipeline works, stop the containers by pressing Ctrl+C in the terminal.
• Clean up the stopped containers by running:

docker compose down

Your entire pipeline now runs inside Docker containers. Next up, you will deploy this containerized stack to a live AWS EC2 instance so anyone can access it from the internet.

Deploy to AWS EC2

Your containerized pipeline works perfectly on your local machine. The next challenge is making it accessible from anywhere on the internet.

In this step, you will launch a virtual server on AWS and deploy your full Docker Compose stack to it. By the end, anyone with the server's public IP address can upload images and watch them flow through your processing pipeline.

In this step, get ready to:

• Launch an EC2 instance on AWS with a security group configured for web traffic.
• Install Docker on the EC2 instance and connect via SSH.
• Deploy and verify the full pipeline on a live server accessible via public IP.

Create an AWS account and launch an EC2 instance

EC2 (Elastic Compute Cloud) is a virtual server you rent from AWS. You will launch one of these servers and run your Docker containers on it, just like you did on your local machine.

Is this free?

Yes. New AWS accounts get 750 free hours per month of t3.micro instances for 12 months. Since a month has roughly 744 hours, one instance running 24/7 costs $0.

AWS requires a credit card during sign-up for identity verification, but you will not be charged for this project.

• If you do not already have an AWS account, open aws.amazon.com and click Create an AWS Account.
• Follow the sign-up wizard to create your account with an email address, password, and payment method.
• Once signed in to the AWS Management Console, type EC2 in the search bar at the top.
• Click EC2 from the search results to open the EC2 Dashboard.
• Click Launch instance.

You are now on the instance configuration page. There are several sections to fill in.

• In the Name and tags field, enter media-pipeline.
• Under Application and OS Images (Amazon Machine Image), select Ubuntu.
• Make sure the AMI dropdown shows Ubuntu Server 24.04 LTS (Free tier eligible).
• Under Instance type, select t3.micro (Free tier eligible).

Why t3.micro and not t2.micro?

AWS updated its free tier for accounts created after July 2025. The t3.micro instance type is now the free tier default. If you see t2.micro marked as free tier eligible instead, your account predates this change and you can use t2.micro safely.

Next, create a key pair so you can connect to your instance over SSH.

• Under Key pair (login), click Create new key pair.
• Enter media-pipeline-key as the key pair name.
• Leave the key pair type as RSA and the file format as .pem.
• Click Create key pair.

Your browser automatically downloads a media-pipeline-key.pem file. Keep this file safe. You need it to connect to your instance.

Now configure the security group to control which network traffic can reach your instance.

• Under Network settings, click Edit.
• The first rule for SSH on port 22 is already there. Change its Source type to My IP so only your computer can SSH in.
• Click Add security group rule.
• Set Type to HTTP (port 80) and Source type to Anywhere.
• Click Add security group rule again.
• Set Type to Custom TCP, Port range to 3000, and Source type to Anywhere.

What is a security group?

A security group acts as a firewall for your EC2 instance. Each rule you added allows a specific type of traffic through. Port 22 allows SSH so you can connect to the server. Port 3000 allows web traffic to reach your Express app.

Finally, configure the storage.

• Under Configure storage, change the size to 20 GiB. Leave the volume type as gp3.
• Click Launch instance.

AWS starts provisioning your instance. After a few seconds, click View all instances to see it in the dashboard. Wait until the Instance state column shows Running and the Status check column shows 2/2 checks passed.

• Click on your instance's Instance ID link to open its detail page.
• Find the Public IPv4 address field and copy the IP address. You will need this throughout the rest of this step.

Instance stuck in Pending?

It can take 1-2 minutes for a new instance to fully start. Click the refresh icon in the Instances table to update the status. If it stays in Pending for more than 5 minutes, terminate it and launch a new one.

Help me troubleshoot my EC2 instance launch.

Connect to EC2 and install Docker

Your EC2 instance is a fresh Ubuntu server with no software installed. You need to connect to it via SSH, then install Docker so it can run your containers.

First, set the correct permissions on your key file so SSH accepts it.

macOS

• Open Terminal by pressing Cmd+Space, typing Terminal, and pressing Enter.
• Restrict the key file permissions by running this command:

chmod 400 ~/Downloads/media-pipeline-key.pem

Windows

• Open PowerShell by pressing the Windows key, typing PowerShell, and pressing Enter.
• Restrict the key file permissions by running these commands:

icacls "$HOME\Downloads\media-pipeline-key.pem" /inheritance:r
icacls "$HOME\Downloads\media-pipeline-key.pem" /grant:r "$($env:USERNAME):(R)"

Why do I need to change the key file permissions?

SSH refuses to use a private key file that other users on your computer can read. Setting restrictive permissions proves to SSH that only you can access the key.

• Connect to your EC2 instance by running this command:

ssh -i ~/Downloads/media-pipeline-key.pem ubuntu@[[EC2_PUBLIC_IP="your EC2 public IPv4 address"]]

• SSH will ask if you want to continue connecting to an unknown host. Type yes and press Enter.

You should see a welcome message from Ubuntu and a command prompt that starts with ubuntu@. You are now running commands directly on your EC2 instance.

Connection timed out or refused?

Make sure the security group rule for SSH (port 22) has My IP selected as the source. If your IP has changed since you created the rule (common on home Wi-Fi), update it in the EC2 console under Security Groups.

Also confirm the instance state is Running and that you are using the correct public IP address from the instance detail page.

Help me troubleshoot my SSH connection to EC2.

• Update the system packages by running this command:

sudo apt-get update && sudo apt-get upgrade -y

• Install Docker and the Docker Compose plugin by running this command:

sudo apt-get install -y docker.io docker-compose-v2

• Enable Docker to start automatically when the server boots by running:

sudo systemctl enable docker

• Add the ubuntu user to the docker group so you can run Docker commands without sudo:

sudo usermod -aG docker ubuntu

• Apply the group change by logging out and back in. Run:

exit

• Reconnect to your instance by running the same SSH command:

ssh -i ~/Downloads/media-pipeline-key.pem ubuntu@[[EC2_PUBLIC_IP="your EC2 public IPv4 address"]]

• Verify Docker is installed and accessible by running:

docker --version

You should see a Docker version number in the output. If you see permission denied instead, make sure you ran the exit and SSH commands above to apply the group change.

Getting permission denied when running docker?

The usermod command only takes effect after you log out and back in. Make sure you ran exit first, then reconnected with the SSH command.

Help me fix Docker permissions on my EC2 instance.

Deploy the pipeline to your EC2 instance

Docker is ready on your server. Now you need to get your project files onto the EC2 instance and start the containers.

You will create a compressed archive of your project on your local machine, copy it to EC2, then extract and run it.

• Open a new terminal window on your local machine. Do not close the SSH session.
• Navigate to your project directory on your local machine by running:

cd ~/Desktop/media-pipeline

• Create a compressed archive of your project (excluding files that Docker will rebuild) by running:

tar --exclude='node_modules' --exclude='uploads' --exclude='processed' --exclude='generated' --exclude='.git' --exclude='.env' -czf ../media-pipeline.tar.gz .

What does this command do?

The tar command bundles all your project files into a single compressed .tar.gz file. The --exclude flags skip directories that Docker rebuilds inside the container (like node_modules and generated). The local .env is also excluded because you will create a separate one on EC2.

• Copy the archive to your EC2 instance by running:

scp -i ~/Downloads/media-pipeline-key.pem ../media-pipeline.tar.gz ubuntu@[[EC2_PUBLIC_IP="your EC2 public IPv4 address"]]:~/

Wait for the upload to finish. The file is small (a few hundred KB) so it should complete in seconds.

• Switch back to your SSH terminal session (the one connected to EC2).
• Create a project directory and extract the archive by running these commands:

mkdir ~/media-pipeline && cd ~/media-pipeline
tar -xzf ~/media-pipeline.tar.gz

• Confirm the files extracted correctly by running:

ls

You should see Dockerfile, docker-compose.yml, package.json, src/, public/, prisma/, and start.sh listed in the output.

Your production docker-compose.yml already has the correct DATABASE_URL and REDIS_URL hardcoded for the Docker network. The only value it reads from the host is HF_TOKEN.

• Create the .env file on EC2 with your Hugging Face token by running:

echo 'HF_TOKEN=your-hugging-face-token-here' > .env

• Replace your-hugging-face-token-here with the actual token from your Hugging Face account at huggingface.co/settings/tokens. This is the same token you used in your local .env file.

Why does the EC2 .env only need HF_TOKEN?

Your local .env had DATABASE_URL and REDIS_URL pointing to localhost for development. In production, docker-compose.yml sets those values directly using Docker's internal network names (postgres and redis). Only HF_TOKEN uses the ${HF_TOKEN} interpolation syntax, which Docker Compose reads from the host .env file.

• Build and start the entire pipeline in detached mode by running:

docker compose up -d --build

What does this command do?

• --build tells Docker to build the image from your Dockerfile before starting. This installs all npm dependencies and generates the Prisma client inside the container.
• -d runs the containers in detached mode so they keep running after you close your SSH session.

The build takes a few minutes on the first run because Docker needs to download base images and install all npm packages. You will see progress output in the terminal.

• Once the build finishes, verify all three containers are running by running:

docker compose ps

You should see three services listed: app, postgres, and redis, all with a status of Up.

App container keeps restarting?

Check the app container logs for the error message by running docker compose logs app. Common causes include a missing or empty .env file, or the PostgreSQL container not being ready when the app tries to run the migration.

If the logs mention a database connection error, wait 30 seconds and run docker compose restart app. PostgreSQL sometimes needs extra time to initialize on the first start.

Help me debug my Docker Compose deployment on EC2.

Now for the moment of truth. Open your browser and visit your live application.

• Open a new browser tab and navigate to:

http://[[EC2_PUBLIC_IP="your EC2 public IPv4 address"]]:3000

You should see the Media Processing Pipeline UI with the drag-and-drop upload zone.

• Upload an image by dragging it onto the upload zone or clicking to select a file.
• Watch the image card appear with a PENDING status that transitions to PROCESSING and then COMPLETED.
• Verify that AI-generated tags appear as colored pills below the image once processing completes.
• Click the Generate Caption button and wait for the AI caption to appear.
• Click Download WebP to confirm the processed image downloads to your computer.

Page doesn't load in the browser?

Make sure you are using http:// (not https://) in the URL. Your EC2 instance does not have an SSL certificate configured.

Double-check that the security group has a rule allowing Custom TCP on port 3000 from Anywhere. You can verify this in the EC2 console under your instance's Security tab.

Help me troubleshoot why my app is not loading on EC2.

AI tags taking a while?

The first request to the Hugging Face Inference API can take 20-30 seconds because the model needs to load. Subsequent requests are much faster. If tags appear empty, check your HF_TOKEN is correct in the .env file on EC2.

Secret mission

Add Real-Time Job Monitoring with Bull Board

Your pipeline processes jobs in the background, but you have no visibility into what's happening inside the queue. Add a real-time monitoring dashboard that lets you watch jobs flow, inspect failures, and retry them with a single click.

Before you go

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project uses an AWS EC2 instance covered by the AWS Free Tier (750 free hours/month for 12 months) and the free Hugging Face Inference API with rate limits.

Cost warning

Your t3.micro EC2 instance is free for 12 months after account creation (750 hours/month). After the free tier expires, leaving the instance running will incur charges.

If you are not planning to use this project again, choose Delete below to avoid any risk of unexpected charges.

Resources you used:

• AWS EC2 instance (t3.micro, Ubuntu 24.04).
• EC2 key pair (.pem file stored locally).
• EC2 security group (SSH, HTTP, port 3000).
• Docker containers on EC2 (Express app, PostgreSQL, Redis, worker).
• Hugging Face account and API token.
• Local project files (source code, uploads, processed images).

Keep everything running

No action needed. Choose this if you are still actively building or want the pipeline accessible via your EC2 public IP.

• Your t3.micro instance is covered by 750 free hours/month. Since a month has roughly 744 hours, one instance running 24/7 costs $0.
• Monitor your usage in the AWS Billing Dashboard to make sure you stay within the free tier.
• The Hugging Face Inference API remains free with rate limits per 5-minute window.
• Set a calendar reminder for 12 months from today so you do not forget about the free tier expiration.

Pause - I'll come back to this later

Stop the EC2 instance to save compute hours while keeping your files and configuration intact.

• Open the EC2 Dashboard in the AWS Console.
• Select your instance, click Instance state, then Stop instance.
• EBS storage charges still apply while the instance is stopped, but 20 GiB gp3 is within the 30 GiB free tier limit.
• When you are ready to resume, select the instance and click Instance state, then Start instance. Your public IP address will change, so use the new IP to access your pipeline.
• After restarting, SSH into the instance and run docker compose up -d in your project directory to bring the containers back up.

Delete - I don't want to use this again

Remove all AWS resources to ensure no charges after the free tier expires.

• Open the EC2 Dashboard in the AWS Console.
• Select your instance, click Instance state, then Terminate instance. This deletes the instance and its attached storage.
• Navigate to Key Pairs in the EC2 sidebar and delete the key pair you created for this project.
• Delete the .pem file from your local machine.
• Optionally delete the local project folder from your Desktop.
• Your Hugging Face account and token are free to keep. Revoke the token at huggingface.co/settings/tokens if you no longer need it.
• Set a calendar reminder for 12 months from today so you do not forget about the free tier expiration on your AWS account.

Mission accomplished

That's a Wrap!

Nice work. You just designed and deployed a full asynchronous media processing pipeline with AI capabilities from scratch.

Here's what you learned:

• Designed a producer/worker architecture with Express, BullMQ, and Redis that decouples API response times from CPU-intensive image processing and AI inference.
• Integrated Hugging Face AI models for automatic image classification and on-demand caption generation, all running inside a containerized background worker pipeline.
• Containerized the full multi-service stack (API, workers, PostgreSQL, Redis) with Docker Compose and deployed it to a live AWS EC2 instance accessible via public IP.
• Added a Bull Board dashboard for real-time job monitoring, failure inspection, and manual retries.

Ready to quiz yourself?