Dockerize a Node.js Job Queue

Build a scalable Docker Compose job queue with Node.js, Redis, and workers.

Introduction

30 Second Summary

A background task can appear healthy until a restart wipes its history or interrupts the work. Adding more workers raises another question: which one handled each job?

In this project, you will build a local job queue that sends text from an Express API through Redis to background workers. You will use Docker Compose to operate the container stack through recreation and worker scaling.

What You'll Build

You will submit text in a browser, watch separate worker containers turn it into uppercase, and see the completed results survive a full stack recreation.

By the end of this project, you'll have:

  • A visible job flow where you submit text through the dashboard before a worker returns the uppercase result.
  • A set of persistent completed jobs that remain available after you remove and recreate every container.
  • A scaled worker pool with three replicas whose service-prefixed logs show how jobs spread across the queue.
  • Secret Mission: Add a SIGTERM-aware shutdown path that lets a worker finish its in-flight job before it closes Redis.

Are there any prerequisites?

You need a Linux machine with Docker Engine, Docker Compose, and Neovim available. Basic Node.js and Express familiarity helps, but you do not need a host Node.js installation or an external account.

Before We Start

The hands-on work makes more sense once each service has one clear responsibility. An Express API accepts jobs. Redis stores pending and completed jobs. Independently replaceable worker containers process them.

Verify the Container Toolchain

Every later build depends on Docker Engine accepting commands from your terminal. A broken daemon would make later image failures look like application problems.

You will also confirm that Docker Compose is available for coordinating the stack. You will then prepare an empty workspace in Neovim.

In this step, get ready to:
  • Verify the Docker Engine Client and Server connection.
  • Confirm that the Docker Compose plugin is available.
  • Create the empty project directories in Neovim.
Verify Docker Engine

The Docker command-line client sends requests to the Docker Engine server. Seeing both sides in the version output proves that the client can reach the running daemon.

  • Open a terminal from your Linux application launcher.
  • Check the Docker Engine connection by running this command:
docker version

What does this command prove?

  • The Client section describes the Docker command-line tool on your machine.
  • The Server section confirms that the tool reached the Docker Engine daemon.

You should see separate Client and Server sections. That clears the first source of ambiguity before you build anything.

Missing the Server section?

  • Start Docker Engine with the service manager provided by your Linux installation.
  • Retry the check after the daemon finishes starting.
  • Confirm that your Linux account has permission to communicate with Docker Engine.

If the daemon still does not respond, help me diagnose my Docker Engine connection.

Confirm the Docker Compose plugin

Docker Compose will define the API, worker, and Redis as one stack later. This check confirms that its plugin is ready before any Compose configuration exists.

  • Check the Compose plugin in the terminal from the previous substep by running this command:
docker compose version

What does this command prove?

The command asks the Docker CLI for its Compose plugin version. A version response confirms that later Compose commands can be recognized.

You should see a Docker Compose version number. Your local container control tools are now responding.

No Compose version?

  • Confirm that the Docker Compose plugin is enabled in the Docker installation you prepared before this project.
  • Restart your terminal after correcting the plugin setup.

If the plugin remains unavailable, help me troubleshoot my Docker Compose plugin.

Create the empty project workspace

A predictable directory structure keeps the application source separate from browser files. The docker-job-queue directory will contain empty src/ and public/ directories for now.

  • Create the three directories on your Desktop by running these commands in the same terminal:
cd ~/Desktop
mkdir -p docker-job-queue/src docker-job-queue/public
ls -R docker-job-queue

What do these commands do?

  • The first command moves the terminal to your Desktop so the project is easy to find.
  • The second command creates docker-job-queue with its two empty subdirectories.
  • The final command lists the new structure recursively so you can verify it immediately.

You should see headings for docker-job-queue/public and docker-job-queue/src. Neither directory should contain a file.

Directories missing from the output?

  • Check that the terminal moved to your Desktop before creating the directories.
  • Check the spelling of docker-job-queue, src, and public.

If the structure still differs, help me fix my project directories.

Before you launch Neovim, which two empty directories do you expect to see inside docker-job-queue?

  • Open the new project in Neovim by running these commands:
cd docker-job-queue
nvim .

What do these commands do?

  • The first command enters the docker-job-queue directory from your Desktop.
  • The second command starts Neovim with the current directory as its project location.

You should see Neovim open a directory view for docker-job-queue containing public/ and src/. The workspace contains no application files yet.

Neovim not showing the folders?

  • Exit Neovim before checking that your terminal is inside the docker-job-queue directory.
  • Launch Neovim again with the current directory argument.

If Neovim opens somewhere else, help me open the correct project directory.

Your container tools are verified. Next, you will turn this empty workspace into a visible Express container.

Package the Express Container

Your Docker Engine is responding. Your empty project is also ready in Neovim. The next proof must come from running the application inside a container.

A Docker image packages the Node.js runtime with your pinned Express dependencies. You will describe that image with a Dockerfile. You will then prove dependency installation, port publishing, and non-root execution with a visible smoke test.

In this step, get ready to:
  • Prepare the application files used by the image build.
  • Package the application as a non-root container image.
  • Run the smoke-test server through a published port.
Define the application and build context

The image needs a dependency manifest before it can install the application packages. A pinned JSON manifest makes every build request the same dependency versions.

  • Create package.json at the top level of the open docker-job-queue directory with Neovim's :edit command.
  • Paste the following dependency manifest into the new buffer:
{
  "name": "docker-job-queue",
  "version": "1.0.0",
  "private": true,
  "type": "module",
  "dependencies": {
    "express": "5.3.0",
    "redis": "6.3.1"
  }
}

What does this manifest define?

  • The name field identifies the application as docker-job-queue.
  • The type field enables the module syntax used by the JavaScript files.
  • Express 5.3.0 provides the smoke-test web server.
  • The Redis client at version 6.3.1 prepares this same image for the queue services added later.
  • Save package.json with Neovim's :write command.
  • Confirm Neovim reports that package.json was written successfully.

Manifest not saving?

Check that every property name and string has matching double quotes. Confirm the final dependency entry has no trailing comma.

Save the current buffer before using :edit to create the next file.

Help me check my package.json file.

A Docker build sends the project directory to the builder as a build context. The .dockerignore file keeps generated dependencies, debug logs, and Git data out of that context.

  • Create .dockerignore at the top level of docker-job-queue with Neovim's :edit command.
  • Paste the following exclusions into the new buffer:
node_modules
npm-debug.log
.git

What does this ignore file do?

  • The node_modules entry prevents host-installed packages from entering the image build.
  • The npm-debug.log entry excludes a generated debug log.
  • The .git entry keeps repository history out of the build context.
  • Save .dockerignore with Neovim's :write command.
  • Confirm Neovim reports that .dockerignore was written successfully.

Ignore file missing?

Confirm the filename starts with a dot. Check that Neovim created it beside package.json.

Help me find my .dockerignore file.

The smoke test needs one route with an observable response. Its startup log provides a terminal-only signal that the application reached the listening state.

  • Create src/starter.js inside the existing src directory with Neovim's :edit command.
  • Paste the following smoke-test server into the new buffer:
import express from "express";

const app = express();

app.get("/", (request, response) => {
  response.send("Express is running inside Docker");
});

app.listen(3000, "0.0.0.0", () => {
  console.log(JSON.stringify({ service: "starter", event: "starter_listening", port: 3000 }));
});

What does this server do?

  • The app value holds the Express application.
  • The / route returns the smoke-test message to a browser request.
  • The server listens on port 3000 across the container's network interfaces.
  • The starter_listening event proves the server reached its listening callback.
  • Save src/starter.js with Neovim's :write command.
  • Confirm Neovim reports that src/starter.js was written successfully.

Starter file not saving?

Confirm the existing src directory appears in the file path. Check every opening brace and parenthesis for a matching closing character.

Help me check my starter server.

Package the non-root image

A Dockerfile builds the runtime as a sequence of image layers. Copying package.json before the source lets Docker reuse the dependency layer when only application code changes.

The final process runs as the image's existing node user. This creates a non-root container without adding a custom operating system account.

Why use one build stage?

This application has no compilation stage. A single stage keeps the build focused because there are no compiler tools or generated artifacts to remove.

  • Create Dockerfile at the top level of docker-job-queue with Neovim's :edit command.
  • Paste the following image definition into the new buffer:
# syntax=docker/dockerfile:1
FROM node:26.13.0-alpine3.24

WORKDIR /app

COPY package.json ./
RUN npm install

COPY --chown=node:node src ./src
COPY --chown=node:node public ./public

USER node
EXPOSE 3000
CMD ["node", "src/api.js"]

How is the image assembled?

  • The node:26.13.0-alpine3.24 base provides the pinned Node.js runtime.
  • The WORKDIR instruction gives the application a consistent directory inside the image.
  • The dependency layer runs npm install before either source directory is copied.
  • The --chown=node:node option gives the non-root user ownership of the copied application files.
  • The USER node instruction sets the account used by the application process.
  • The EXPOSE 3000 instruction documents the application port.
  • The default command points to src/api.js. The smoke-test command overrides that default with the existing src/starter.js file.
  • Save Dockerfile with Neovim's :write command.
  • Confirm Neovim reports that Dockerfile was written successfully.

Dockerfile not saving?

Confirm the filename is exactly Dockerfile with no extension. Check that it sits beside package.json.

Help me check my Dockerfile.

Build and run the smoke test

The saved files now contain everything required for the first image build. The build also checks that the dependency manifest and Dockerfile can be processed together.

The first build can take a few minutes while Docker downloads the base image. A longer pause during that download is expected.

  • Build the docker-job-queue image from the open project directory by running this command:
docker build -t docker-job-queue .

What does this build command do?

  • The docker build command processes the Dockerfile in the current directory.
  • The -t docker-job-queue option assigns a reusable name to the finished image.
  • The final . selects the current directory as the build context.
  • Confirm the build finishes successfully and returns control to the terminal.

That is the image built. Your pinned runtime, application dependencies, and source files now share one reusable package.

Image build failing?

If the build cannot contact Docker Engine, confirm the verified daemon is still running. If dependency installation fails, compare package.json with the manifest above.

If a copy instruction fails, confirm the existing src and public directories are still inside docker-job-queue.

Help me debug my Docker build.

Port publishing connects a host port to a port inside the container. The next command keeps the terminal attached so you can see the application log immediately.

Before you start the container, which two signals do you expect to prove that the server works?

  • Start the smoke-test container with port 3000 published by running this command:
docker run --rm --name docker-job-queue-api -p 3000:3000 docker-job-queue node src/starter.js

What does this run command do?

  • The --rm option removes the stopped smoke-test container automatically.
  • The --name docker-job-queue-api option gives the temporary container a readable name.
  • The -p 3000:3000 option publishes the container port through the matching host port.
  • The final arguments override the image's default command with node src/starter.js.

You will see a JSON log containing starter_listening. This proves the application reached its listening callback inside the container.

  • Enter http://localhost:3000 in a browser address bar if a graphical browser is available.

You will see “Express is running inside Docker”. The page proves that the published host port reaches the Express route inside the container.

Page not loading?

Confirm the terminal still shows the running container. Another local process may already be using host port 3000.

If the terminal exits immediately, compare src/starter.js with the smoke-test server above. The terminal output identifies whether startup or dependency loading failed.

Help me debug the smoke-test container.

  • Return to the terminal running the smoke-test container.
  • Stop the container by pressing Ctrl+C.

You will return to the terminal prompt. The temporary container is removed automatically while the reusable docker-job-queue image remains available.

✔️ Awesome, I've got everything!

Great work. Confirm all four files are saved and the smoke-test container has stopped.

ⓧ I'd like to double check the full code

The files below show the exact end state for this step.

{
  "name": "docker-job-queue",
  "version": "1.0.0",
  "private": true,
  "type": "module",
  "dependencies": {
    "express": "5.3.0",
    "redis": "6.3.1"
  }
}

node_modules
npm-debug.log
.git

# syntax=docker/dockerfile:1
FROM node:26.13.0-alpine3.24

WORKDIR /app

COPY package.json ./
RUN npm install

COPY --chown=node:node src ./src
COPY --chown=node:node public ./public

USER node
EXPOSE 3000
CMD ["node", "src/api.js"]

import express from "express";

const app = express();

app.get("/", (request, response) => {
  response.send("Express is running inside Docker");
});

app.listen(3000, "0.0.0.0", () => {
  console.log(JSON.stringify({ service: "starter", event: "starter_listening", port: 3000 }));
});

Your reusable image now serves an Express route through Docker. Next, you will reuse that image to connect an API, a worker, and Redis as one job queue stack.

Build the Job Queue Stack

Your starter Docker image proves that Express can serve a page inside a container. A job queue needs separate processes that can accept work and process it independently.

In this step, you will connect an API and worker through Redis. You will run all three services with Docker Compose. You will also recreate the stack to expose what happens to state stored inside a container.

In this step, get ready to:
  • Build the job-processing flow across the API and worker.
  • Launch all three services through Docker Compose.
  • Recreate the stack to test completed-job history.
Create the API and worker services

The API owns HTTP requests. It stores new jobs in jobs:pending and reads processed jobs from jobs:completed.

  • Create src/api.js in the existing src directory from Neovim.
  • Paste the Redis connection and Express middleware into src/api.js.
import express from "express";
import { createClient } from "redis";

const redis = createClient({ url: "redis://redis:6379" });

redis.on("error", (error) => {
  console.error(JSON.stringify({
    service: "api",
    event: "redis_error",
    message: error.message
  }));
});

await redis.connect();

const app = express();

app.use(express.json());
app.use(express.static("public"));

How does the API connect?

  • The Redis client uses redis as the hostname because Compose services can discover each other by service name.
  • The error listener writes structured JSON when the Redis connection has a problem.
  • The Express middleware accepts JSON request bodies and serves files from public.
  • Add the health route below app.use(express.static("public"));.
app.get("/health", async (request, response) => {
  try {
    const reply = await redis.PING();
    response.status(reply === "PONG" ? 200 : 503).json({
      status: reply === "PONG" ? "ready" : "not_ready",
      redis: reply
    });
  } catch (error) {
    response.status(503).json({ status: "not_ready", redis: error.message });
  }
});

What does the health route prove?

The /health route sends a Redis ping. The response shows whether the API can reach the queue service.

  • Add the job-submission route below the /health route.
app.post("/jobs", async (request, response) => {
  const text = typeof request.body?.text === "string"
    ? request.body.text.trim()
    : "";

  if (!text) {
    return response.status(400).json({ error: "text is required" });
  }

  const job = {
    id: `${Date.now()}-${Math.random().toString(16).slice(2)}`,
    text,
    createdAt: new Date().toISOString()
  };

  await redis.LPUSH("jobs:pending", JSON.stringify(job));
  console.log(JSON.stringify({ service: "api", event: "job_queued", jobId: job.id }));

  return response.status(202).json(job);
});

How does a job enter the queue?

  • The route rejects empty text before creating a job.
  • Each accepted job receives an ID and creation time.
  • The API serializes the job as JSON before pushing it into jobs:pending.
  • Add the completed-jobs route below the /jobs submission route.
  • Add the server listener at the bottom of src/api.js.
app.get("/jobs/completed", async (request, response) => {
  const storedJobs = await redis.LRANGE("jobs:completed", 0, 19);
  response.json(storedJobs.map((storedJob) => JSON.parse(storedJob)));
});

app.listen(3000, "0.0.0.0", () => {
  console.log(JSON.stringify({ service: "api", event: "api_listening", port: 3000 }));
});

How does the API expose results?

The completed-jobs route reads up to 20 results from Redis. The listener exposes the API on port 3000 inside the container.

  • Save src/api.js.

Seeing a JavaScript syntax problem?

Check that every route ends with matching braces and parentheses. Confirm that the final listener remains at the bottom of src/api.js.

Compare the spelling of jobs:pending and jobs:completed with the snippets above.

Help me check my API file.

The worker waits for jobs in the pending list. Each job takes three seconds to process before its uppercase result moves into the completed list.

  • Create src/worker.js in the existing src directory from Neovim.
  • Paste the Redis connection and readiness log into src/worker.js.
import { createClient } from "redis";

const redis = createClient({ url: "redis://redis:6379" });

redis.on("error", (error) => {
  console.error(JSON.stringify({
    service: "worker",
    event: "redis_error",
    message: error.message
  }));
});

await redis.connect();
console.log(JSON.stringify({ service: "worker", event: "worker_ready" }));

Why does the worker use the same connection?

The API and worker share the same Redis service. Their separate clients let each process perform its own role without sharing application memory.

  • Add the processing loop below the worker_ready log.
for (;;) {
  const queuedJob = await redis.BRPOP("jobs:pending", 1);

  if (!queuedJob) {
    continue;
  }

  const job = JSON.parse(queuedJob.element);
  console.log(JSON.stringify({ service: "worker", event: "job_started", jobId: job.id }));

  await new Promise((resolve) => setTimeout(resolve, 3000));

  const completedJob = {
    ...job,
    result: job.text.toUpperCase(),
    completedAt: new Date().toISOString()
  };

  await redis.LPUSH("jobs:completed", JSON.stringify(completedJob));
  console.log(JSON.stringify({ service: "worker", event: "job_completed", jobId: job.id }));
}

How does the worker process jobs?

  • The blocking pop waits for an item in jobs:pending.
  • The delay makes active processing visible in the logs.
  • The worker converts the original text to uppercase before storing the completed job.
  • Save src/worker.js.

Worker file not matching?

Confirm that queuedJob.element supplies the JSON passed to JSON.parse(). Check that the completed result uses job.text.toUpperCase().

Help me check my worker file.

Add the dashboard and Compose services

The dashboard turns queue activity into a visible loop. It submits jobs to the API and refreshes the completed list once per second.

  • Create public/index.html in the existing public directory from Neovim.
  • Paste the page structure into public/index.html.
<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>Docker Job Queue</title>
  <style>
  </style>
</head>
<body>
  <main>
    <h1>Docker Job Queue</h1>
    <p>Submit text to the API. A separate worker container converts it to uppercase.</p>

    <form id="job-form">
      <input id="job-text" name="text" placeholder="hello containers" required>
      <button type="submit">Queue job</button>
    </form>

    <p id="status"></p>
    <h2>Completed jobs</h2>
    <ul id="completed-jobs"></ul>
  </main>

  <script>
  </script>
</body>
</html>

What does the page structure provide?

The form supplies the job text. The status area shows the queued job ID. The list provides a destination for completed results.

  • Add the first style group inside the empty style element.
    body {
      max-width: 760px;
      margin: 3rem auto;
      padding: 0 1rem;
      font-family: system-ui, sans-serif;
      color: #172033;
      background: #f4f7fb;
    }

    main {
      padding: 2rem;
      border-radius: 14px;
      background: white;
      box-shadow: 0 12px 30px rgba(23, 32, 51, 0.1);
    }

    form {
      display: flex;
      gap: 0.75rem;
    }

What does the first style group change?

These selectors center the dashboard and place its content on a white panel. The form uses a horizontal layout for its input and button.

  • Add the remaining styles below the form selector.
    input {
      flex: 1;
      padding: 0.75rem;
    }

    button {
      padding: 0.75rem 1rem;
      cursor: pointer;
    }

    li {
      margin: 0.75rem 0;
      padding: 0.75rem;
      border-radius: 8px;
      background: #eef3ff;
    }

    #status {
      min-height: 1.5rem;
      color: #3757a6;
    }

What does the second style group change?

The input expands across the form. Completed jobs appear as separate list cards. The reserved status height keeps the layout steady when a job ID appears.

  • Add the element references and refresh function inside the empty script element.
    const form = document.querySelector("#job-form");
    const input = document.querySelector("#job-text");
    const status = document.querySelector("#status");
    const completedJobs = document.querySelector("#completed-jobs");

    async function refreshCompletedJobs() {
      const response = await fetch("/jobs/completed");
      const jobs = await response.json();

      completedJobs.replaceChildren(
        ...jobs.map((job) => {
          const item = document.createElement("li");
          item.textContent = `${job.text} -> ${job.result}`;
          return item;
        })
      );
    }

How does the completed list refresh?

The function requests completed jobs from the API. It replaces the current list with an item for every returned result.

  • Add the form handler below refreshCompletedJobs().
    form.addEventListener("submit", async (event) => {
      event.preventDefault();
      status.textContent = "Queueing job...";

      const response = await fetch("/jobs", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({ text: input.value })
      });

      if (!response.ok) {
        const error = await response.json();
        status.textContent = error.error ?? "Job could not be queued";
        return;
      }

      const job = await response.json();
      status.textContent = `Queued ${job.id}`;
      input.value = "";
    });

    refreshCompletedJobs();
    setInterval(refreshCompletedJobs, 1000);

How does the form submit work?

The handler sends the input text as JSON. A successful response displays the queued job ID and clears the input.

The final timer refreshes completed jobs every second. This reveals the worker result without a manual page refresh.

  • Save public/index.html.

Dashboard markup looking incomplete?

Confirm that the CSS remains inside the style element. Confirm that the JavaScript remains inside the script element.

Check that the IDs in the form match the selectors used by the script.

Help me check my dashboard file.

Compose gives the image two application roles. The API uses the image's API command while the worker overrides that command with node src/worker.js.

  • Create compose.yaml beside the existing Dockerfile from Neovim.
  • Paste the three-service configuration into compose.yaml.
services:
  api:
    build: .
    command: ["node", "src/api.js"]
    ports:
      - "3000:3000"
    depends_on:
      - redis

  worker:
    build: .
    command: ["node", "src/worker.js"]
    depends_on:
      - redis

  redis:
    image: redis:7.4.12-alpine

How does this stack fit together?

  • The API and worker build from the same Dockerfile.
  • The API publishes port 3000 to the host.
  • The worker receives its own command without publishing a port.
  • The short dependency list starts Redis before the application services. It does not wait for Redis to become healthy.
  • Save compose.yaml.

✔️ Awesome, I've got everything!

Your API, worker, dashboard, and naive Compose configuration are ready. Save every file before starting the stack.

ⓧ I'd like to double check the full code

Compare each file with the complete project state below.

node_modules
npm-debug.log
.git

What to compare in .dockerignore

Confirm that dependencies, npm debug logs, and Git metadata stay outside the build context.

# syntax=docker/dockerfile:1
FROM node:26.13.0-alpine3.24

WORKDIR /app

COPY package.json ./
RUN npm install

COPY --chown=node:node src ./src
COPY --chown=node:node public ./public

USER node
EXPOSE 3000
CMD ["node", "src/api.js"]

What to compare in the Dockerfile

Confirm that source files belong to the non-root user and that the default command starts the API.

services:
  api:
    build: .
    command: ["node", "src/api.js"]
    ports:
      - "3000:3000"
    depends_on:
      - redis

  worker:
    build: .
    command: ["node", "src/worker.js"]
    depends_on:
      - redis

  redis:
    image: redis:7.4.12-alpine

What to compare in compose.yaml

Confirm that all three services are under services. The Redis service has no volume or health check in this version.

{
  "name": "docker-job-queue",
  "version": "1.0.0",
  "private": true,
  "type": "module",
  "dependencies": {
    "express": "5.3.0",
    "redis": "6.3.1"
  }
}

What to compare in package.json

Confirm that the project uses ES modules with the pinned Express and Redis dependencies.

<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>Docker Job Queue</title>
  <style>
    body {
      max-width: 760px;
      margin: 3rem auto;
      padding: 0 1rem;
      font-family: system-ui, sans-serif;
      color: #172033;
      background: #f4f7fb;
    }

    main {
      padding: 2rem;
      border-radius: 14px;
      background: white;
      box-shadow: 0 12px 30px rgba(23, 32, 51, 0.1);
    }

    form {
      display: flex;
      gap: 0.75rem;
    }

    input {
      flex: 1;
      padding: 0.75rem;
    }

    button {
      padding: 0.75rem 1rem;
      cursor: pointer;
    }

    li {
      margin: 0.75rem 0;
      padding: 0.75rem;
      border-radius: 8px;
      background: #eef3ff;
    }

    #status {
      min-height: 1.5rem;
      color: #3757a6;
    }
  </style>
</head>
<body>
  <main>
    <h1>Docker Job Queue</h1>
    <p>Submit text to the API. A separate worker container converts it to uppercase.</p>

    <form id="job-form">
      <input id="job-text" name="text" placeholder="hello containers" required>
      <button type="submit">Queue job</button>
    </form>

    <p id="status"></p>
    <h2>Completed jobs</h2>
    <ul id="completed-jobs"></ul>
  </main>

  <script>
    const form = document.querySelector("#job-form");
    const input = document.querySelector("#job-text");
    const status = document.querySelector("#status");
    const completedJobs = document.querySelector("#completed-jobs");

    async function refreshCompletedJobs() {
      const response = await fetch("/jobs/completed");
      const jobs = await response.json();

      completedJobs.replaceChildren(
        ...jobs.map((job) => {
          const item = document.createElement("li");
          item.textContent = `${job.text} -> ${job.result}`;
          return item;
        })
      );
    }

    form.addEventListener("submit", async (event) => {
      event.preventDefault();
      status.textContent = "Queueing job...";

      const response = await fetch("/jobs", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({ text: input.value })
      });

      if (!response.ok) {
        const error = await response.json();
        status.textContent = error.error ?? "Job could not be queued";
        return;
      }

      const job = await response.json();
      status.textContent = `Queued ${job.id}`;
      input.value = "";
    });

    refreshCompletedJobs();
    setInterval(refreshCompletedJobs, 1000);
  </script>
</body>
</html>

What to compare in the dashboard

Confirm that the form IDs match the JavaScript selectors. Check that the two API paths begin with /jobs.

import express from "express";
import { createClient } from "redis";

const redis = createClient({ url: "redis://redis:6379" });

redis.on("error", (error) => {
  console.error(JSON.stringify({
    service: "api",
    event: "redis_error",
    message: error.message
  }));
});

await redis.connect();

const app = express();

app.use(express.json());
app.use(express.static("public"));

app.get("/health", async (request, response) => {
  try {
    const reply = await redis.PING();
    response.status(reply === "PONG" ? 200 : 503).json({
      status: reply === "PONG" ? "ready" : "not_ready",
      redis: reply
    });
  } catch (error) {
    response.status(503).json({ status: "not_ready", redis: error.message });
  }
});

app.post("/jobs", async (request, response) => {
  const text = typeof request.body?.text === "string"
    ? request.body.text.trim()
    : "";

  if (!text) {
    return response.status(400).json({ error: "text is required" });
  }

  const job = {
    id: `${Date.now()}-${Math.random().toString(16).slice(2)}`,
    text,
    createdAt: new Date().toISOString()
  };

  await redis.LPUSH("jobs:pending", JSON.stringify(job));
  console.log(JSON.stringify({ service: "api", event: "job_queued", jobId: job.id }));

  return response.status(202).json(job);
});

app.get("/jobs/completed", async (request, response) => {
  const storedJobs = await redis.LRANGE("jobs:completed", 0, 19);
  response.json(storedJobs.map((storedJob) => JSON.parse(storedJob)));
});

app.listen(3000, "0.0.0.0", () => {
  console.log(JSON.stringify({ service: "api", event: "api_listening", port: 3000 }));
});

What to compare in the API

Confirm that the queue keys match the worker. Check that the API listens on 0.0.0.0 at port 3000.

import express from "express";

const app = express();

app.get("/", (request, response) => {
  response.send("Express is running inside Docker");
});

app.listen(3000, "0.0.0.0", () => {
  console.log(JSON.stringify({ service: "starter", event: "starter_listening", port: 3000 }));
});

What to compare in the starter server

The original smoke-test server remains unchanged for reference.

import { createClient } from "redis";

const redis = createClient({ url: "redis://redis:6379" });

redis.on("error", (error) => {
  console.error(JSON.stringify({
    service: "worker",
    event: "redis_error",
    message: error.message
  }));
});

await redis.connect();
console.log(JSON.stringify({ service: "worker", event: "worker_ready" }));

for (;;) {
  const queuedJob = await redis.BRPOP("jobs:pending", 1);

  if (!queuedJob) {
    continue;
  }

  const job = JSON.parse(queuedJob.element);
  console.log(JSON.stringify({ service: "worker", event: "job_started", jobId: job.id }));

  await new Promise((resolve) => setTimeout(resolve, 3000));

  const completedJob = {
    ...job,
    result: job.text.toUpperCase(),
    completedAt: new Date().toISOString()
  };

  await redis.LPUSH("jobs:completed", JSON.stringify(completedJob));
  console.log(JSON.stringify({ service: "worker", event: "job_completed", jobId: job.id }));
}

What to compare in the worker

Confirm that the worker pops from jobs:pending and pushes to jobs:completed.

Before you start the stack, which service do you expect to announce readiness first?

  • Start the API, worker, and Redis services by running this command from the docker-job-queue directory:
docker compose up

What does this command do?

Compose builds the application image and starts all three services. It keeps the terminal attached so you can watch their combined logs.

You will see structured logs containing api_listening and worker_ready once the application services connect to Redis.

Application service stopped during startup?

The short dependency configuration starts Redis first without waiting for readiness. Stop the attached run with Ctrl+C if an application service exits.

Start the stack again after Redis has finished starting.

Help me diagnose my Compose startup.

  • Open http://localhost:3000 in your browser.
  • Enter hello containers in the text field.
  • Click Queue job.

The status changes to a queued job ID. After about three seconds, you will see hello containers -> HELLO CONTAINERS under completed jobs.

That is the queue working end to end. The attached terminal shows job_queued before job_completed.

Prove container-local data disappears

Your completed job currently lives inside the Redis container's writable layer. Removing the stack gives you a direct test of whether that history belongs to the application or to one container instance.

  • Return to the terminal running Compose.
  • Press Ctrl+C to stop the attached services.

Before you recreate the stack, do you expect the completed job to survive?

  • Remove the service containers and Compose network by running this command:
docker compose down

What does this teardown remove?

The command removes the service containers and default Compose network. This configuration has no named volume holding Redis data.

  • Recreate the naive stack by running this command:
docker compose up

What does this second start create?

Compose creates fresh containers from the same configuration. The Redis service starts with a new writable layer.

  • Refresh http://localhost:3000 in your browser.

You will see an empty completed-jobs list. The missing result proves that container-local Redis state does not survive container removal.

Completed job still visible?

Confirm that you ran docker compose down before the second startup. A browser refresh is also required to replace the displayed list with the new API response.

Help me verify the data-loss test.

You have a working three-service queue and evidence of its first reliability gap. Next, you will make startup and state survive container recreation.

Make Startup and State Reliable

Your three-service queue can process jobs. The last recreation exposed the reliability gap this step fixes.

The Redis container kept completed jobs in its disposable writable layer. Docker Compose also started the application services without confirming that Redis could accept connections.

You will add a named volume for Redis data. A health check will become the readiness signal for the API and worker.

In this step, get ready to:
  • Add a Redis health check with snapshot persistence.
  • Make the API and worker wait for healthy Redis.
  • Prove that a completed job survives container recreation.
Configure Redis health and persistent storage

A container can disappear during routine recreation. The named volume stores Redis data outside that container lifecycle.

  • Switch back to compose.yaml in Neovim.
  • Find the existing redis service near the bottom of the file.
  • Replace the redis service through the end of the file with this configuration:
  redis:
    image: redis:7.4.12-alpine
    command: ["redis-server", "--save", "60", "1", "--loglevel", "warning"]
    volumes:
      - redis-data:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5
      start_period: 10s

volumes:
  redis-data:

What does this configuration do?

  • The pinned Redis image keeps the server version consistent across rebuilds.
  • The custom command saves a snapshot after at least one write occurs within 60 seconds.
  • The redis-data volume stores that snapshot at /data outside the container lifecycle.
  • The health check waits for a successful Redis response before reporting the service as healthy.
  • Save compose.yaml.
  • Start the updated stack in the background by running this command:
docker compose up -d

What does this command do?

The detached mode starts the services without occupying your terminal. Compose also creates the named volume declared in compose.yaml.

  • Inspect the running services by running this command:
docker compose ps

What does this status show?

Compose lists each container with its current state. The Redis row now includes the healthy status because its health check is passing.

Good. Redis now reports real readiness and has a storage location that survives container replacement.

Redis not becoming healthy?

Check the indentation beneath healthcheck. Each timing setting must line up beneath that key.

Confirm that the health check uses the exact list shown above. A typo prevents Redis from returning the expected response.

Help me troubleshoot an unhealthy Redis service.

Wait for healthy Redis before starting the app

A startup order only proves that Redis began starting first. A health-based dependency proves that Redis can answer before the API or worker starts.

  • Find the complete api service in compose.yaml.
  • Replace that service with this configuration:
  api:
    build: .
    command: ["node", "src/api.js"]
    init: true
    ports:
      - "3000:3000"
    depends_on:
      redis:
        condition: service_healthy

How does the API start safely?

  • The init setting adds a small init process for signal forwarding and process cleanup.
  • The service_healthy condition keeps the API waiting until the Redis health check passes.
  • The published port keeps the dashboard available at http://localhost:3000.
  • Save compose.yaml.
  • Apply the updated API configuration by running this command:
docker compose up -d

What happens to the API?

Compose reconciles the running stack with the saved configuration. The API starts after Redis reaches its healthy state.

  • Refresh http://localhost:3000 in your browser.

You should see the Docker Job Queue form with its text field and Queue job button.

Dashboard not loading?

Confirm that the API service still publishes port 3000. Check that its dependency condition is nested beneath the redis key.

Help me troubleshoot the API startup condition.

  • Find the complete worker service in compose.yaml.
  • Replace that service with this configuration:
  worker:
    build: .
    command: ["node", "src/worker.js"]
    init: true
    stop_grace_period: 10s
    depends_on:
      redis:
        condition: service_healthy

How does the worker lifecycle improve?

  • The worker waits for the same Redis health signal as the API.
  • The init process forwards stop signals to the Node.js process.
  • The 10-second grace period gives the worker time to respond before Compose forces the container to exit.
  • Save compose.yaml.
  • Apply the updated worker configuration by running this command:
docker compose up -d

What does Compose update?

Compose recreates services whose configuration changed. The application image stays shared between the API and worker.

  • Confirm the three services are running by running this command:
docker compose ps

What should the stack contain?

You should see the API, worker, and Redis services listed. Redis should carry the healthy status.

Worker missing from the service list?

Check that worker remains aligned with api under services. A nesting error can make Compose treat it as part of another service.

Help me restore the worker service.

✔️ Awesome, I've got everything!

Your saved compose.yaml now includes health-based startup and persistent Redis storage.

ⓧ I'd like to double check the full code

Compare your complete compose.yaml with this version:

services:
  api:
    build: .
    command: ["node", "src/api.js"]
    init: true
    ports:
      - "3000:3000"
    depends_on:
      redis:
        condition: service_healthy

  worker:
    build: .
    command: ["node", "src/worker.js"]
    init: true
    stop_grace_period: 10s
    depends_on:
      redis:
        condition: service_healthy

  redis:
    image: redis:7.4.12-alpine
    command: ["redis-server", "--save", "60", "1", "--loglevel", "warning"]
    volumes:
      - redis-data:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5
      start_period: 10s

volumes:
  redis-data:
Prove that completed jobs survive recreation

The configuration is only useful if stored work survives a real teardown. You will create a completed job before replacing every container again.

  • Return to the dashboard at http://localhost:3000.
  • Enter persistence check in the text field.
  • Click Queue job.

After the worker finishes its three-second delay, you should see persistence check converted to uppercase in the completed-jobs list.

This wait is the slow part. Redis needs at least 60 seconds after the write to create the configured snapshot.

  • Wait at least 60 seconds after the completed job appears.

Before you tear the stack down, do you think the completed job will survive the container replacement this time?

  • Remove the running service containers and Compose network by running this command:
docker compose down

Why does the data remain?

This teardown removes the service containers and project network. The declared named volume remains because the command does not request volume removal.

Expect the next startup to pause briefly while Redis passes its health check. That pause protects the API and worker from connecting too early.

  • Recreate the stack in the background by running this command:
docker compose up -d

What does recreation test?

Compose creates fresh containers from the saved configuration. Redis attaches the existing redis-data volume to the new container.

Before you inspect the stack, which service should report the health signal that unlocks the API and worker?

  • Check the recreated service states by running this command:
docker compose ps

What proves startup is ready?

You should see the API, worker, and Redis running. The Redis row should report healthy.

  • Refresh the dashboard at http://localhost:3000.

You should see the completed persistence check job reappear. That result proves the Redis snapshot survived the container lifecycle in redis-data.

Completed job missing after restart?

Confirm that you waited at least 60 seconds after the completed job appeared. Redis needs that interval before the configured snapshot is written.

Check that redis-data:/data is nested beneath the Redis service. Confirm that the top-level redis-data declaration is still present.

Help me troubleshoot Redis persistence.

That reliability layer is now working: Redis starts healthy and completed jobs survive container replacement. Next, you will scale the worker service and watch several replicas share the queue.

Scale and Observe the Workers

Your reliable stack now waits for healthy Redis before the application services start. Completed jobs also survive a full teardown.

A single worker still processes the queue serially. Scaling only the worker service lets several jobs move through the shared queue at once.

You'll use Docker Compose to start three worker replicas. Service-prefixed logs will expose which replica processes each job.

In this step, get ready to:
  • Run three worker replicas against the shared queue.
  • Submit at least six jobs through the API.
  • Correlate Compose service state with worker log prefixes.
Scale the worker service

A worker replica is an independently named container created from the same service definition. Your worker service has no fixed container_name.

Scaling changes the live Compose state. It leaves every source file unchanged.

Why scale only the worker?

The API provides one stable entry point through port 3000. Redis provides shared queue state.

Workers publish no host port. Multiple replicas can therefore consume the same queue without port collisions.

Before you scale the service, how many worker containers do you expect Compose to report afterward? Hold that number in mind.

  • Start the stack with three worker replicas by running:
docker compose up -d --scale worker=3

What does this command do?

  • The up command reconciles the running Compose project with the requested state.
  • The -d option leaves the containers running in the background.
  • The --scale worker=3 option requests three instances of the worker service.
  • Check the startup summary in your terminal.

You'll see the existing application services running. You'll also see three worker containers created or started.

Fewer than three workers?

  • Return to the project terminal for the docker-job-queue folder.
  • Check that the worker service in compose.yaml has no container_name entry.

Help me diagnose worker scaling.

Submit a burst and follow worker logs

All three workers block on the same jobs:pending list in Redis. A burst gives several waiting workers a chance to claim work.

  • Submit six jobs through the running API by running:
docker compose exec api node --eval '(async () => { for (let index = 1; index <= 6; index += 1) { const response = await fetch("http://localhost:3000/jobs", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ text: `scale test ${index}` }) }); console.log(response.status); } })()'

What does this command do?

  • The docker compose exec api portion runs a command inside the existing API container.
  • The node --eval portion evaluates the inline script with the container's Node.js runtime.
  • The loop sends six POST requests to /jobs with numbered text values.

You'll see six 202 status codes. Each code confirms that the API accepted one job.

Jobs not accepted?

  • Check that the API container remains running before retrying the burst.
  • Check that Redis still reports a healthy state in the Compose stack.

Help me troubleshoot the job burst.

The -f option keeps the terminal attached to new worker logs. Pressing Ctrl+C ends only the log view.

Before you follow the logs, do you expect one worker prefix or three different prefixes to appear? Keep your prediction in mind.

  • Follow the worker service logs by running:
docker compose logs -f worker

How do these logs show scaling?

  • The worker argument limits the output to the worker service.
  • The -f option follows new log output as each job progresses.
  • Each log prefix identifies the worker replica that produced that line.

You'll see three different worker prefixes around job_started events. Matching job_completed events appear as the replicas finish the burst.

That burst proves the replicas are active participants. The queue is distributing work across independently running workers.

Only one worker prefix processing jobs?

  • Submit another six-job burst while the log stream is still following.
  • Check that the scaling command finished before submitting the new burst.

Help me inspect worker distribution.

  • Stop following the live log stream by pressing Ctrl+C.
Inspect the replica state

Logs prove that work happened. The Compose service list confirms the topology that produced those events.

Before you inspect the stack, predict how many worker rows you will see beside the healthy Redis service. Hold that picture in mind.

  • Inspect every running service by running:
docker compose ps

How do you read this status?

  • Each worker row represents one independently named replica of the same Compose service.
  • The Redis row reports the current health state from its configured health check.
  • The API row shows the published port that keeps the local dashboard available.

You'll see three separately named worker rows. Redis will report a healthy state.

The API row will list its published 3000:3000 port mapping. The worker rows will have no published ports.

  • Match the three worker rows to the replica prefixes from the log stream.

That is the operational proof you were aiming for. Three workers are processing one shared queue independently.

Secret mission

Finish In-Flight Work Before Shutdown

A worker can already own a job when its container receives a stop signal. Add signal-aware lifecycle control so the worker finishes that job before closing its Redis connection and exiting.

Clean Up Your Resources

Clean Up Your Resources

Everything in this project runs locally, so there are no ongoing service costs. The application image and named volume still consume disk space until you remove them.

Decide whether to keep the lab running, pause its containers, or delete its resources entirely.

Resources you used:

  • Local Docker Engine application image named docker-job-queue.
  • API and worker containers managed by Docker Compose.
  • One Redis container connected to the Compose network.
  • Named volume redis-data containing queued and completed jobs.
  • Local docker-job-queue folder containing the application source and container configuration.

Keep everything running

No action is needed if every service is already running. Choose this option if you want the dashboard to remain available at http://localhost:3000.

  • Return to the terminal in the docker-job-queue folder from earlier.
  • Start any stopped service in detached mode by running this command:
docker compose up -d

What Stays Available?

Compose keeps the API, worker, and Redis services running in the background. The redis-data volume keeps your queued and completed jobs available.

Pause - I'll come back to this later

Shut down the running processes to free system resources while keeping the containers and Redis data for a later start.

  • Return to the terminal in the docker-job-queue folder from earlier.
  • Stop the Compose services by running this command:
docker compose stop

What Does Pausing Keep?

Compose stops the containers without removing them. The redis-data volume remains available for your next start.

Delete - I don't want to use this again

Deleting the named volume is permanent. Your queued jobs and completed jobs disappear with it.

  • Return to the terminal in the docker-job-queue folder from earlier.
  • Remove the service containers, Compose network, and named volume by running this command:
docker compose down -v

What Does This Remove?

Compose removes the API, worker, and Redis containers. It also removes the project network and the redis-data volume.

  • Move to the folder that contains docker-job-queue and remove the project files by running these commands:
cd ..
rm -rf docker-job-queue

What Happens to the Project Files?

The first command leaves the project folder. The second command permanently removes the docker-job-queue folder and its source files.

  • Remove the local application image named docker-job-queue separately through the image removal workflow in your Docker interface.
  • Confirm the docker-job-queue folder no longer exists.
  • Confirm the local application image no longer appears in your image list.

Nice Work!

Nice Work!

You did it! Your Docker Compose job queue now stays reliable across the full container lifecycle.

You've learned how to:

  • Built a reusable Docker image for the Node.js runtime. Used it for the Express API. Used it again for the worker. Ran each role as a non-root process.
  • Connected the services through service discovery. Gated startup on a Redis health check. Preserved jobs in redis-data across container recreation.
  • Scaled the worker service to three replicas. Followed service-prefixed logs to see jobs spread across workers. Connected the replica count to each worker's runtime state.
  • Secret Mission: Made the worker finish an in-flight job after SIGTERM. Closed Redis before the worker exited within the configured grace period.

Ready to quiz yourself?