Document Processing Pipeline with BullMQ

Build a task queue system with retries, priorities, and job flows.

Introduction

30 Second Summary

When you upload a document to a service and see "Processing... we'll notify you when it's ready," there's a task queue working behind the scenes. It accepts your request, handles it in the background, retries if something goes wrong, and lets you know when it's done.

In this project, you will build a document processing pipeline using BullMQ and TypeScript. You will queue jobs, retry failures with exponential backoff, prioritize urgent work, orchestrate multi-step workflows, and monitor everything on a live dashboard.

What You'll Build

Submit a document and watch it flow through extraction, transformation, and notification stages on a live monitoring dashboard. When the simulated conversion service fails, your pipeline retries automatically until it succeeds.

By the end of this project, you'll have:

  • A live Bull Board dashboard where you submit a document and watch it move through waiting, active, failed, and completed states in real time.
  • Automatic retry logic with exponential backoff that recovers gracefully when a flaky conversion service fails, visible as retry attempts ticking up on the dashboard.
  • A multi-step pipeline using parent-child job flows where you submit one document and watch three child jobs (extract, thumbnail, notify) complete before the parent finalizes.
  • Secret Mission: a scheduled cleanup job that automatically removes old completed jobs from Redis every 30 seconds.

Are there any prerequisites?

You should be comfortable with basic TypeScript or JavaScript and familiar with npm. You also need Docker installed with a Redis container running. Step 1 covers verifying your Redis connection.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters. Understanding the problem a task queue solves will give you a clear mental model for every step ahead.

Set Up the Project

To build a document processing pipeline, you need a TypeScript project that can talk to Redis.

In this step, you will create the project folder, install BullMQ and its supporting packages, configure TypeScript, and verify that your code can connect to the Redis container running in Docker.

In this step, get ready to:
  • Initialize a Node.js project and install BullMQ, Express, and Bull Board.
  • Create a TypeScript config and a Redis connection module.
  • Verify the connection to Redis prints a success message.
Initialize the project and install dependencies
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir doc-pipeline
cd doc-pipeline
  • Initialize a new Node.js project by running this command:
npm init -y

You should see the generated package.json contents printed in your terminal.

  • Install the project's runtime dependencies by running this command:
npm install bullmq@5.81.2 express@5.2.1 @bull-board/api@8.3.2 @bull-board/express@8.3.2

What are these packages?

  • bullmq is the task queue library that manages jobs in Redis.
  • express is a web framework you will use to serve a monitoring dashboard.
  • @bull-board/api and @bull-board/express provide a visual dashboard for watching jobs move through your queues.
  • Install the dev dependencies by running this command:
npm install -D tsx@4.23.1 typescript@5.9.3 @types/node@24.13.3 @types/express@5.0.6

What is tsx?

tsx lets you run TypeScript files directly without a separate compile step. It uses esbuild under the hood, so it is fast and requires zero configuration.

You should see npm report that packages were added successfully.

  • Open package.json in your editor and replace its entire contents with the following:
{
  "name": "doc-pipeline",
  "version": "1.0.0",
  "type": "module",
  "scripts": {
    "producer": "npx tsx src/producer.ts",
    "worker": "npx tsx src/worker.ts",
    "dashboard": "npx tsx src/dashboard.ts",
    "flow": "npx tsx src/flow-producer.ts",
    "finalize-worker": "npx tsx src/finalize-worker.ts"
  },
  "dependencies": {
    "bullmq": "5.81.2",
    "express": "5.2.1",
    "@bull-board/api": "8.3.2",
    "@bull-board/express": "8.3.2"
  },
  "devDependencies": {
    "tsx": "4.23.1",
    "typescript": "5.9.3",
    "@types/node": "24.13.3",
    "@types/express": "5.0.6"
  }
}

What did you change?

  • "type": "module" tells Node.js to treat .js files as ES modules so you can use import/export syntax.
  • The scripts section adds shortcut commands you will use later to run different parts of the pipeline.
  • Save package.json.
  • Create the TypeScript configuration file by creating a new file called tsconfig.json in the root of your doc-pipeline folder with the following contents:
{
  "compilerOptions": {
    "module": "ESNext",
    "moduleResolution": "bundler",
    "target": "ES2022",
    "outDir": "dist",
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "forceConsistentCasingInFileNames": true
  },
  "include": ["src"]
}

What does this config do?

This tells TypeScript to use modern ES module syntax, target a recent JavaScript version, and only look at files inside the src folder. The bundler module resolution matches how tsx resolves imports at runtime.

  • Save tsconfig.json.
  • Confirm your project structure by running:
ls

You should see node_modules, package.json, package-lock.json, and tsconfig.json listed.

Missing node_modules or package-lock.json?

If node_modules is missing, run npm install again. Make sure you are inside the doc-pipeline folder when you run the command.

Help me troubleshoot my npm install

Create the connection config and test script

BullMQ needs to know where Redis is running. You will create a shared connection config that every file in the project imports.

  • Create the folder structure by running this command:
mkdir -p src/config
  • Create a new file at src/config/connection.ts with the following contents:
import type { ConnectionOptions } from "bullmq";

export const connection: ConnectionOptions = {
  host: "localhost",
  port: 6379,
};

What does this code do?

This exports a ConnectionOptions object that tells BullMQ to connect to Redis at localhost:6379. Every queue and worker in the project will import this single object so you never repeat the connection details.

  • Save src/config/connection.ts.
  • Create a new file at src/index.ts with the following contents:
import { Queue } from "bullmq";
import { connection } from "./config/connection.js";

async function testConnection() {
  const queue = new Queue("test-connection", { connection });

  try {
    await queue.getJobCounts();
    console.log("Connected to Redis successfully");
  } catch (error) {
    console.error("Failed to connect to Redis:", error);
  } finally {
    await queue.close();
  }
}

testConnection();

What does this code do?

  • It creates a temporary Queue called test-connection using your Redis connection config.
  • getJobCounts() sends a command to Redis. If Redis responds, the connection works.
  • The finally block closes the queue so the script exits cleanly.
  • Save src/index.ts.

Why .js in the import path?

TypeScript with ES modules requires you to write the extension as .js in import paths, even though the source file is .ts. This is because the import refers to the resolved output file, and tsx handles the mapping at runtime.

Run the verification

Everything is wired up. Time to confirm that your code can reach Redis.

  • First, confirm your Docker Redis container is running:
docker ps

You should see a container with redis in the IMAGE column. If it is not running, start it before continuing.

Redis container not listed?

If you do not see a Redis container, start one by running docker run -d -p 6379:6379 redis. Then run docker ps again to confirm it is active.

Help me get my Redis container running

Before you run the test, what do you think will happen? Will the script print a success message or an error?

  • Run the connection test by executing:
npx tsx src/index.ts

You should see this printed in your terminal:

Connected to Redis successfully

Seeing 'Failed to connect to Redis'?

  • Make sure Docker Desktop is running and your Redis container is active (docker ps should show it).
  • Verify no other process is using port 6379.
  • Check that src/config/connection.ts has host: "localhost" and port: 6379.

Help me debug my Redis connection

✔️ Awesome, I've got everything!

Great. Double check you have saved all four files: package.json, tsconfig.json, src/config/connection.ts, and src/index.ts.

ⓧ I'd like to double check the full code

Scroll up and confirm each of these files matches what you have:

  • package.json has "type": "module", the scripts section, and your dependencies listed.
  • tsconfig.json has "module": "ESNext", "moduleResolution": "bundler", and "include": ["src"].
  • src/config/connection.ts exports a connection object with host: "localhost" and port: 6379.
  • src/index.ts imports Queue from bullmq, calls getJobCounts(), and logs the success message.

Your project is wired up and talking to Redis. Next up, you will create your first queue and worker to see jobs flow from submitted to completed.

Create Your First Queue and Worker

In the previous step, you confirmed that your TypeScript project can talk to Redis. Now it's time to put Redis to work.

The core of any background job system is a simple pattern: a producer adds jobs to a queue, and a worker picks them up and processes them independently. This separation means your main application never blocks while waiting for slow tasks like file conversions or API calls to finish.

In this step, get ready to:
  • Create a producer script that adds document conversion jobs to a queue.
  • Create a worker script that picks up queued jobs and processes them.
  • Run both scripts together and watch a job flow from submitted to completed.
Create a producer that adds document jobs

The producer is the entry point for work. It creates a named queue and pushes jobs into it. Each job carries a payload of data that tells the worker what to do.

  • Create a new file called src/producer.ts in VS Code.
  • Add the following code to set up the queue and submit a document job:
import { Queue, QueueEvents } from "bullmq";
import { connection } from "./config/connection.js";

// Create a queue named 'document-processing'
const queue = new Queue("document-processing", { connection });
// QueueEvents listens for global job lifecycle events
const queueEvents = new QueueEvents("document-processing", { connection });

async function addDocuments() {
  // Add a job named 'convert' with document metadata
  const job = await queue.add("convert", { filename: "report.pdf", format: "docx-to-pdf" });
  console.log(`Added job ${job.id}: report.pdf`);
}

// Listen for job completion across all workers
queueEvents.on("completed", ({ jobId }) => {
  console.log(`Job ${jobId} completed`);
});

// Listen for job failures across all workers
queueEvents.on("failed", ({ jobId, failedReason }) => {
  console.log(`Job ${jobId} failed: ${failedReason}`);
});

addDocuments().catch(console.error);

What does this code do?

This script does two things: it submits work and it listens for results.

  • Queue("document-processing") creates (or connects to) a named queue in Redis. All producers and workers sharing this name are part of the same pipeline.
  • queue.add("convert", {...}) pushes a job with a name and a data payload. The job sits in Redis until a worker is ready.
  • QueueEvents subscribes to lifecycle events so you can react when jobs complete or fail, without polling.
  • Save src/producer.ts.
  • Run the producer to verify it can queue a job:
npx tsx src/producer.ts

You should see output like Added job 1: report.pdf. The process stays running because QueueEvents keeps listening for events.

  • Press Ctrl+C to stop the producer.

Seeing a connection error?

  • Confirm your Redis container is still running with docker ps. If it's stopped, restart it.
  • Check that src/config/connection.ts points to localhost:6379.

Help me debug a Redis connection error in BullMQ

Create a worker that processes document jobs

The worker is the other half of the pattern. It connects to the same named queue and pulls jobs as they arrive. Each job runs through a processor function that you define.

  • Create a new file called src/worker.ts in VS Code.
  • Add the following code to create a worker that simulates document conversion:
import { Worker } from "bullmq";
import { connection } from "./config/connection.js";

const worker = new Worker(
  "document-processing",
  async (job) => {
    // Extract document info from the job payload
    const { filename, format } = job.data;
    console.log(`Processing document: ${filename} (${format})`);

    // Simulate a 2-second conversion process
    await new Promise((resolve) => setTimeout(resolve, 2000));

    console.log(`Document ${filename} processed successfully`);
    return { filename, status: "converted", timestamp: new Date().toISOString() };
  },
  { connection }
);

// Log when the worker finishes a job
worker.on("completed", (job) => {
  console.log(`Job ${job.id} completed: ${job.data.filename}`);
});

// Log when the worker fails a job
worker.on("failed", (job, err) => {
  console.log(`Job ${job?.id} failed: ${err.message}`);
});

console.log("Worker started. Waiting for jobs...");

What does this code do?

  • The Worker constructor takes three arguments: the queue name to listen on, a processor function that handles each job, and a connection config.
  • The processor receives the full job object including its data payload. Whatever you return becomes the job's result.
  • The 2-second setTimeout simulates real work (like calling a conversion API). In production, this would be actual processing logic.
  • Event listeners on the worker let you react locally when jobs finish or fail.
  • Save src/worker.ts.

Seeing a TypeScript error on save?

  • Make sure the import path ends in .js (not .ts). ESM resolution requires the .js extension even for TypeScript source files.
  • Check that src/worker.ts is inside the src folder, not at the project root.

Help me fix a TypeScript import error in my BullMQ worker

Run both scripts and observe the job lifecycle

Now for the payoff. You will run the producer and worker in separate terminals and watch a job travel through the full BullMQ lifecycle: waiting, active, and completed.

  • Open a second terminal in VS Code by clicking the + icon in the terminal panel.
  • In terminal 1, start the worker by running:
npx tsx src/worker.ts

You should see Worker started. Waiting for jobs... appear. The worker immediately picks up the job you queued earlier and processes it.

  • Before you run the producer in the second terminal, predict: what will you see in the producer terminal when the worker finishes the job?
  • In terminal 2, run the producer:
npx tsx src/producer.ts

In terminal 1 (worker), you should see:

Processing document: report.pdf (docx-to-pdf)
Document report.pdf processed successfully
Job 2 completed: report.pdf

In terminal 2 (producer), you should see:

Added job 2: report.pdf
Job 2 completed

What just happened?

The producer pushed a job into Redis and kept listening. The worker pulled the job from Redis, ran your processor function for 2 seconds, then marked it as completed. The QueueEvents listener in the producer detected the completion and logged it.

The producer and worker are completely independent processes. They never call each other directly. Redis is the only bridge between them.

  • Press Ctrl+C in both terminals to stop the scripts.

Worker not picking up the job?

  • Confirm the queue name "document-processing" is identical in both src/producer.ts and src/worker.ts. A single character difference means they connect to different queues.
  • Make sure the worker terminal is running BEFORE you run the producer. If the worker is not running, the job sits in Redis waiting.
  • Check that Docker Redis is still running with docker ps.

Help me figure out why my BullMQ worker is not picking up jobs

✔️ Awesome, I've got everything!

Great. Double check you have saved both files before moving on.

ⓧ I'd like to double check the full code

Here is the full src/producer.ts:

import { Queue, QueueEvents } from "bullmq";
import { connection } from "./config/connection.js";

const queue = new Queue("document-processing", { connection });
const queueEvents = new QueueEvents("document-processing", { connection });

async function addDocuments() {
  const job = await queue.add("convert", { filename: "report.pdf", format: "docx-to-pdf" });
  console.log(`Added job ${job.id}: report.pdf`);
}

queueEvents.on("completed", ({ jobId }) => {
  console.log(`Job ${jobId} completed`);
});

queueEvents.on("failed", ({ jobId, failedReason }) => {
  console.log(`Job ${jobId} failed: ${failedReason}`);
});

addDocuments().catch(console.error);

Here is the full src/worker.ts:

import { Worker } from "bullmq";
import { connection } from "./config/connection.js";

const worker = new Worker(
  "document-processing",
  async (job) => {
    const { filename, format } = job.data;
    console.log(`Processing document: ${filename} (${format})`);
    await new Promise((resolve) => setTimeout(resolve, 2000));
    console.log(`Document ${filename} processed successfully`);
    return { filename, status: "converted", timestamp: new Date().toISOString() };
  },
  { connection }
);

worker.on("completed", (job) => {
  console.log(`Job ${job.id} completed: ${job.data.filename}`);
});

worker.on("failed", (job, err) => {
  console.log(`Job ${job?.id} failed: ${err.message}`);
});

console.log("Worker started. Waiting for jobs...");

You just built the fundamental pattern behind every background job system. Your producer and worker can start, stop, and crash independently without losing work. Next up, you will make the worker deliberately fail and watch BullMQ automatically retry with exponential backoff.

Handle Failures with Retries and Backoff

Your worker processes documents without a hitch right now. In production, that never lasts. External services go down, networks hiccup, and APIs rate-limit you without warning.

Without retries, a single failed job is lost forever. In this step, you'll make your worker deliberately flaky and watch BullMQ automatically retry with exponential backoff. You'll also set up a monitoring dashboard to visualize the entire job lifecycle in real time.

In this step, get ready to:
  • Make the worker simulate a flaky external service that fails intermittently.
  • Set up a Bull Board dashboard to visualize job states in real time.
  • Watch jobs fail, retry with increasing delays, and eventually succeed.
Update the worker to simulate failures

Real services are unreliable. A PDF conversion API might time out on the first call but succeed if you try again a few seconds later. You'll simulate this by making your worker throw an error on the first two attempts.

  • Replace the entire contents of src/worker.ts with the following code. Start with the Worker constructor:
import { Worker } from "bullmq";
import { connection } from "./config/connection.js";

const worker = new Worker(
  "document-processing",
  async (job) => {
    const { filename, format } = job.data;
    console.log(`[Attempt ${job.attemptsMade + 1}] Processing: ${filename} (${format})`);

    // Simulate a flaky service: fail on first 2 attempts
    if (job.attemptsMade < 2) {
      throw new Error(`Conversion service unavailable (attempt ${job.attemptsMade + 1})`);
    }

    // Simulate processing time
    await new Promise((resolve) => setTimeout(resolve, 1500));
    console.log(`Document ${filename} converted successfully!`);
    return { filename, status: "converted", timestamp: new Date().toISOString() };
  },
  {
    connection,
    concurrency: 3,
    limiter: { max: 2, duration: 1000 },
  }
);

What does this code do?

The processor function now checks job.attemptsMade before doing any real work. On attempts 0 and 1 (the first two tries), it throws an error that simulates a flaky external service.

The limiter option throttles the worker to a maximum of 2 jobs per second globally across all workers. The concurrency: 3 setting allows up to 3 jobs to be processed in parallel.

  • Directly below the closing ); of the Worker constructor, add the event listeners:
worker.on("completed", (job) => {
  console.log(`Job ${job.id} completed: ${job.data.filename}`);
});

worker.on("failed", (job, err) => {
  console.log(`Job ${job?.id} failed (attempt ${job?.attemptsMade}): ${err.message}`);
});

worker.on("error", (err) => {
  console.error("Worker error:", err);
});

console.log("Worker started. Waiting for jobs...");

What does the error event catch?

The error event fires for unexpected internal problems like Redis connection failures. It does not fire for job processing errors. Those are caught by the failed event instead. You must always attach an error listener to prevent unhandled exceptions from crashing your process.

  • Save src/worker.ts.
  • Before you run this, think about what will happen. The old producer adds a job without any attempts configuration. What do you think happens when a job fails with no retry config?
  • Start the worker in one terminal by running:
npx tsx src/worker.ts
  • In a second terminal, run the producer:
npx tsx src/producer.ts

The worker terminal shows the job failing once and staying in the failed state. Without an attempts option on the job, BullMQ does not retry it.

Not seeing any output?

Make sure your Docker Redis container is still running. Run docker ps to verify. If it's stopped, start it again before retrying.

If you see a connection error, double-check that Redis is accessible on localhost:6379.

help me debug why my BullMQ worker can't connect to Redis

Add retry configuration to the producer

The job failed once and was abandoned. That single network hiccup lost the document forever. To fix this, you'll configure jobs with retry attempts and exponential backoff.

What is exponential backoff?

Exponential backoff means each retry waits longer than the last. With a base delay of 1000ms, the formula is 2^(attempts-1) * delay. So the second attempt waits 1 second, the third waits 2 seconds, the fourth waits 4 seconds, and so on.

This prevents hammering a struggling service with rapid retries. It gives the external system time to recover before you try again.

  • Stop the worker by pressing Ctrl+C in its terminal.
  • Replace the entire contents of src/producer.ts with the following code. Start with the imports and normal-priority jobs:
import { Queue, QueueEvents } from "bullmq";
import { connection } from "./config/connection.js";

const queue = new Queue("document-processing", { connection });
const queueEvents = new QueueEvents("document-processing", { connection });

async function addDocuments() {
  // Normal-priority jobs (priority 10)
  const normalJobs = [
    { filename: "report.pdf", format: "docx-to-pdf" },
    { filename: "invoice.pdf", format: "xlsx-to-pdf" },
    { filename: "manual.pdf", format: "md-to-pdf" },
  ];

  for (const doc of normalJobs) {
    const job = await queue.add("convert", doc, {
      priority: 10,
      attempts: 5,
      backoff: { type: "exponential", delay: 1000 },
    });
    console.log(`Added job ${job.id}: ${doc.filename} (priority: 10)`);
  }

What do these options do?

Each job now carries attempts: 5 (up to 5 total tries) and backoff: { type: "exponential", delay: 1000 } (wait longer between each retry). If the worker throws, BullMQ automatically reschedules the job with increasing delay.

The priority: 10 sets these as normal-priority jobs. In BullMQ, lower numbers mean higher priority among prioritized jobs. You'll see why this matters with the rush job below.

  • Directly below the closing } of the for loop, add the rush job, delayed job, event listeners, and function call:
  // High-priority job (priority 1 = processed first among prioritized jobs)
  const rushJob = await queue.add(
    "convert",
    { filename: "urgent-contract.pdf", format: "docx-to-pdf" },
    { priority: 1, attempts: 5, backoff: { type: "exponential", delay: 1000 } }
  );
  console.log(`Added RUSH job ${rushJob.id}: urgent-contract.pdf (priority: 1)`);

  // Delayed job (processes after 10 seconds)
  const delayedJob = await queue.add(
    "convert",
    { filename: "scheduled-report.pdf", format: "docx-to-pdf" },
    { delay: 10000, priority: 10, attempts: 5, backoff: { type: "exponential", delay: 1000 } }
  );
  console.log(`Added DELAYED job ${delayedJob.id}: scheduled-report.pdf (delay: 10s)`);
}

queueEvents.on("completed", ({ jobId }) => {
  console.log(`Job ${jobId} completed`);
});

queueEvents.on("failed", ({ jobId, failedReason }) => {
  console.log(`Job ${jobId} failed: ${failedReason}`);
});

addDocuments().catch(console.error);

What are priority and delay doing here?

The rush job uses priority: 1. Since lower numbers process first among prioritized jobs, it jumps ahead of the normal jobs (priority 10) despite being added last.

The delayed job uses delay: 10000 (10 seconds). It sits in a "delayed" state until its timer expires, then moves to the queue for processing. This simulates scheduling a conversion for later.

  • Save src/producer.ts.
  • Before you run this, predict what will happen. Each job will fail twice before succeeding. How long do you think the delays between retries will be?
  • Start the worker again in the first terminal:
npx tsx src/worker.ts
  • In the second terminal, run the updated producer:
npx tsx src/producer.ts

Watch the worker terminal closely. You'll see each job fail on attempts 1 and 2, then succeed on attempt 3 after increasing delays (1 second, then 2 seconds).

Jobs failing without retrying?

If jobs fail once and stop, you may have stale jobs from the previous run. Stop both processes, flush the Redis data with docker exec <container-name> redis-cli FLUSHALL, then restart the worker and producer.

help me troubleshoot why my BullMQ jobs are not retrying with exponential backoff

Set up the Bull Board dashboard

Watching terminal logs works, but a visual dashboard makes it much easier to see job states, retry counts, and queue health at a glance.

  • Create a new file src/dashboard.ts with the following code:
import express from "express";
import { Queue } from "bullmq";
import { createBullBoard } from "@bull-board/api";
import { BullMQAdapter } from "@bull-board/api/bullMQAdapter";
import { ExpressAdapter } from "@bull-board/express";
import { connection } from "./config/connection.js";

const documentQueue = new Queue("document-processing", { connection });
const finalizeQueue = new Queue("finalize", { connection });

const serverAdapter = new ExpressAdapter();
serverAdapter.setBasePath("/admin/queues");

createBullBoard({
  queues: [
    new BullMQAdapter(documentQueue),
    new BullMQAdapter(finalizeQueue),
  ],
  serverAdapter,
});

const app = express();

app.use("/admin/queues", serverAdapter.getRouter());

app.listen(3000, () => {
  console.log("Bull Board dashboard running at http://localhost:3000/admin/queues");
});

What does this code do?

Bull Board gives you a web-based UI to monitor your queues. ExpressAdapter connects Bull Board to an Express server, and BullMQAdapter wraps each queue so the dashboard can read its data from Redis.

The finalize queue is registered now even though you haven't used it yet. It will come into play when you build parent-child flows in the next step.

  • Save src/dashboard.ts.
  • Stop any running processes with Ctrl+C. You'll now run three processes in three separate terminals.
  • In terminal 1, start the dashboard:
npx tsx src/dashboard.ts
  • In terminal 2, start the worker:
npx tsx src/worker.ts
  • In terminal 3, run the producer:
npx tsx src/producer.ts
  • Open http://localhost:3000/admin/queues in your browser.

You should see the Bull Board dashboard with your document-processing queue listed. Click into it to see jobs in various states. Jobs that are retrying show their attempt count, and you can watch them move from failed to active to completed as the backoff timer expires.

Dashboard not loading?

If http://localhost:3000/admin/queues shows an error, check that nothing else is using port 3000. Also confirm the dashboard terminal shows the startup message before you open the URL.

If the dashboard loads but shows no jobs, make sure you ran the producer after starting the dashboard. Jobs submitted before the dashboard started will still appear since Bull Board reads directly from Redis.

help me troubleshoot why my Bull Board dashboard is not showing any queues or jobs

✔️ Awesome, I've got everything!

Great. Make sure you've saved both src/worker.ts and src/producer.ts, and that src/dashboard.ts is saved too.

ⓧ I'd like to double check the full code

Your src/worker.ts should contain the Worker constructor chunk (with the flaky service logic and limiter options) followed by the event listeners chunk. Scroll up and confirm both code blocks are present in order.

Your src/producer.ts should contain the imports and normal jobs chunk followed by the rush job, delayed job, and event listeners chunk. Scroll up and confirm both code blocks are present in order.

Your src/dashboard.ts should match this exactly:

import express from "express";
import { Queue } from "bullmq";
import { createBullBoard } from "@bull-board/api";
import { BullMQAdapter } from "@bull-board/api/bullMQAdapter";
import { ExpressAdapter } from "@bull-board/express";
import { connection } from "./config/connection.js";

const documentQueue = new Queue("document-processing", { connection });
const finalizeQueue = new Queue("finalize", { connection });

const serverAdapter = new ExpressAdapter();
serverAdapter.setBasePath("/admin/queues");

createBullBoard({
  queues: [
    new BullMQAdapter(documentQueue),
    new BullMQAdapter(finalizeQueue),
  ],
  serverAdapter,
});

const app = express();

app.use("/admin/queues", serverAdapter.getRouter());

app.listen(3000, () => {
  console.log("Bull Board dashboard running at http://localhost:3000/admin/queues");
});

Your pipeline now handles failure gracefully. Jobs retry automatically, and you can monitor everything from a visual dashboard. Next up, you'll explore how delayed jobs and priority queues give you fine-grained control over when and in what order jobs get processed.

Add Delayed Jobs and Priority Queues

Real systems need more than basic queuing. Sometimes a document conversion should happen later ("process this report at 6 AM"). Other times, an urgent contract needs to jump ahead of 100 routine jobs already waiting.

Your producer already includes both of these features from the previous step. Now you will run the system and observe how delayed jobs and priority queues behave in practice.

In this step, get ready to:
  • Observe how a delayed job sits in "delayed" state before processing after its timer expires.
  • Verify that a high-priority job jumps ahead of normal-priority jobs despite being added last.
  • Confirm priority ordering and delayed job transitions on Bull Board.
Observe delayed jobs on Bull Board

A delayed job enters the queue but does not become eligible for processing until its timer expires. This is how you schedule future work without external cron tools.

  • In src/producer.ts, find the delayed job section near the bottom of the addDocuments function:
const delayedJob = await queue.add(
  "convert",
  { filename: "scheduled-report.pdf", format: "docx-to-pdf" },
  { delay: 10000, priority: 10, attempts: 5, backoff: { type: "exponential", delay: 1000 } }
);
console.log(
  `Added DELAYED job ${delayedJob.id}: scheduled-report.pdf (delay: 10s)`
);

What does delay: 10000 do?

The delay option tells BullMQ to hold the job in a "delayed" state for 10,000 milliseconds (10 seconds). Once the timer expires, BullMQ automatically promotes the job to the prioritized set where the worker can pick it up.

No external scheduler is needed. BullMQ handles the timer internally using Redis sorted sets.

  • Stop any processes still running from the previous step by pressing Ctrl+C in each terminal.
  • Start the dashboard by running this command in your first terminal:
npx tsx src/dashboard.ts
  • Start the worker in a second terminal by running:
npx tsx src/worker.ts
  • Run the producer in a third terminal by running:
npx tsx src/producer.ts
  • Immediately open http://localhost:3000/admin/queues in your browser.
  • Click into the document-processing queue.
  • Look for the Delayed tab. You should see scheduled-report.pdf sitting there with a countdown.

After 10 seconds, the delayed job disappears from the Delayed tab and moves into Active (then Completed once it succeeds after retries).

Don't see the delayed job?

If you opened Bull Board more than 10 seconds after running the producer, the job may have already been promoted. Re-run the producer to add another batch and check immediately.

Make sure your worker is running in a separate terminal. Without a worker, jobs stay in their queue state indefinitely.

Why can't I see my delayed job on Bull Board?

Verify priority ordering in the terminal

Priority queues let urgent work skip the line. In your producer, the rush job is added last but has the lowest priority number, which means it gets processed first.

  • In src/producer.ts, find the section where the rush job is added:
const rushJob = await queue.add(
  "convert",
  { filename: "urgent-contract.pdf", format: "docx-to-pdf" },
  { priority: 1, attempts: 5, backoff: { type: "exponential", delay: 1000 } }
);
console.log(`Added RUSH job ${rushJob.id}: urgent-contract.pdf (priority: 1)`);

How does BullMQ priority work?

BullMQ priorities range from 1 to 2,097,151. A lower number means higher priority. Your normal jobs have priority: 10 and the rush job has priority: 1. The worker picks up priority 1 before priority 10.

All your jobs use explicit priority numbers. This is important because jobs with no explicit priority (value 0) are processed before any prioritized jobs. Giving every job an explicit number ensures the ordering works as expected.

Before you look at the terminal output, predict: which filename do you expect the worker to process first?

  • Switch to your worker terminal and read the processing log.

You should see urgent-contract.pdf appear in the first processing attempt, even though it was the last job added by the producer.

What about the rate limiter?

Your worker has limiter: { max: 2, duration: 1000 } set from the previous step. This means it processes at most 2 jobs per second. You will see the jobs trickle through at a steady pace rather than all at once. The priority ordering still holds within that rate limit.

  • Return to Bull Board and click the Completed tab.
  • Verify that urgent-contract.pdf completed before the normal-priority jobs by checking the timestamps.

Priority order looks wrong?

Because of the retry logic (jobs fail on the first 2 attempts), timing can overlap. Focus on which job the worker picks up first on its very first attempt. The first [Attempt 1] Processing: line should show urgent-contract.pdf.

If you see normal jobs first, check that your producer has priority: 1 on the rush job and priority: 10 on the normal jobs.

Why is my priority ordering not working as expected?

✔️ Awesome, I've got everything!

Great. Make sure you can see urgent-contract.pdf completing first in your worker output, and the delayed job transitioning from Delayed to Completed on Bull Board.

ⓧ I'd like to double check the full code

No code was changed in this step. Your files should still match exactly what you had at the end of the previous step. Scroll up and confirm:

  • src/producer.ts adds 3 normal-priority jobs (priority 10), 1 rush job (priority 1), and 1 delayed job (delay 10s).
  • src/worker.ts has flaky retry logic, concurrency 3, and a rate limiter of 2 jobs per second.
  • src/dashboard.ts serves Bull Board at /admin/queues on port 3000.

You now have two powerful scheduling primitives working. Delayed jobs let you schedule future processing without cron, and priority queues ensure urgent work always gets attention first. Next, you will orchestrate a multi-step pipeline using parent-child flows.

Apply Rate Limiting and Orchestrate Flows

External APIs enforce rate limits. If your worker fires 100 requests per second at an API that only allows 2 per second, you get banned.

Your worker already has a rate limiter configured from the previous step. In this step, you will confirm it throttles correctly, then build a multi-step pipeline using FlowProducer where a parent job waits for all its children to complete before finalizing.

In this step, get ready to:
  • Observe rate limiting throttle your worker to 2 jobs per second.
  • Create a flow that orchestrates three child jobs with a parent that waits for all of them.
  • Build a finalize worker that aggregates results from completed children.
Confirm rate limiting is working

Your worker already includes a limiter option from the previous step. The setting max: 2, duration: 1000 means the worker processes at most 2 jobs per second, globally across all workers on that queue.

  • Open src/worker.ts and find the limiter config at the bottom of the Worker constructor:
limiter: { max: 2, duration: 1000 },

What does limiter do?

This setting is global across all workers connected to the same queue. Even if you scale to 10 worker instances, the combined throughput stays at 2 jobs per second.

This protects downstream services from being overwhelmed. The excess jobs wait patiently in the queue until a slot opens.

  • Start your worker in one terminal by running:
npx tsx src/worker.ts
  • In a second terminal, run your producer to submit jobs:
npx tsx src/producer.ts

Watch the worker terminal. You should see jobs processing at a steady pace of roughly 2 per second, not all at once. The gaps between log lines are the rate limiter holding back the worker.

Jobs processing too fast?

If jobs appear to process instantly without gaps, check that limiter: { max: 2, duration: 1000 } is inside the third argument (options object) of the Worker constructor, alongside connection and concurrency.

help me debug why my BullMQ rate limiter isn't throttling jobs

Create the flow producer

A FlowProducer defines parent-child relationships between jobs. The parent job enters a "waiting-children" state and will not be processed until every child completes successfully.

  • Create a new file src/flow-producer.ts and paste the flow definition:
import { FlowProducer } from "bullmq";
import { connection } from "./config/connection.js";

const flowProducer = new FlowProducer({ connection });

async function submitDocumentPipeline() {
  const flow = await flowProducer.add({
    name: "finalize-document",
    queueName: "finalize",
    data: { filename: "annual-report.pdf" },
    children: [
      {
        name: "extract-text",
        queueName: "document-processing",
        data: { filename: "annual-report.pdf", step: "extract" },
      },
      {
        name: "generate-thumbnail",
        queueName: "document-processing",
        data: { filename: "annual-report.pdf", step: "thumbnail" },
      },
      {
        name: "send-notification",
        queueName: "document-processing",
        data: {
          filename: "annual-report.pdf",
          step: "notify",
          recipient: "user@example.com",
        },
      },
    ],
  });

What does this flow define?

The flowProducer.add() call atomically creates 4 jobs: one parent (finalize-document in the finalize queue) and three children (extract-text, generate-thumbnail, send-notification in the document-processing queue).

The parent stays in "waiting-children" state until all three children complete. Only then does it move to "waiting" where a worker can pick it up.

  • Still in src/flow-producer.ts, add this code directly below to log the results and invoke the function:

  console.log(`Flow submitted! Parent job ID: ${flow.job.id}`);
  console.log(`Children: ${flow.children.map((c) => c.job.id).join(", ")}`);
}

submitDocumentPipeline().catch(console.error);
  • Save src/flow-producer.ts.

Why are children in a different queue than the parent?

BullMQ flows allow children and parents to live in completely different queues. This means different workers with different concurrency settings and rate limits can handle each stage of your pipeline independently.

✔️ Awesome, I've got everything!

Great. Make sure you have saved src/flow-producer.ts.

ⓧ I'd like to double check the full code

Your file has two sections you pasted in order. Scroll up and confirm both chunks are present: the imports and flowProducer.add() call first, then the console.log lines, closing brace, and submitDocumentPipeline().catch(console.error) at the end.

Create the finalize worker and run the pipeline

The finalize worker handles the parent job after all children complete. It calls job.getChildrenValues() to gather the return values from every child.

  • Create src/finalize-worker.ts and paste the worker definition:
import { Worker } from "bullmq";
import { connection } from "./config/connection.js";

const finalizeWorker = new Worker(
  "finalize",
  async (job) => {
    console.log(`Finalizing document: ${job.data.filename}`);

    const childrenValues = await job.getChildrenValues();
    console.log("Children results:", JSON.stringify(childrenValues, null, 2));

    const totalSteps = Object.keys(childrenValues).length;
    console.log(`All ${totalSteps} pipeline steps completed for ${job.data.filename}`);

    return {
      filename: job.data.filename,
      status: "finalized",
      stepsCompleted: totalSteps,
      timestamp: new Date().toISOString(),
    };
  },
  { connection }
);

What does getChildrenValues do?

The getChildrenValues() method returns an object where each key is the child job's qualified ID and each value is what that child's processor returned. This lets the parent aggregate, summarize, or act on the combined results.

  • Still in src/finalize-worker.ts, add the event listeners and startup log below the worker:

finalizeWorker.on("completed", (job) => {
  console.log(`Finalize job ${job.id} completed!`);
});

finalizeWorker.on("error", (err) => {
  console.error("Finalize worker error:", err);
});

console.log("Finalize worker started. Waiting for parent jobs...");
  • Save src/finalize-worker.ts.

Seeing a module not found error?

Make sure the import path ends with .js (./config/connection.js). TypeScript with ESM resolution requires the .js extension even though your source files are .ts.

help me fix a module not found error in my BullMQ TypeScript project

Now run the full pipeline. You need three terminals running simultaneously.

  • In terminal 1, start the dashboard by running:
npx tsx src/dashboard.ts
  • In terminal 2, start both workers by running:
npx tsx src/worker.ts
  • In terminal 3, start the finalize worker by running:
npx tsx src/finalize-worker.ts

Before you submit the flow, think about this: what state do you expect the parent job to be in while the children are processing?

  • In a fourth terminal, submit the flow by running:
npx tsx src/flow-producer.ts

The parent job enters "waiting-children" state until all three children complete. Once the last child finishes, the parent moves to "active" and the finalize worker picks it up.

  • In your finalize worker terminal (terminal 3), you should see output like:
Finalize worker started. Waiting for parent jobs...
Finalizing document: annual-report.pdf
Children results: { ... }
All 3 pipeline steps completed for annual-report.pdf
Finalize job <id> completed!
  • Open http://localhost:3000/admin/queues in your browser.
  • Click on the finalize queue. You should see the parent job in the Completed tab.

What should I see on Bull Board?

The document-processing queue shows the three child jobs (extract-text, generate-thumbnail, send-notification) as completed. The finalize queue shows the parent job as completed.

If you submit the flow quickly and check Bull Board fast enough, you can catch the parent in "waiting-children" state before the children finish.

Your pipeline now orchestrates multiple processing stages as a coordinated flow. Next up, you will tackle the Secret Mission and automate queue cleanup with scheduled jobs.

Secret mission

Scheduled Cleanup with Job Schedulers

Production queues accumulate completed and failed jobs in Redis, consuming memory over time. Your mission: create a recurring cleanup job that automatically purges stale jobs every 30 seconds using BullMQ's upsertJobScheduler feature.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally using Docker and your file system, so there are no ongoing cloud costs.

Resources you used:

  • Redis data stored in your Docker container (local only, no cost).
  • The doc-pipeline project folder on your Desktop.

Keep everything running

No action needed. Choose this if you are still actively building or experimenting with BullMQ.

  • Your Docker Redis container continues running in the background.
  • Your doc-pipeline folder stays on your Desktop with all source files and node_modules intact.
  • You can restart any worker or producer at any time with npx tsx.

Pause - I'll come back to this later

Shut down running processes to free up memory. Your files and Redis data persist so you can pick up where you left off.

  • Press Ctrl+C in any terminal windows running workers, producers, or the dashboard.
  • Stop the Redis container by running:
docker stop [[REDIS_CONTAINER="your Redis container name"]]
  • Your Redis data persists in the container volume. When you are ready to resume, start it again with:
docker start [[REDIS_CONTAINER="your Redis container name"]]

Delete - I don't want to use this again

Remove all project resources and start fresh. This permanently deletes your Redis data and all project files.

  • Press Ctrl+C in any terminal windows running workers, producers, or the dashboard.
  • Remove the Redis container by running:
docker rm -f [[REDIS_CONTAINER="your Redis container name"]]
  • Delete the project folder. On macOS or Linux, run:
rm -rf ~/Desktop/doc-pipeline
  • On Windows, run:
rmdir /s /q %USERPROFILE%\Desktop\doc-pipeline

Find your Redis container name

If you are unsure what your Redis container is called, run docker ps -a to list all containers. Look for the one running the redis image.

That's a Wrap!

That's a Wrap!

Nice work! You just built a fully functional document processing pipeline powered by BullMQ and TypeScript from scratch.

You've learned how to:

  • Built a task queue system with queues, workers, retries with exponential backoff, delayed jobs, priority ordering, and rate limiting to handle real-world processing constraints.
  • Orchestrated multi-step workflows using FlowProducer with parent-child job dependencies where a parent job waits for all children to complete before finalizing.
  • Set up a Bull Board monitoring dashboard to visualize job states, retries, and queue health in real time.
  • Secret Mission: Automated queue cleanup with scheduled recurring jobs using upsertJobScheduler.

Ready to quiz yourself?