Distributed Task Queue with FastAPI

Introduction

30 Second Summary

When you order a package online, you don't stand at the warehouse door waiting for it to be packed. You get a tracking number and check back later.

In this project, you will build a distributed task processing system where a REST API accepts job requests, immediately returns a tracking ID, and dispatches work to background workers via a message broker.

What You'll Build

A multi-container backend system with a FastAPI REST API, Celery workers, and a Redis message broker, all orchestrated with Docker Compose.

By the end of this project, you'll have:

• A FastAPI application with endpoints to submit tasks, check their status, and list all jobs.
• Celery workers with retry logic, exponential backoff, and idempotent task execution.
• A Docker Compose file orchestrating your API, worker, scheduler, and Redis broker as a multi-service architecture.
• A Flower monitoring dashboard for real-time worker and task visibility.
• Secret Mission: Add priority queues and route tasks to dedicated workers based on urgency.

Are there any prerequisites?

Basic Python knowledge (functions, imports, decorators) is all you need. Familiarity with REST APIs is helpful but not required.

Before We Start

Every backend system eventually hits a problem: some operations take too long to run inside a web request. Sending emails, processing images, running ML inference, generating reports. If you make a user wait for those to finish, the request times out or the experience degrades.

The industry solution is a distributed task queue. In this step, you will install Docker Desktop, create the project directory, and verify that Python 3.12 is available locally for IDE support.

In this step, get ready to:

• Install and verify Docker Desktop with Docker Compose.
• Create the project directory and dependencies file.
• Confirm Python 3.12 is available for code completion in your editor.

Install and verify Docker Desktop

Docker Desktop bundles the Docker engine and Docker Compose into a single installer. Your entire application (API, worker, and broker) will run inside Docker containers, so nothing needs to be installed on your host machine beyond Docker itself.

• Open a terminal by pressing Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
• Type Terminal (macOS) or PowerShell (Windows) and press Enter to open it.
• Check if Docker is installed by running this command:

docker --version

✔️ I see a version number

Docker is installed and ready to go. Continue to the next check below.

ⓧ Command not found

You need to install Docker Desktop first.

• Download Docker Desktop from the official site:
• macOS: https://docs.docker.com/desktop/setup/install/mac-install/
• Windows: https://docs.docker.com/desktop/setup/install/windows-install/
• Run the installer and follow the on-screen prompts.
• On Windows, the installer will enable the WSL 2 backend automatically.
• Once installation is complete, launch Docker Desktop and wait for the engine to start (you will see a green indicator in the system tray).
• Return to your terminal and run docker --version again to confirm it works.

• Verify Docker Compose is available by running:

docker compose version

Why two commands?

Docker Desktop bundles Docker Compose V2 as a plugin, so you invoke it as docker compose (two words, no hyphen). If both commands return version numbers, your Docker setup is complete.

docker compose not recognised?

Make sure Docker Desktop is running (check for the whale icon in your system tray or menu bar). If you just installed it, close and reopen your terminal so the PATH updates take effect.

help me troubleshoot docker compose not being recognised

Create the project directory and dependencies file

With Docker confirmed, you need a folder to hold all your project files. You will also create a requirements.txt that pins every Python dependency the containers will install.

• Move to your Desktop by running this command:

cd ~/Desktop

• Create your project folder and move into it by running these commands:

mkdir task-queue
cd task-queue

• Confirm the folder exists by running:

ls

The output should be empty (no files yet) since you just created the folder.

• Open VS Code by pressing Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
• Type VS Code (or Cursor if you use Cursor) and press Enter to open it.
• In VS Code, click File in the top menu bar.
• Click Open Folder.
• Navigate to your Desktop, select the task-queue folder, then click Open.

Why VS Code?

VS Code (or Cursor) gives you syntax highlighting, auto-complete, and an integrated terminal. You can use whichever editor you prefer, but instructions in this project reference VS Code.

• Create a new file by clicking File then New File.
• Save it immediately with Cmd+S (macOS) or Ctrl+S (Windows) and name it requirements.txt.
• Paste the following dependency list into requirements.txt:

fastapi[standard]==0.136.3
celery[redis]==5.6.3
flower==2.0.1
pydantic>=2.7.0,<3.0.0
uvicorn[standard]>=0.30.0

What does this file do?

requirements.txt lists every Python package and version your project needs. When Docker builds your container image, it reads this file and installs these exact versions.

• fastapi[standard]==0.136.3 is your web framework for building the REST API.
• celery[redis]==5.6.3 is the distributed task queue that processes background jobs.
• flower==2.0.1 is a monitoring dashboard for Celery workers.
• pydantic>=2.7.0,<3.0.0 handles data validation for API requests.
• uvicorn[standard]>=0.30.0 is the ASGI server that runs FastAPI.

• Save requirements.txt with Cmd+S (macOS) or Ctrl+S (Windows).

✔️ Awesome, I've got everything!

Great. Double check you have saved requirements.txt inside your task-queue folder.

ⓧ I'd like to double check the full code

Here is the complete requirements.txt file:

fastapi[standard]==0.136.3
celery[redis]==5.6.3
flower==2.0.1
pydantic>=2.7.0,<3.0.0
uvicorn[standard]>=0.30.0

File not appearing in the sidebar?

Make sure you opened the task-queue folder (not your entire Desktop) in VS Code. The file should appear in the Explorer sidebar on the left.

help me find my requirements.txt file in VS Code

Verify Python 3.12 for IDE support

Your application runs inside Docker containers, so you do not need Python installed to run the project. However, having Python 3.12 locally lets your editor provide auto-complete and type checking for the code you write.

• Check your Python version by running this command in your terminal:

python3 --version

✔️ I see version 3.12 or higher

Python is ready for IDE support. You are all set to start building.

ⓧ I see an older version

Your system has an older Python. Since the application runs inside Docker (which uses its own Python 3.12), you can still complete this project without upgrading.

If you want full IDE support, install Python 3.12 from https://www.python.org/downloads/.

macOS ships an older Python

macOS often ships with Python 3.9 or 3.10. After installing a newer version, you may need to run python3.12 --version to invoke the correct binary.

ⓧ Command not found

Python is not installed on your system. Since the application runs inside Docker, you can still complete this project without installing Python locally.

If you want IDE auto-complete, install Python 3.12 from https://www.python.org/downloads/. On Windows, make sure to check the box that says Add Python to PATH during installation.

Your environment is ready. Next up, you will build the FastAPI application that accepts job requests and dispatches them to background workers.

Build the FastAPI Application

Your project structure and dependencies are ready. Now it is time to build the core of your distributed system: the API that accepts job requests and dispatches them to background workers.

In this step, you will create a FastAPI application that receives tasks from clients, immediately responds with a tracking ID, and pushes the real work onto a Celery queue backed by Redis. Clients never wait for the job to finish.

In this step, get ready to:

• Configure the Celery app to connect to Redis as both message broker and result store.
• Build REST endpoints that submit, track, and list background jobs.
• Verify the API runs locally and serves interactive documentation.

Configure the Celery worker

Before building the API, you need the messaging layer that connects it to background workers. Celery is a distributed task queue that uses a message broker (Redis, in this project) to pass jobs between the API and workers.

• Create the app/ package by running these commands in your terminal:

mkdir app
touch app/__init__.py

What does this do?

The app/ directory becomes a Python package thanks to the empty __init__.py file. This lets other modules import code from inside app/ using standard Python imports.

• Confirm the files exist by running:

ls app/

You should see __init__.py listed in the output.

• Create the Celery configuration file by adding a new file called worker.py inside the app/ folder, then paste this code:

from celery import Celery

celery_app = Celery(
    "worker",
    broker="redis://redis:6379/0",
    backend="redis://redis:6379/1",
)

celery_app.conf.update(
    task_serializer="json",
    accept_content=["json"],
    result_serializer="json",
    timezone="UTC",
    enable_utc=True,
    task_track_started=True,
    task_acks_late=True,
    worker_prefetch_multiplier=1,
)

What does this code do?

• celery_app = Celery("worker", broker=..., backend=...) creates the Celery application instance. The broker URL tells Celery where to send messages. The backend URL tells Celery where to store task results.
• redis://redis:6379/0 uses Redis database 0 for the message queue. Database 1 stores results separately, keeping the two concerns isolated.
• task_acks_late=True means the worker only confirms a task as "handled" after it finishes successfully. If the worker crashes mid-task, Redis will re-deliver the message to another worker.
• worker_prefetch_multiplier=1 prevents a single worker from grabbing many tasks at once, which improves fairness across multiple workers.

• Save app/worker.py.

Why does the broker URL say redis instead of localhost?

The hostname redis refers to the Redis container's service name inside Docker Compose. When you run the full stack in containers, Docker creates a network where services find each other by name. This won't resolve on your local machine, but that is fine for now.

✔️ Awesome, I've got everything!

Great. Make sure you have saved the file before moving on.

ⓧ I'd like to double check the full code

from celery import Celery

celery_app = Celery(
    "worker",
    broker="redis://redis:6379/0",
    backend="redis://redis:6379/1",
)

celery_app.conf.update(
    task_serializer="json",
    accept_content=["json"],
    result_serializer="json",
    timezone="UTC",
    enable_utc=True,
    task_track_started=True,
    task_acks_late=True,
    worker_prefetch_multiplier=1,
)

Build the FastAPI endpoints

With the Celery configuration in place, you can now build the REST API that clients interact with. The API needs to accept job requests, return tracking IDs, and let clients poll for status updates.

• First, create a task stub so the API can import it. Create a new file called tasks.py inside the app/ folder and paste this code:

import time
import logging

from app.worker import celery_app

logger = logging.getLogger(__name__)


@celery_app.task(bind=True)
def process_job(self, payload: str, delay_seconds: int = 5):
    job_id = self.request.id
    logger.info(f"Task {job_id} started | payload={payload}")

    # Simulate processing
    time.sleep(delay_seconds)
    result = f"Processed: {payload}"
    logger.info(f"Task {job_id} completed | result={result}")
    return result

What does this code do?

• @celery_app.task(bind=True) registers this function as a Celery task. The bind=True argument gives the function access to self, which contains metadata about the running task (like its ID).
• time.sleep(delay_seconds) simulates a long-running job. In a real system, this could be sending an email, processing an image, or calling an external API.
• This is a basic stub. You will add retry logic and idempotency in the next step.

• Save app/tasks.py.

Now you can build the main API file. This is where clients send requests and receive responses.

• Create a new file called main.py inside the app/ folder and paste the imports, app instance, and data models:

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from celery.result import AsyncResult

from app.worker import celery_app
from app.tasks import process_job

app = FastAPI(title="Distributed Task Queue", version="1.0.0")


class JobRequest(BaseModel):
    payload: str
    delay_seconds: int = 5


class JobResponse(BaseModel):
    job_id: str
    status: str


class JobStatusResponse(BaseModel):
    job_id: str
    status: str
    result: str | None = None

What does this code do?

• FastAPI(title=..., version=...) creates the application instance. FastAPI auto-generates interactive API docs from this metadata.
• The three Pydantic models define the shape of data flowing through the API. JobRequest is what clients send. JobResponse and JobStatusResponse are what clients receive back.
• AsyncResult from Celery lets you look up a task's current state by its ID, without needing to query the worker directly.

• Now add the four endpoint functions below the models in the same file:

@app.post("/jobs", status_code=202, response_model=JobResponse)
def submit_job(request: JobRequest):
    # Dispatch the task to the Celery worker queue
    task = process_job.delay(request.payload, request.delay_seconds)
    return JobResponse(job_id=task.id, status="PENDING")


@app.get("/jobs/{job_id}", response_model=JobStatusResponse)
def get_job_status(job_id: str):
    # Look up the task result in Redis by its ID
    result = AsyncResult(job_id, app=celery_app)
    if result.state == "PENDING":
        return JobStatusResponse(job_id=job_id, status="PENDING")
    elif result.state == "FAILURE":
        return JobStatusResponse(
            job_id=job_id, status="FAILURE", result=str(result.info)
        )
    elif result.state == "SUCCESS":
        return JobStatusResponse(
            job_id=job_id, status="SUCCESS", result=str(result.result)
        )
    else:
        return JobStatusResponse(job_id=job_id, status=result.state)


@app.get("/jobs", response_model=list[JobStatusResponse])
def list_jobs():
    return []


@app.get("/health")
def health_check():
    return {"status": "healthy"}

What does this code do?

• submit_job calls process_job.delay() which pushes the task onto the Redis queue and returns immediately. The client gets a 202 Accepted response with a tracking ID instead of waiting for the job to finish.
• get_job_status uses AsyncResult to query Redis for the task's current state (PENDING, SUCCESS, or FAILURE). Clients poll this endpoint to track progress.
• list_jobs returns an empty list for now. A production version would query a database, but Redis result storage is sufficient for this project.
• health_check provides a lightweight endpoint for Docker health checks and load balancers to verify the API is alive.

What is a 202 Accepted response?

A 202 Accepted status code tells the client "I received your request and will process it later." This is the standard HTTP response for async task processing. The client does not block waiting for a result.

• Save app/main.py.

Seeing red squiggles in your editor?

Your editor may show import errors because the dependencies are not installed in your local environment yet. You will fix this in the next substep when you install the packages. The errors will disappear once the packages are available.

Help me fix import errors in my FastAPI project

✔️ Awesome, I've got everything!

Great. Double check that you have saved both files before moving on.

ⓧ I'd like to double check the full code

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from celery.result import AsyncResult

from app.worker import celery_app
from app.tasks import process_job

app = FastAPI(title="Distributed Task Queue", version="1.0.0")


class JobRequest(BaseModel):
    payload: str
    delay_seconds: int = 5


class JobResponse(BaseModel):
    job_id: str
    status: str


class JobStatusResponse(BaseModel):
    job_id: str
    status: str
    result: str | None = None


@app.post("/jobs", status_code=202, response_model=JobResponse)
def submit_job(request: JobRequest):
    task = process_job.delay(request.payload, request.delay_seconds)
    return JobResponse(job_id=task.id, status="PENDING")


@app.get("/jobs/{job_id}", response_model=JobStatusResponse)
def get_job_status(job_id: str):
    result = AsyncResult(job_id, app=celery_app)
    if result.state == "PENDING":
        return JobStatusResponse(job_id=job_id, status="PENDING")
    elif result.state == "FAILURE":
        return JobStatusResponse(
            job_id=job_id, status="FAILURE", result=str(result.info)
        )
    elif result.state == "SUCCESS":
        return JobStatusResponse(
            job_id=job_id, status="SUCCESS", result=str(result.result)
        )
    else:
        return JobStatusResponse(job_id=job_id, status=result.state)


@app.get("/jobs", response_model=list[JobStatusResponse])
def list_jobs():
    return []


@app.get("/health")
def health_check():
    return {"status": "healthy"}

Run the API locally and verify

The API code is complete. Now you will install the project dependencies and start the server locally to confirm everything wires together correctly.

• Create a virtual environment by running this command:

python3 -m venv .venv

• Activate the virtual environment:

macOS

source .venv/bin/activate

Windows

.venv\Scripts\activate

You should see (.venv) appear at the beginning of your terminal prompt.

• Install the project dependencies by running:

pip install -r requirements.txt

Install failing or taking too long?

Make sure your virtual environment is activated (you should see (.venv) in your prompt). If you see permission errors, do not use sudo. Deactivate and recreate the venv instead.

Help me fix pip install errors

• Start the FastAPI development server by running:

uvicorn app.main:app --port 8000

What does this command do?

uvicorn is the ASGI server that runs your FastAPI application. app.main:app tells it to look for the app variable inside app/main.py.

• Open your browser and navigate to http://localhost:8000/docs.

You should see the FastAPI interactive documentation page (Swagger UI) with all four routes listed: POST /jobs, GET /jobs/{job_id}, GET /jobs, and GET /health.

Page not loading or seeing an error?

Check your terminal for error messages. If you see ModuleNotFoundError, make sure your virtual environment is activated and dependencies are installed.

If port 8000 is already in use, stop the other process or change the port with --port 8001.

Help me troubleshoot my FastAPI server not starting

• Press Ctrl+C in your terminal to stop the server.

Your API layer is built and verified. Next up, you will add production-grade reliability to the Celery worker with retry logic, exponential backoff, and idempotent execution.

Build the Celery Worker with Reliability Patterns

Your FastAPI application can accept jobs and return tracking IDs. But without a worker to actually process those jobs, tasks sit in the queue forever.

In this step, you will build the Celery worker that picks up tasks from Redis and processes them. You will also add three production reliability patterns: retry with exponential backoff, idempotent execution, and structured logging.

In this step, get ready to:

• Create the task with retry logic and exponential backoff.
• Implement idempotent execution and structured logging.
• Configure periodic health check scheduling with Celery Beat.

Write the task with retry, idempotency, and logging

The app/tasks.py file currently exists as a stub. You will replace it with a fully implemented task that handles failures gracefully.

• In VS Code, open app/tasks.py and replace any existing content with the following code:

import time
import logging

from app.worker import celery_app

logger = logging.getLogger(__name__)


@celery_app.task(
    bind=True,
    max_retries=3,
    default_retry_delay=5,
    acks_late=True,
)
def process_job(self, payload: str, delay_seconds: int = 5):
    job_id = self.request.id
    logger.info(f"Task {job_id} started | payload={payload}")

    # Idempotency check: if task already completed, skip
    from celery.result import AsyncResult

    existing = AsyncResult(job_id, app=celery_app)
    if existing.state == "SUCCESS":
        logger.info(f"Task {job_id} already completed, skipping (idempotent)")
        return existing.result

    try:
        # Simulate processing
        time.sleep(delay_seconds)
        result = f"Processed: {payload}"
        logger.info(f"Task {job_id} completed | result={result}")
        return result
    except Exception as exc:
        logger.error(f"Task {job_id} failed | error={exc}")
        raise self.retry(exc=exc, countdown=2**self.request.retries)


@celery_app.task
def periodic_health_check():
    logger.info("Periodic health check: workers alive")
    return "healthy"

What does this code do?

This file defines two Celery tasks. The main task, process_job, contains three production reliability patterns:

• Retry with exponential backoff: bind=True gives the task access to self, which exposes self.request.retries (the current retry count, starting at 0). The countdown=2**self.request.retries formula doubles the wait time on each retry: 1 second, then 2, then 4. This prevents overwhelming a recovering service.
• Idempotent execution: Before doing any work, the task checks whether it has already completed successfully via AsyncResult. If a task gets re-delivered (network hiccup, worker crash), it returns the existing result instead of processing again.
• Structured logging: Every lifecycle event (started, completed, failed, skipped) is logged with the job_id and relevant context. This makes distributed debugging possible when you have multiple workers.

The acks_late=True setting tells Celery to acknowledge the message only after the task completes. If the worker crashes mid-task, Redis will re-deliver it to another worker.

Why exponential backoff instead of fixed delays?

When an external service goes down, all waiting tasks would retry at the same instant with a fixed delay. This creates a thundering herd that can keep the service down even longer.

Exponential backoff spreads retries over increasing intervals. Combined with the randomness that natural timing differences introduce, this gives the failing service breathing room to recover.

• Save app/tasks.py.

You will not see a change yet since the worker requires Redis to run. You will test this end-to-end in Step 4 when Docker Compose brings everything up.

Seeing import errors in your editor?

If VS Code underlines the from app.worker import celery_app line, make sure you have a Python interpreter selected that has celery installed. You can also ignore this for now. The code runs inside Docker, not locally.

Help me fix Python import errors in VS Code for a Docker-based project

Configure periodic health check scheduling

Celery Beat is a scheduler that dispatches tasks at regular intervals. You will configure it to run a health check every 60 seconds so you can confirm workers are alive.

• Create a new file called app/celerybeat_schedule.py inside the app folder with the following code:

from celery.schedules import crontab

beat_schedule = {
    "health-check-every-60s": {
        "task": "app.tasks.periodic_health_check",
        "schedule": 60.0,
    },
}

What does this schedule do?

The beat_schedule dictionary tells Celery Beat to dispatch the periodic_health_check task every 60 seconds. Each dispatch creates a message in Redis, which a worker picks up and executes. The worker then logs confirmation that it is alive.

• Create another new file called app/celeryconfig.py inside the app folder with the following code:

from app.celerybeat_schedule import beat_schedule

beat_schedule = beat_schedule

Why a separate config file?

Separating the beat schedule into its own module keeps the worker configuration clean. The celeryconfig.py file acts as the single entry point that Celery loads for all extra configuration. As your project grows, you can add more settings here without touching the worker or schedule definitions.

• Save both app/celerybeat_schedule.py and app/celeryconfig.py.
• Confirm the files exist by running:

ls app/

You should see __init__.py, celerybeat_schedule.py, celeryconfig.py, main.py, tasks.py, and worker.py in the output.

Missing a file?

Double check that you created the new files inside the app/ directory, not in the project root. The filenames must match exactly, including the underscore in celerybeat_schedule.py.

Help me find my missing files after running ls

✔️ Awesome, I've got everything!

Great. Double check you have saved all three files.

ⓧ I'd like to double check the full code

Here is the complete app/tasks.py file:

import time
import logging

from app.worker import celery_app

logger = logging.getLogger(__name__)


@celery_app.task(
    bind=True,
    max_retries=3,
    default_retry_delay=5,
    acks_late=True,
)
def process_job(self, payload: str, delay_seconds: int = 5):
    job_id = self.request.id
    logger.info(f"Task {job_id} started | payload={payload}")

    # Idempotency check: if task already completed, skip
    from celery.result import AsyncResult

    existing = AsyncResult(job_id, app=celery_app)
    if existing.state == "SUCCESS":
        logger.info(f"Task {job_id} already completed, skipping (idempotent)")
        return existing.result

    try:
        # Simulate processing
        time.sleep(delay_seconds)
        result = f"Processed: {payload}"
        logger.info(f"Task {job_id} completed | result={result}")
        return result
    except Exception as exc:
        logger.error(f"Task {job_id} failed | error={exc}")
        raise self.retry(exc=exc, countdown=2**self.request.retries)


@celery_app.task
def periodic_health_check():
    logger.info("Periodic health check: workers alive")
    return "healthy"

Here is app/celerybeat_schedule.py:

from celery.schedules import crontab

beat_schedule = {
    "health-check-every-60s": {
        "task": "app.tasks.periodic_health_check",
        "schedule": 60.0,
    },
}

Here is app/celeryconfig.py:

from app.celerybeat_schedule import beat_schedule

beat_schedule = beat_schedule

Your worker is ready with production-grade reliability patterns. Next up, you will containerize the entire system with Docker Compose so the API, worker, scheduler, and Redis broker all run together.

Containerize with Docker Compose

In the previous step, you built a Celery worker with retry logic, idempotency checks, and structured logging. Now it is time to package your entire distributed system into containers so every service starts together in one command.

Right now your API, worker, and Redis broker are separate pieces that need manual coordination. Docker Compose lets you define all of them in a single file and launch the full stack with one command.

In this step, get ready to:

• Write a Dockerfile that packages the Python application into a container image.
• Create a docker-compose.yml that orchestrates Redis, the API, the worker, the scheduler, and a monitoring dashboard.
• Start the full stack and verify jobs flow end-to-end.

Write the Dockerfile

A Dockerfile is a recipe that tells Docker how to build an image of your application. It specifies the base operating system, installs dependencies, copies your source code, and defines the default startup command.

• Create a new file called Dockerfile in the root of your project folder (the same level as requirements.txt).
• Paste the following code into Dockerfile:

FROM python:3.12-slim AS base

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

What does this Dockerfile do?

• FROM python:3.12-slim AS base starts from a lightweight Python 3.12 image.
• WORKDIR /app sets the working directory inside the container.
• COPY requirements.txt . and RUN pip install installs dependencies first so Docker can cache this layer.
• COPY . . copies the rest of your application code.
• CMD defines the default command that runs when a container starts from this image.

• Save Dockerfile.
• Confirm the file exists by running:

ls Dockerfile

You should see Dockerfile listed in the output.

Don't see the file?

Make sure you created the file at the project root, not inside the app/ folder. The filename must be exactly Dockerfile with a capital D and no file extension.

Help me find my Dockerfile

Create the Docker Compose configuration

Docker Compose reads a YAML file that describes every service in your system, how they connect, and what order they start in. Your compose file will define five services: a Redis broker, the FastAPI server, a Celery worker, a Celery Beat scheduler, and the Flower monitoring dashboard.

• Create a new file called docker-compose.yml in the root of your project folder.
• Paste the following code into docker-compose.yml:

services:
  redis:
    image: redis:7.4-alpine
    ports:
      - "6379:6379"
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 5s
      retries: 3

  api:
    build: .
    command: uvicorn app.main:app --host 0.0.0.0 --port 8000
    ports:
      - "8000:8000"
    depends_on:
      redis:
        condition: service_healthy
    environment:
      - CELERY_BROKER_URL=redis://redis:6379/0
      - CELERY_RESULT_BACKEND=redis://redis:6379/1

  worker:
    build: .
    command: celery -A app.worker worker --loglevel=info
    depends_on:
      redis:
        condition: service_healthy
    environment:
      - CELERY_BROKER_URL=redis://redis:6379/0
      - CELERY_RESULT_BACKEND=redis://redis:6379/1

  beat:
    build: .
    command: celery -A app.worker beat --loglevel=info --schedule=/tmp/celerybeat-schedule
    depends_on:
      redis:
        condition: service_healthy
    environment:
      - CELERY_BROKER_URL=redis://redis:6379/0
      - CELERY_RESULT_BACKEND=redis://redis:6379/1

  flower:
    build: .
    command: celery -A app.worker flower --port=5555
    ports:
      - "5555:5555"
    depends_on:
      redis:
        condition: service_healthy
    environment:
      - CELERY_BROKER_URL=redis://redis:6379/0
      - CELERY_RESULT_BACKEND=redis://redis:6379/1

What does each service do?

• redis pulls the official Redis 7.4 Alpine image and defines a health check so other services wait until Redis is ready.
• api builds your Dockerfile and runs the FastAPI server on port 8000.
• worker builds the same image but starts a Celery worker that consumes tasks from Redis.
• beat runs Celery Beat, which triggers your periodic health check task every 60 seconds.
• flower runs the Flower monitoring dashboard on port 5555 for real-time visibility into workers and tasks.

The depends_on block with condition: service_healthy tells Compose to wait until Redis passes its health check before starting each dependent service.

Why one image for multiple services?

All five application services share the same Dockerfile. Docker Compose builds the image once and reuses it across the api, worker, beat, and flower services. Each service just overrides the command to start a different process.

• Save docker-compose.yml.
• Confirm both files exist at the project root by running:

ls Dockerfile docker-compose.yml

You should see both filenames in the output.

Seeing a YAML syntax error?

YAML is sensitive to indentation. Make sure you use spaces (not tabs) and that each level is indented by exactly 2 spaces. Check that the services: key is at the top level with no indentation.

Help me fix my YAML syntax error

Start the full stack and verify

With both files in place, you can now launch your entire distributed system. Docker Compose will build the image, pull Redis, start all five services, and wire them together on a shared network.

• Make sure Docker Desktop is running.
• Build and start all services by running this command:

docker compose up --build

What does this command do?

The --build flag tells Compose to rebuild the Docker image before starting. Without it, Compose would reuse a previously cached image and miss your latest code changes.

You will see logs streaming from all five services in your terminal. Wait until you see lines indicating the worker is ready and the API is accepting connections.

• Open your browser and navigate to http://localhost:8000/docs.

You should see the FastAPI interactive documentation page with your three endpoints listed: POST /jobs, GET /jobs/{job_id}, and GET /jobs.

Page not loading?

Check the terminal logs for errors. If you see a connection refused error, the API container may still be waiting for Redis. Give it 15-20 seconds for the health check to pass.

If the build failed, look for a missing file error in the logs. Make sure Dockerfile and requirements.txt are both at the project root.

Help me troubleshoot my Docker Compose build

Now test the full flow by submitting a job and polling for its result.

• In the /docs UI, expand the POST /jobs section.
• Click Try it out.
• Replace the request body with:

{
  "payload": "hello world",
  "delay_seconds": 3
}

• Click Execute.

You should receive a 202 response with a job_id and status: "PENDING".

• Copy the job_id value from the response.
• Expand the GET /jobs/{job_id} section.
• Click Try it out, paste the job_id, and click Execute.

Wait 3 seconds (the delay you set), then execute again. The status should change to SUCCESS with a result of Processed: hello world.

What just happened?

Your API accepted the job request and immediately returned a tracking ID. Behind the scenes, it pushed a message to Redis.

The Celery worker (running in a separate container) picked up the message from Redis, processed it for 3 seconds, and stored the result back in Redis. When you polled the status endpoint, the API read the result from Redis and returned it to you.

Status stuck on PENDING?

Check the terminal where docker compose up is running. Look for worker log lines that mention a task starting. If you do not see any worker logs, the worker container may have crashed.

Run docker compose ps in a new terminal tab to check that all five containers show a running state.

Help me figure out why my task is stuck on PENDING

✔️ Awesome, I've got everything!

Great. Double check you have saved both Dockerfile and docker-compose.yml.

ⓧ I'd like to double check the full code

FROM python:3.12-slim AS base

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

services:
  redis:
    image: redis:7.4-alpine
    ports:
      - "6379:6379"
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 5s
      retries: 3

  api:
    build: .
    command: uvicorn app.main:app --host 0.0.0.0 --port 8000
    ports:
      - "8000:8000"
    depends_on:
      redis:
        condition: service_healthy
    environment:
      - CELERY_BROKER_URL=redis://redis:6379/0
      - CELERY_RESULT_BACKEND=redis://redis:6379/1

  worker:
    build: .
    command: celery -A app.worker worker --loglevel=info
    depends_on:
      redis:
        condition: service_healthy
    environment:
      - CELERY_BROKER_URL=redis://redis:6379/0
      - CELERY_RESULT_BACKEND=redis://redis:6379/1

  beat:
    build: .
    command: celery -A app.worker beat --loglevel=info --schedule=/tmp/celerybeat-schedule
    depends_on:
      redis:
        condition: service_healthy
    environment:
      - CELERY_BROKER_URL=redis://redis:6379/0
      - CELERY_RESULT_BACKEND=redis://redis:6379/1

  flower:
    build: .
    command: celery -A app.worker flower --port=5555
    ports:
      - "5555:5555"
    depends_on:
      redis:
        condition: service_healthy
    environment:
      - CELERY_BROKER_URL=redis://redis:6379/0
      - CELERY_RESULT_BACKEND=redis://redis:6379/1

Your distributed system is now running in containers. Next up, you will explore the Flower monitoring dashboard to get real-time visibility into your workers and tasks.

Add Monitoring with Flower

Your distributed system is processing tasks, but you have no window into what the workers are doing. When something goes wrong in production, you need real-time visibility into task states, worker health, and queue depths.

In this step, you will explore the Flower monitoring dashboard that is already running as part of your Docker Compose stack. You will submit multiple jobs and watch them flow through the system in real time.

In this step, get ready to:

• Understand the Flower service configuration in your compose file.
• Access the Flower dashboard and explore its worker and task views.
• Submit multiple jobs and observe real-time task activity in Flower.

Understand the Flower service in your compose file

Before jumping into the dashboard, take a moment to understand how Flower connects to your system.

• In your editor, open docker-compose.yml and scroll to the flower service at the bottom of the file.
• Find this section:

  flower:
    build: .
    command: celery -A app.worker flower --port=5555
    ports:
      - "5555:5555"
    depends_on:
      redis:
        condition: service_healthy
    environment:
      - CELERY_BROKER_URL=redis://redis:6379/0
      - CELERY_RESULT_BACKEND=redis://redis:6379/1

What does this service do?

Flower is an open-source web application for monitoring and managing Celery clusters. It provides real-time information about worker status and task execution.

• command: celery -A app.worker flower --port=5555 launches Flower using your Celery app configuration. The --port=5555 flag sets the web UI port.
• ports: "5555:5555" maps container port 5555 to your machine so you can access the dashboard in your browser.
• The CELERY_BROKER_URL environment variable tells Flower which Redis instance to monitor. It connects to the same broker your workers use.

Why add observability to a task queue?

In a distributed system, tasks can silently fail, pile up in queues, or retry endlessly without anyone noticing. Flower gives you a live dashboard to spot these problems before they become outages.

In production systems, this visibility is often extended with Prometheus and Grafana for alerting and historical metrics. Flower is a great starting point for real-time debugging.

Access the Flower dashboard

Your Flower container is already running as part of the stack you started in the previous step.

• Open your browser and navigate to http://localhost:5555.
• You should see the Flower dashboard with a navigation bar showing links for Dashboard, Tasks, and Broker.
• Click Dashboard in the navigation bar.
• You should see your celery@worker listed with its status showing as Online.

What can you see on the Dashboard page?

The Dashboard page shows all connected workers with their hostname, status, active task count, tasks processed, and tasks failed. This is your at-a-glance health check for the entire cluster.

Click a worker name to drill into its details: registered tasks, active queues, resource usage, and configuration.

• Click Tasks in the navigation bar.
• This page shows task history with columns for name, UUID, state, received time, and runtime. You may already see a periodic_health_check task from the Celery Beat scheduler.

Don't see the Flower dashboard?

• Make sure your Docker Compose stack is still running. Switch back to your terminal and check that all five containers are up.
• Confirm nothing else is using port 5555 on your machine.
• If the page loads but shows no workers, wait 10-15 seconds and refresh. Flower needs a moment to receive the first heartbeat from workers.

Flower dashboard not loading?

Submit multiple jobs and observe them in Flower

Now you will generate some real traffic and watch it flow through the dashboard in real time.

• Open a new browser tab and navigate to http://localhost:8000/docs.
• Expand the POST /jobs endpoint and click Try it out.
• Submit a job with the following request body:

{
  "payload": "first-task",
  "delay_seconds": 3
}

• Click Execute.
• Submit two more jobs quickly by changing the payload value each time (e.g. second-task and third-task).
• Switch back to the Flower tab at http://localhost:5555.
• Click Tasks in the navigation bar.

You should see your three process_job tasks listed with their state transitioning from STARTED to SUCCESS as each completes.

• Click on any task UUID to see its full details: the arguments passed, received time, started time, runtime, and result.
• Click Broker in the navigation bar to see queue statistics, including the celery queue and its current message count.

What does the Broker page show?

The Broker page displays all active queues with their current message counts. When tasks pile up faster than workers can process them, you will see the message count climb. This is your early warning signal for capacity issues.

Stress-test your system

Try submitting 5-10 jobs with delay_seconds set to 10 and watch the queue depth grow on the Broker page while the single worker processes them one at a time. This demonstrates why scaling workers horizontally matters in production.

You now have full observability over your distributed task queue. You can see every task that enters the system, track its lifecycle from pending to success or failure, and monitor queue depths in real time.

Secret mission

Priority Queues and Task Routing

Right now every task lands in the same queue and gets processed in FIFO order. What happens when a critical payment job is stuck behind a thousand thumbnail-generation tasks? In this mission, you'll add priority-based routing so urgent work jumps to a dedicated high-priority worker.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

This project runs entirely locally via Docker, so there are no ongoing financial costs. The containers do consume RAM and CPU while running.

Resources you used:

• Docker containers (redis, api, worker, worker-high, beat, flower) running locally.
• Project directory with Python source files, Dockerfile, and docker-compose.yml.

Keep everything running

No action needed. Choose this if you are still actively building or experimenting with the task queue.

• Your six containers continue running and using system resources (RAM and CPU).
• You can submit jobs at http://localhost:8000/docs and monitor workers at http://localhost:5555 any time.
• Docker Desktop remains installed on your machine for future projects.

Pause - I'll come back to this later

Shut down running containers to free up memory without removing them. You can resume later without rebuilding.

• In your terminal, navigate to your project directory and stop all containers by running:

docker compose stop

• When you are ready to resume, start the containers again by running:

docker compose start

Your containers, networks, and volumes are preserved. Everything picks up where you left off.

Delete - I don't want to use this again

Remove all containers, networks, and volumes created by this project. This frees up all system resources.

• In your terminal, navigate to your project directory and remove all containers, networks, and volumes by running:

docker compose down -v

• Remove the project directory from your Desktop. On macOS or Linux, run:

cd ~/Desktop
rm -rf distributed-task-queue

• On Windows, run:

cd ~\Desktop
rmdir /s /q distributed-task-queue

• Optionally, remove the Docker images built for this project by running:

docker image prune -a

What about Docker Desktop?

Docker Desktop stays installed on your machine. It is useful for future projects and does not consume resources when idle.

That's a Wrap!

That's a Wrap!

Nice work. You just built a production-grade distributed task queue from scratch using FastAPI, Celery, and Redis.

You've learned how to:

• Built a FastAPI REST API that offloads long-running work to background workers and immediately returns 202 Accepted responses with tracking IDs.
• Configured Celery workers with production reliability patterns including retry with exponential backoff and idempotent task execution.
• Containerized a multi-service architecture using Docker Compose with real-time observability through the Flower monitoring dashboard.
• Secret Mission: Implemented priority-based task routing with dedicated workers consuming from separate queues based on urgency.

Ready to quiz yourself?