Build a Dead Letter Queue with PySpark

Route bad records to a Dead Letter Queue using PySpark and Docker.

Introduction

30 Second Summary

Every data pipeline eventually encounters bad records. A misspelled field, a negative price, a missing ID. The question is whether those broken records silently corrupt your results or get caught and explained.

In this project, you will build a local PySpark batch pipeline that reads customer orders, validates each one against business rules, and routes failures to a Dead Letter Queue with metadata explaining exactly what went wrong.

What You'll Build

You run one command and watch your pipeline split 10 orders into two clean outputs: valid records ready for downstream use, and broken records quarantined with human-readable error reasons you can open and inspect immediately.

By the end of this project, you'll have:

A batch pipeline that processes 10 customer orders in one command, accepting 6 valid records and quarantining 4 broken ones into a separate Dead Letter Queue folder.
Enriched error metadata on every quarantined record that tells you exactly what rule failed and when the record was caught.
A fully containerized setup running inside Docker with zero local installs beyond Docker itself.
Secret Mission: A reprocessing step that reads from the DLQ, fixes correctable records, and routes them back to the valid output.

Are there any prerequisites?

You need basic Python familiarity and Docker installed and running on your machine. No Java, no Spark install, no cloud account. Docker handles everything else.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

When a data pipeline encounters a bad record, the tempting solution is to just drop it. But dropped records vanish without a trace. Nobody knows what was lost or why. A Dead Letter Queue captures every failed record alongside the reason it failed, so you can investigate, fix, and reprocess later.

Set Up Your Spark Environment

To build a data pipeline that separates good records from bad ones, you need a data processing engine. Apache Spark is one of the most widely used engines in production data engineering.

Docker gives you a zero-install path to a full Spark environment. One image download bundles Java, Spark, and Python together so you can start processing data immediately.

In this step, get ready to:

Pull the official Spark Docker image.
Create the project folder and verify Spark works.
Create the input data file with sample orders.

Pull the official Spark Docker image

The Spark image contains everything your pipeline needs to run. Pulling it now means every future command starts instantly without waiting for downloads.

Open a terminal in VS Code by pressing Ctrl+` (backtick key, top-left of your keyboard).
Pull the Spark image by running this command:

docker pull spark:python3

You should see download progress bars as Docker pulls each layer. When complete, your terminal returns to the prompt.

What did this download?

The spark:python3 image is about 770 MB. It contains Apache Spark 4.1.2 with Python 3 support, plus the Java runtime Spark needs.

Everything runs inside this container. You do not need to install Java or Spark on your own machine.

Why Docker instead of pip install?

PySpark can be installed via pip, but it requires Java installed separately. On Windows especially, configuring Java paths is a common source of frustration.

The Docker image bundles Java, Spark, and Python in one download. Zero path configuration needed.

Seeing a connection error or 'daemon not running'?

Make sure Docker Desktop is running. Look for the Docker whale icon in your system tray (Windows) or menu bar (macOS).

If it is not there, open Docker Desktop and wait for the engine to start before retrying the pull command.

Help me fix my Docker pull error

Create the project folder and verify Spark works

Move to your Desktop by running this command:

cd ~/Desktop

Create your project folder and move into it by running these commands:

mkdir spark-dlq
cd spark-dlq

Open this folder in VS Code.
Click File in the top menu bar.
Click Open Folder.
Navigate to your Desktop, select the spark-dlq folder, and click Open.

VS Code now shows spark-dlq as your workspace. If prompted to trust the folder, click Yes, I trust the authors.

Open a new terminal in VS Code by pressing Ctrl+` (backtick).

Before you run the next command, what do you think Spark will print when you ask for its version?

Verify Spark works inside Docker by running this command:

docker run --rm spark:python3 /opt/spark/bin/spark-submit --version

You should see output containing 4.1.2 in the version line. This confirms Spark is accessible and ready to process data.

What does this command do?

docker run --rm starts a temporary container that deletes itself after exiting.
spark:python3 is the image you just pulled.
/opt/spark/bin/spark-submit --version runs Spark's built-in version reporter inside the container.

Not seeing the version output?

If you see an error about the image not being found, make sure the docker pull spark:python3 command from the previous substep completed successfully.

If the command hangs, Docker Desktop may need more memory. Open Docker Desktop settings and increase the memory limit to at least 4 GB.

Help me troubleshoot my Spark version check

Create the input data file

Your pipeline needs sample data to process. You will create a small file with 10 customer orders: 6 valid and 4 intentionally broken.

In the VS Code Explorer sidebar, click the new file icon (or press Ctrl+N).
Save the file immediately as orders.json in the spark-dlq folder.
Paste the following 10 records into orders.json:

{"order_id": "ORD001", "customer_name": "Alice", "product": "Laptop", "price": 999.99, "quantity": 1}
{"order_id": "ORD002", "customer_name": "Bob", "product": "Mouse", "price": 25.50, "quantity": 2}
{"order_id": null, "customer_name": "Charlie", "product": "Keyboard", "price": 75.00, "quantity": 1}
{"order_id": "ORD004", "customer_name": "", "product": "Monitor", "price": 300.00, "quantity": 1}
{"order_id": "ORD005", "customer_name": "Eve", "product": "Headphones", "price": -49.99, "quantity": 1}
{"order_id": "ORD006", "customer_name": "Frank", "product": "", "price": 120.00, "quantity": 3}
{"order_id": "ORD007", "customer_name": "Grace", "product": "Webcam", "price": 89.00, "quantity": 1}
{"order_id": "ORD008", "customer_name": "Hank", "product": "USB Hub", "price": 35.00, "quantity": 2}
{"order_id": "ORD009", "customer_name": "Ivy", "product": "SSD", "price": 110.00, "quantity": 1}
{"order_id": "ORD010", "customer_name": "Jack", "product": "Cable", "price": 12.99, "quantity": 5}

What are the broken records?

This is a JSON Lines file (one JSON object per line). PySpark reads this format natively without any extra configuration.

Four of the 10 records are intentionally broken:

Line 3: order_id is null (missing ID).
Line 4: customer_name is an empty string.
Line 5: price is negative (-49.99).
Line 6: product is an empty string.

Save orders.json.
Confirm the file exists by running this in your terminal:

ls

You should see orders.json listed in the output.

✔️ Awesome, I've got everything!

Great. Make sure you have saved orders.json before moving on.

ⓧ I'd like to double check the full code

Your orders.json file should contain exactly these 10 lines:

{"order_id": "ORD001", "customer_name": "Alice", "product": "Laptop", "price": 999.99, "quantity": 1}
{"order_id": "ORD002", "customer_name": "Bob", "product": "Mouse", "price": 25.50, "quantity": 2}
{"order_id": null, "customer_name": "Charlie", "product": "Keyboard", "price": 75.00, "quantity": 1}
{"order_id": "ORD004", "customer_name": "", "product": "Monitor", "price": 300.00, "quantity": 1}
{"order_id": "ORD005", "customer_name": "Eve", "product": "Headphones", "price": -49.99, "quantity": 1}
{"order_id": "ORD006", "customer_name": "Frank", "product": "", "price": 120.00, "quantity": 3}
{"order_id": "ORD007", "customer_name": "Grace", "product": "Webcam", "price": 89.00, "quantity": 1}
{"order_id": "ORD008", "customer_name": "Hank", "product": "USB Hub", "price": 35.00, "quantity": 2}
{"order_id": "ORD009", "customer_name": "Ivy", "product": "SSD", "price": 110.00, "quantity": 1}
{"order_id": "ORD010", "customer_name": "Jack", "product": "Cable", "price": 12.99, "quantity": 5}

Don't see orders.json in the ls output?

Make sure your terminal is in the spark-dlq folder. Run pwd to check your current directory.

If you saved the file to a different location, drag it into the spark-dlq folder in the VS Code Explorer sidebar.

Help me find my orders.json file

Your Spark environment is ready and your sample data is in place. Next up, you will write a pipeline that reads these orders and see what happens when bad data slips through unchecked.

Process Orders Without Validation

Your data file has 10 customer orders sitting in orders.json. Some are perfectly valid. Others have missing fields, empty strings, or nonsensical values like negative prices.

In a real data pipeline, you would want to catch those bad records before they reach your analytics or reports. But first, let's see what happens when you process everything without any validation at all.

In this step, get ready to:

Write a PySpark script that reads all orders and prints them.
Run the script via Docker and observe how bad records silently mix in with valid data.

Write a basic PySpark script

You need a Python script that starts a Spark session, reads the JSON file, and displays the data. This is the simplest possible pipeline: read everything, show everything.

In VS Code, create a new file called pipeline.py in your spark-dlq folder.
Add the following code to pipeline.py:

from pyspark.sql import SparkSession
from pyspark.sql.functions import when, col, lit, current_timestamp

# Create Spark session
spark = SparkSession.builder.appName("DLQ Pipeline").master("local[*]").getOrCreate()
spark.sparkContext.setLogLevel("WARN")

# Read orders
orders_df = spark.read.json("/app/orders.json")

print("=== All Orders (Raw Input) ===")
orders_df.show(truncate=False)

spark.stop()

What does this code do?

SparkSession.builder creates a local Spark instance named "DLQ Pipeline" that uses all available CPU cores (local[*]).
setLogLevel("WARN") hides the verbose Spark startup logs so you can focus on your data output.
spark.read.json("/app/orders.json") reads the JSON Lines file into a DataFrame. The path /app/ is where Docker will mount your project folder.
.show(truncate=False) prints all rows to the terminal without cutting off long values.
The extra imports (when, col, lit, current_timestamp) are not used yet. You will need them in the next step when you add validation.

Save pipeline.py.

✔️ Awesome, I've got everything!

Great. Make sure you have saved pipeline.py before moving on to run it.

ⓧ I'd like to double check the full code

Here is the complete pipeline.py for this step:

from pyspark.sql import SparkSession
from pyspark.sql.functions import when, col, lit, current_timestamp

# Create Spark session
spark = SparkSession.builder.appName("DLQ Pipeline").master("local[*]").getOrCreate()
spark.sparkContext.setLogLevel("WARN")

# Read orders
orders_df = spark.read.json("/app/orders.json")

print("=== All Orders (Raw Input) ===")
orders_df.show(truncate=False)

spark.stop()

Run the pipeline with Docker

Your script reads from /app/orders.json inside the container. You need to mount your project folder to /app so Docker makes your local files visible to Spark.

Before you run this, think about it: you have 4 broken records in your data. What do you think will happen to them?

Run your pipeline by entering this command in your terminal:

macOS

docker run --rm -v "$(pwd)":/app spark:python3 /opt/spark/bin/spark-submit /app/pipeline.py

Windows

docker run --rm -v "%cd%":/app spark:python3 /opt/spark/bin/spark-submit /app/pipeline.py

PowerShell users

If you are using PowerShell instead of CMD, replace %cd% with ${PWD}.

You should see a table printed with all 10 orders. Look closely at the output. The broken records are right there in the table, mixed in with the good ones.

A record with a null order_id sits quietly next to valid orders. An empty customer_name shows as a blank string. A negative price of -49.99 raises no alarm. Nothing in the output tells you these records are problematic.

Why is this a problem?

In a real system, this data would flow downstream into reports, dashboards, or billing calculations. A negative price could corrupt revenue totals. A missing customer name could break a join with your CRM data.

The pipeline succeeded, but it silently passed along garbage. That is exactly what a Dead Letter Queue prevents.

Not seeing any output?

Make sure Docker Desktop is running. If the command hangs, Docker might still be starting up.
Check that you are running the command from inside the spark-dlq folder. The volume mount maps your current directory into the container.
If you see an error mentioning orders.json not found, confirm the file is saved in the same folder as pipeline.py.

Help me debug why my Spark pipeline isn't producing output in Docker.

Your pipeline ran successfully, but it has a serious blind spot. In the next step, you will add validation rules that catch these broken records and separate them from the good ones.

Add Validation and Split Records

In the previous step, you saw the problem firsthand: bad records mixed in silently with valid data. Nothing in the output warned you that a record had a null order_id or a negative price.

Now you will add validation rules that inspect each record and tag anything that breaks a business rule. Once tagged, you can split your data into two clean groups: records that passed and records that failed.

In this step, get ready to:

Define validation rules that flag broken records with a specific error reason.
Split the DataFrame into valid records and a Dead Letter Queue.
Run the pipeline and confirm the split produces 6 valid and 4 invalid records.

Define validation rules as filter conditions

Each record needs to pass four business rules: it must have an order_id, a customer_name, a positive price, and a product. The first rule that fails becomes that record's error_reason.

In VS Code, open pipeline.py.
Find the line orders_df.show(truncate=False) and add the following code below it, before spark.stop():

# Validation: assign error_reason for the first failing rule
validated_df = orders_df.withColumn(
    "error_reason",
    when(col("order_id").isNull() | (col("order_id") == ""), lit("missing order_id"))
    .when(col("customer_name").isNull() | (col("customer_name") == ""), lit("missing customer_name"))
    .when(col("price") <= 0, lit("invalid price"))
    .when(col("product").isNull() | (col("product") == ""), lit("missing product"))
    .otherwise(lit(None))
)

What does this code do?

The withColumn call adds a new column called error_reason to every row.

Each .when() checks one business rule in order. The first rule that fails wins.
If a record passes all four checks, .otherwise(lit(None)) sets the column to null, meaning the record is clean.
Only the first failing rule is recorded. This keeps the logic simple and avoids multi-error complexity.

Why when/otherwise instead of a Python function?

PySpark can push when/otherwise chains directly into Spark's optimized execution engine. A Python function (UDF) forces Spark to send every row through the Python interpreter one by one, which is much slower on large datasets.

Save pipeline.py.

Split the DataFrame into valid and DLQ records

Now that every record has an error_reason (either a specific message or null), you can filter into two groups.

Add the following code directly below the validation block you just pasted, still above spark.stop():

# Split into valid and DLQ
valid_df = validated_df.filter(col("error_reason").isNull()).drop("error_reason")
dlq_df = validated_df.filter(col("error_reason").isNotNull())

print("=== Valid Orders ===")
valid_df.show(truncate=False)

print("=== Dead Letter Queue ===")
dlq_df.show(truncate=False)

What does this code do?

valid_df keeps only rows where error_reason is null (clean records), then drops the column since it is no longer needed.
dlq_df keeps rows where error_reason is not null. These are your quarantined records.
Both DataFrames are printed with .show(truncate=False) so you can see full values without cutoff.

Save pipeline.py.

Run the pipeline and verify the split

Your pipeline now validates every record and separates them into two groups. Before you run it, think: how many of the 10 records in orders.json do you expect to fail validation?

Run the pipeline using the Docker command for your operating system:

macOS

docker run --rm -v "$(pwd)":/app spark:python3 /opt/spark/bin/spark-submit /app/pipeline.py

Windows

docker run --rm -v "%cd%":/app spark:python3 /opt/spark/bin/spark-submit /app/pipeline.py

Your output now shows three tables. First, all 10 raw records. Then 6 valid orders. Then 4 records in the Dead Letter Queue, each with a specific error_reason:

The record with a null order_id shows missing order_id.
The record with an empty customer_name shows missing customer_name.
The record with -49.99 shows invalid price.
The record with an empty product shows missing product.

Seeing an error instead of the tables?

A SyntaxError usually means a missing dot before .when( or a missing closing parenthesis. Each .when() must chain directly off the previous line.
If you see AnalysisException, check that column names in your code match exactly: order_id, customer_name, price, product.
If you see all 10 records in the Valid Orders table, make sure your validation code is between orders_df.show() and spark.stop(), not after spark.stop().

Help me debug my PySpark validation pipeline

✔️ Awesome, I've got everything!

Great. Double check you have saved pipeline.py and your output shows exactly 6 valid orders and 4 DLQ records.

ⓧ I'd like to double check the full code

Scroll up and confirm your pipeline.py contains these sections in order:

Imports (SparkSession, when, col, lit, current_timestamp).
Spark session creation with appName("DLQ Pipeline") and setLogLevel("WARN").
Read orders.json and print all raw orders.
Validation block: validated_df = orders_df.withColumn("error_reason", ...) with four .when() checks and one .otherwise(lit(None)).
Split: valid_df filtered on null error_reason (with .drop("error_reason")), dlq_df filtered on not-null error_reason.
Print both DataFrames with .show(truncate=False).
spark.stop() at the very end.

Your pipeline no longer lets bad data slip through silently. Every broken record is tagged with exactly what went wrong. Next up, you will write these tagged records to a proper Dead Letter Queue folder on disk, enriched with timestamps so you can trace when each failure was caught.

Write the DLQ With Metadata

Your pipeline now catches bad records and separates them from good ones. Identifying the problem is only half the pattern.

A production Dead Letter Queue stores broken records with metadata so someone can investigate and fix them later. In this step, you'll enrich each failed record with a timestamp and write everything to disk as JSON files.

In this step, get ready to:

Enrich DLQ records with an error timestamp.
Write valid orders and enriched DLQ records to separate output folders.
Inspect the DLQ output to confirm metadata is attached.

Enrich DLQ records and write to disk

Knowing that a record failed is not enough for a real investigation. You also need to know WHEN it failed so you can correlate with deployments, data refreshes, or upstream changes.

In pipeline.py, find the spark.stop() line at the bottom of the file.
Add the following code directly above spark.stop():

# Enrich DLQ with timestamp
dlq_enriched = dlq_df.withColumn("error_timestamp", current_timestamp())

# Write outputs
valid_df.write.mode("overwrite").json("/app/output/valid_orders")
dlq_enriched.write.mode("overwrite").json("/app/output/dlq")

print("Pipeline complete. Valid orders written to output/valid_orders/")
print("DLQ records written to output/dlq/")

What does this code do?

dlq_enriched = dlq_df.withColumn("error_timestamp", current_timestamp()) adds a new column recording the exact moment each bad record was flagged.
valid_df.write.mode("overwrite").json(...) saves the 6 clean orders to a JSON folder. The overwrite mode means re-running the pipeline replaces old output cleanly.
dlq_enriched.write.mode("overwrite").json(...) saves the 4 broken records with their error_reason and error_timestamp columns intact.

Save pipeline.py.

Why write to /app/ paths?

The Docker volume mount maps your local spark-dlq folder to /app inside the container. When Spark writes to /app/output/dlq, the files appear in output/dlq/ on your local machine.

✔️ Awesome, I've got everything!

Great. Double check you saved pipeline.py and the new code sits directly above spark.stop().

ⓧ I'd like to double check the full code

Your pipeline.py should now contain these sections in order. Scroll through your file and confirm each is present:

Imports (SparkSession, when, col, lit, current_timestamp).
Spark session creation and log level set to WARN.
Read orders.json and print all orders.
Validation logic with withColumn("error_reason", ...).
Split into valid_df and dlq_df, both printed.
Enrich DLQ with error_timestamp.
Write valid_df to /app/output/valid_orders and dlq_enriched to /app/output/dlq.
Two print statements confirming completion.
spark.stop() at the very end.

Code not matching up?

Make sure the enrichment and write lines are between the dlq_df.show(truncate=False) line and spark.stop(). If they appear after spark.stop(), Spark will have already shut down and the writes will fail.

help me place the enrichment and write code in the correct location in pipeline.py

Run the pipeline and inspect the DLQ output

Before you run this, what do you expect to find on disk after the pipeline finishes?

Run your pipeline using the same Docker command from earlier:

macOS

docker run --rm -v "$(pwd)":/app spark:python3 /opt/spark/bin/spark-submit /app/pipeline.py

Windows

docker run --rm -v "%cd%":/app spark:python3 /opt/spark/bin/spark-submit /app/pipeline.py

You should see the familiar Valid Orders and Dead Letter Queue tables printed, followed by two new lines:

Pipeline complete. Valid orders written to output/valid_orders/
DLQ records written to output/dlq/

Confirm the output folders exist by running:

ls output/

You should see two folders: dlq and valid_orders.

Inspect the DLQ folder contents by running:

ls output/dlq/

You should see one or more part-*.json files and a _SUCCESS marker. Spark writes data as part files inside a folder rather than a single JSON file.

Open one of the .json files in the output/dlq/ folder by clicking it in the VS Code Explorer sidebar.

Each record contains the original order fields plus error_reason and error_timestamp. You can now see exactly what went wrong and when it was caught.

What should I see in a DLQ record?

A typical DLQ record looks like this (fields may be in any order):

{"order_id":"ORD005","customer_name":"Eve","product":"Headphones","price":-49.99,"quantity":1,"error_reason":"invalid price","error_timestamp":"2026-07-16T..."}

The error_reason tells you the rule that failed. The error_timestamp tells you when the pipeline flagged it.

Don't see the output folders?

Make sure you ran the Docker command from inside your spark-dlq folder. The volume mount maps your current directory to /app, so the output only appears where you ran the command.

If you see an error about the path already existing, confirm you included .mode("overwrite") in both write lines.

help me troubleshoot why my DLQ pipeline is not writing output files to disk

Your pipeline now implements the full Dead Letter Queue pattern. Bad records never silently corrupt your data, and every failure is traceable with a reason and a timestamp.

Secret mission

Reprocess the DLQ

In production, not all DLQ records are permanently broken. Some failures are fixable. Build a reprocessing pipeline that reads quarantined records, corrects what it can, and routes recovered data back to the valid output.

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally using Docker, so there are no ongoing costs.

Resources you used:

Docker image spark:python3 (~770 MB on disk).
Project folder spark-dlq on your Desktop (contains orders.json, pipeline.py, reprocess_dlq.py, and all output folders).

Keep everything running

No action needed. Choose this if you plan to experiment more with PySpark or extend the pipeline.

The spark:python3 Docker image is useful for any future Spark project. It takes up ~770 MB on disk.
The output folders contain small JSON files and use negligible disk space.
Nothing is actively running. All Docker containers were started with --rm, so they removed themselves after each pipeline run.

Pause - I'll come back to this later

Nothing is actively running since all Docker containers were started with the --rm flag. Your files and the Docker image remain on disk until you decide to delete them.

Your project is already paused. There are no running processes to stop and no ongoing costs.
When you want to resume, open the spark-dlq folder in VS Code and run your Docker commands again.

Delete - I don't want to use this again

Remove all project resources and start fresh. This frees up ~770 MB of disk space from the Docker image plus the small project files.

Remove the Docker image by running:

docker rmi spark:python3

Delete the project folder. On macOS/Linux, run:

rm -rf ~/Desktop/spark-dlq

On Windows (Command Prompt), run:

rmdir /s /q %USERPROFILE%\Desktop\spark-dlq

After both commands, your machine is back to its pre-project state.

That's a Wrap!

Nice work! You just built a production-grade data pipeline that implements the Dead Letter Queue pattern using PySpark and Docker.

You've learned how to:

Build a PySpark batch pipeline that reads JSON data, validates each record against business rules, and splits records into valid and invalid DataFrames.
Implement the Dead Letter Queue pattern so bad records are quarantined with error_reason and error_timestamp metadata instead of silently corrupting your data.
Run an entire Spark pipeline inside Docker with zero local installs or cloud infrastructure.
Secret Mission: Built a DLQ reprocessing pipeline that reads quarantined records, applies fixes to correctable errors, and routes recovered records back to valid output.

Ready to quiz yourself?