Build a Serverless Retail ETL Pipeline

Introduction

30 Second Summary

A shop can receive dozens of orders that look fine until one broken date or impossible quantity slips into the totals. Without a reliable checkpoint, one bad row can distort every chart built from it.

In this project, you will build a serverless retail ETL pipeline on AWS that rejects malformed orders before they reach analytics. Clean records become a star schema that powers a downloadable sales dashboard.

What You'll Build

Your finished workflow turns a corrected batch of retail orders into a private dashboard showing 6 orders with $465.00 in revenue.

By the end of this project, you'll have:

• A validation gate that returns a structured report for malformed orders, including the zero quantity and malformed date in the sample.
• A successful AWS Step Functions workflow that leaves fact_orders, dim_customers, and dim_products ready to browse in AWS Glue Data Catalog.
• A downloadable retail dashboard from private Amazon S3 storage showing 6 orders with $465.00 in revenue.
• Secret Mission: A quarantine branch that saves rejected validation reports to quarantine/rejected-orders.json before the workflow fails.

What do I need for this project?

You need an eligible active AWS Free Plan plus a Windows computer with Microsoft Edge. The AWS Management Console handles the whole build, so no local coding tools are required.

Before We Start

Before any hands-on work begins, this is your moment to commit to the serverless ETL pipeline you are about to build. A clear purpose connects the raw retail orders to the trusted dashboard you want to produce.

Set Up a Free Private Workspace

Your retail pipeline eventually connects several AWS services. Keeping them in one AWS Region prevents broken references across the pipeline.

The raw orders also need a private storage location before any transformation begins.

In this step, you'll confirm your Free Plan. You'll create a private Amazon S3 bucket in your selected Region.

In this step, get ready to:

• Sign in to the AWS Management Console with an eligible active Free Plan.
• Choose one AWS Region for every project resource.
• Create a private Amazon S3 bucket for the pipeline.

Confirm your Free Plan and Region

Microsoft Edge gives you browser access to the AWS Management Console. This project needs no local coding tools.

Billing checks can feel high-stakes. This project stays at $0 while you use an eligible active AWS Free Plan.

• Press the Windows key to open the search bar.
• Type Microsoft Edge into the search bar.
• Press Enter to open Microsoft Edge.
• Search for AWS Management Console.
• Open the official AWS result to reach the console sign-in page.
• Sign in to the AWS account you plan to use for this project.
• Confirm the account plan information shows an active Free Plan.
• Continue only with an account that has an eligible active Free Plan.

Can't Confirm an Active Free Plan?

Pause before creating resources. The Free Plan ends after six months or when its credits run out.

Keep the account on the Free Plan. Do not upgrade to a Paid Plan for this project.

Need help checking your plan?

Your selected Region becomes the home for every resource in this project. Using one Region keeps later service connections consistent.

• Choose one AWS Region from the menu in the top-right area of the console.
• Record the selected Region here: your selected AWS Region.

You'll see your selected Region in the console header. Keep this Region selected throughout the project.

Create the private S3 bucket

An S3 bucket stores objects under a unique name. This bucket becomes the private landing zone for your pipeline data.

• Enter Amazon S3 in the console search bar.
• Select Amazon S3 from the search results.
• Choose General purpose buckets.
• Choose Create bucket.

How Should I Name the Bucket?

Amazon S3 bucket names must be globally unique. Use 3 to 63 characters.

Use lowercase letters, numbers, periods, or hyphens. Start with a letter or number.

End with a letter or number. The pattern retail-etl-your-initials-randomdigits reduces the chance of another account using the same name.

New buckets enable all four Block Public Access settings by default. Keeping these settings enabled prevents public access to your pipeline data.

• Choose a globally unique lowercase bucket name based on this pattern: retail-etl-your-initials-randomdigits.
• Enter retail-etl-your-initials-randomdigits in the Bucket name field.
• Confirm the bucket Region matches your selected AWS Region.
• Keep all four Block Public Access settings enabled.
• Choose Create bucket.

That's your first AWS resource ready. Your pipeline now has a private storage boundary for its data.

Verify the private bucket

The final check proves that the exact bucket exists in the selected Region. It also confirms that the bucket starts empty with public access blocked.

Before you refresh, where do you expect your new bucket to appear?

• Return to General purpose buckets.
• Refresh the page in Microsoft Edge.
• Find retail-etl-your-initials-randomdigits in the bucket list.
• Select retail-etl-your-initials-randomdigits.
• Confirm the Objects area contains no objects.
• Confirm the Block Public Access summary shows all four settings enabled.

You'll see retail-etl-your-initials-randomdigits in General purpose buckets. Its Objects area is empty.

The Block Public Access summary confirms all four settings are enabled. Your private workspace is ready for pipeline data.

Don't See the Private Bucket?

• Check that the console still shows your selected AWS Region.
• Refresh General purpose buckets after confirming the Region.
• Choose a different random suffix if Amazon S3 reports that the bucket name is unavailable.
• Enable any disabled setting in the bucket's Block Public Access section before continuing.

Help me troubleshoot my S3 bucket setup.

Your free private workspace is ready. Next, you'll build the validation gate that catches malformed retail orders.

Validate Retail Orders

Your private Amazon S3 bucket is ready to receive pipeline data. A reliable ETL pipeline checks source records before storage.

This step adds a validation gate with AWS Lambda. You will run a deliberately flawed order batch through it so the failure becomes visible before any transformation starts.

In this step, get ready to:

• Create the validate-retail-orders Lambda function.
• Deploy the order validation code.
• Run the invalid-orders test event.

Create the validator function

Creating a cloud function can feel like a billing step. Your active AWS Free Plan covers this project's small workload while your account remains on the Free Plan.

• Use the search in the AWS Management Console from earlier to open AWS Lambda.
• Confirm the console still shows the Region you selected earlier.
• Choose Create function.
• Choose Author from scratch.
• Enter validate-retail-orders under Function name.
• Select Python 3.14 under Runtime.
• Choose Create function.

You will see the function page with lambda_function.py in the Code area. The managed runtime uses lambda_function.lambda_handler as the handler.

Build the input contract

An input contract defines the fields that every retail order must contain. The handler also needs a structured response when the incoming batch is missing.

• Select all existing code in lambda_function.py.
• Replace it with the field contract and empty-input guard below:

from datetime import datetime

REQUIRED_FIELDS = (
    "order_id",
    "order_date",
    "customer_id",
    "customer_name",
    "segment",
    "product_id",
    "product_name",
    "category",
    "quantity",
    "unit_price",
)


def lambda_handler(event, context):
    orders = event.get("orders")
    errors = []

    if not isinstance(orders, list) or not orders:
        return {
            "valid": False,
            "record_count": 0,
            "errors": ["orders must be a non-empty list"],
        }

What does this code do?

• The REQUIRED_FIELDS tuple defines the contract for each retail order.
• The event.get("orders") call reads the order batch without assuming that the key exists.
• The first return produces a structured report when the batch is absent or empty.

• Choose Deploy to save this version of lambda_function.py.

The console confirms that the function code was updated.

Deployment Not Completing?

Confirm that the file is named lambda_function.py. Check that every opening bracket has a matching closing bracket.

Ask for help with the first validator chunk.

• Place your cursor below the empty-input return block in lambda_function.py.
• Add the per-record checks by pasting this code:

    for index, order in enumerate(orders):
        row_label = f"orders[{index}]"

        if not isinstance(order, dict):
            errors.append(f"{row_label} must be an object")
            continue

        missing = [
            field
            for field in REQUIRED_FIELDS
            if field not in order or order[field] in (None, "")
        ]
        if missing:
            errors.append(f"{row_label} missing fields: {', '.join(missing)}")
            continue

        try:
            datetime.strptime(str(order["order_date"]), "%Y-%m-%d")
        except ValueError:
            errors.append(f"{row_label}.order_date must use YYYY-MM-DD")

How Are Records Checked?

• The loop assigns each order a position such as orders[0] so every error points to a specific record.
• The missing-field check compares each order with REQUIRED_FIELDS.
• The datetime.strptime() check accepts dates that follow YYYY-MM-DD.

• Choose Deploy to save the per-record checks.

The console confirms that the expanded validator was deployed.

Seeing an Indentation Problem?

Keep the for loop inside lambda_handler(). Keep its checks indented beneath the loop.

Ask for help aligning the validation loop.

• Place your cursor below the date validation block in lambda_function.py.
• Complete the handler by pasting the numeric checks and final report below:

        try:
            quantity = int(order["quantity"])
            if quantity <= 0:
                raise ValueError
        except (TypeError, ValueError):
            errors.append(f"{row_label}.quantity must be a positive integer")

        try:
            unit_price = float(order["unit_price"])
            if unit_price < 0:
                raise ValueError
        except (TypeError, ValueError):
            errors.append(f"{row_label}.unit_price must be zero or greater")

    return {
        "valid": len(errors) == 0,
        "record_count": len(orders),
        "errors": errors,
    }

How Is the Result Built?

• The quantity check requires a positive integer.
• The unit price check accepts zero or a positive number.
• The final return sets valid according to whether the errors list is empty.
• The record_count value reports how many orders entered the gate.

• Choose Deploy to save the complete validator.

The console confirms that the complete validation gate is deployed.

Complete Validator Not Deploying?

Keep the quantity and unit price checks inside the for loop. Align the final return with the loop so it runs after every record is checked.

Ask for help checking the completed handler.

✔️ Awesome, I've got everything!

Your deployed validator now contains the complete field contract and validation report.

ⓧ I'd like to double check the full code

from datetime import datetime

REQUIRED_FIELDS = (
    "order_id",
    "order_date",
    "customer_id",
    "customer_name",
    "segment",
    "product_id",
    "product_name",
    "category",
    "quantity",
    "unit_price",
)


def lambda_handler(event, context):
    orders = event.get("orders")
    errors = []

    if not isinstance(orders, list) or not orders:
        return {
            "valid": False,
            "record_count": 0,
            "errors": ["orders must be a non-empty list"],
        }

    for index, order in enumerate(orders):
        row_label = f"orders[{index}]"

        if not isinstance(order, dict):
            errors.append(f"{row_label} must be an object")
            continue

        missing = [
            field
            for field in REQUIRED_FIELDS
            if field not in order or order[field] in (None, "")
        ]
        if missing:
            errors.append(f"{row_label} missing fields: {', '.join(missing)}")
            continue

        try:
            datetime.strptime(str(order["order_date"]), "%Y-%m-%d")
        except ValueError:
            errors.append(f"{row_label}.order_date must use YYYY-MM-DD")

        try:
            quantity = int(order["quantity"])
            if quantity <= 0:
                raise ValueError
        except (TypeError, ValueError):
            errors.append(f"{row_label}.quantity must be a positive integer")

        try:
            unit_price = float(order["unit_price"])
            if unit_price < 0:
                raise ValueError
        except (TypeError, ValueError):
            errors.append(f"{row_label}.unit_price must be zero or greater")

    return {
        "valid": len(errors) == 0,
        "record_count": len(orders),
        "errors": errors,
    }

What Should Match?

This reference shows the complete deployed contents of lambda_function.py. Each validation check contributes messages to one structured errors list.

Run the invalid order test

A test event sends a controlled JSON payload directly to the function. This batch contains two orders so the validation gate can evaluate realistic source data.

• Choose Create test event on the Lambda function page.
• Name the event invalid-orders.
• Select the existing payload under Event JSON.
• Replace the payload with the order batch below:

{
  "orders": [
    {
      "order_id": "O1001",
      "order_date": "2026-08-20",
      "customer_id": "C01",
      "customer_name": "Ava Patel",
      "segment": "Consumer",
      "product_id": "P01",
      "product_name": "Travel Mug",
      "category": "Home",
      "quantity": 2,
      "unit_price": 25.0
    },
    {
      "order_id": "O1002",
      "order_date": "20-08-2026",
      "customer_id": "C02",
      "customer_name": "Noah Kim",
      "segment": "Small Business",
      "product_id": "P02",
      "product_name": "Wireless Headphones",
      "category": "Electronics",
      "quantity": 0,
      "unit_price": 80.0
    }
  ]
}

What Does This Event Represent?

The event represents a small retail order batch entering the pipeline. The validator evaluates every record against the same field and value rules.

• Save the invalid-orders test event.

Before you run the event, do you think the complete batch will pass or fail the validation gate?

• Choose Test to run the event.

Under Execution result, you will see valid set to false.

The errors array reports orders[1].order_date must use YYYY-MM-DD. It also reports orders[1].quantity must be a positive integer.

That is the result you wanted. Your validation gate now stops the malformed O1002 record before transformation.

Test Result Missing the Validation Errors?

Confirm that you deployed the final version of lambda_function.py. Confirm that the selected test event is invalid-orders.

Ask for help comparing the event with the validator.

Your pipeline can now identify bad retail records before they reach storage. Next, you will transform corrected orders into fact and dimension datasets.

Transform Orders into a Star Schema

Your validation gate has exposed the malformed source records. Now you can transform corrected orders into datasets built for retail analysis.

A star schema keeps each sale in a fact table. Separate dimension tables hold reusable customer details. They also hold reusable product details.

A second AWS Lambda function performs the transformation. The output connects each sale to its customer record and product record.

In this step, get ready to:

• Create the transform-retail-orders Lambda function.
• Transform corrected orders into one fact dataset and two dimension datasets.
• Test the star schema with six valid retail orders.

Create the transformation function

This function stays in the same account and Region as your validator. Keeping both functions together prepares them for one orchestrated workflow.

• Return to the AWS Management Console in Microsoft Edge.
• Use the console service search to open AWS Lambda.
• Choose Create function.
• Select Author from scratch.
• Enter transform-retail-orders under Function name.
• Select Python 3.14 under Runtime.
• Choose Create function.

Good progress. Your second Lambda function now has its own editor and the default lambda_function.lambda_handler handler.

The workflow later stores each modeled dataset as JSON Lines. This format places one JSON object on each line.

• Choose the Code tab.
• Select all existing code in lambda_function.py.
• Replace the selected code with the JSON Lines helper below:

import json


def to_json_lines(rows):
    return "\n".join(
        json.dumps(row, separators=(",", ":"), sort_keys=True)
        for row in rows
    )

What does this helper do?

• The json module converts each dictionary into JSON text.
• The to_json_lines() helper joins those JSON objects with newline characters.
• The compact separators keep the output small. Sorting the keys makes each line consistent.

• Choose Deploy.

The console confirms that the helper scaffold has been deployed.

Having trouble deploying the helper?

• Check that import json is the first line.
• Check that the body of to_json_lines() is indented with four spaces.

Need another pair of eyes? Help me check my JSON Lines helper.

Build the star schema transformation

The transformation uses dictionaries keyed by customer and product IDs. Repeated customers and products overwrite matching entries, which leaves one dimension row per unique ID.

• Place your cursor two blank lines below to_json_lines() in lambda_function.py.
• Add the transformation setup by pasting this code:

def lambda_handler(event, context):
    orders = event["orders"]
    customers = {}
    products = {}
    facts = []

    for order in orders:
        customer_id = str(order["customer_id"])
        product_id = str(order["product_id"])
        quantity = int(order["quantity"])
        unit_price = float(order["unit_price"])

        customers[customer_id] = {
            "customer_id": customer_id,
            "customer_name": str(order["customer_name"]),
            "segment": str(order["segment"]),
        }
        products[product_id] = {
            "product_id": product_id,
            "product_name": str(order["product_name"]),
            "category": str(order["category"]),
            "unit_price": unit_price,
        }

How are the dimensions built?

• The function reads the corrected records from event["orders"].
• The customers dictionary keeps one record for each customer_id.
• The products dictionary keeps one record for each product_id.
• Converting numeric values here gives every transformed row a consistent type.

• Choose Deploy.

The console accepts the customer and product transformation logic.

Does this chunk fail to deploy?

• Check that def lambda_handler(event, context): starts at the left edge.
• Check that both dictionary assignments sit inside the for order in orders: loop.

Still stuck? Help me fix the transformation setup.

The fact dataset now needs one row per order. Each row calculates revenue and carries the IDs that connect it to both dimensions.

• Place your cursor directly below the closing brace of the products[product_id] dictionary.
• Complete the transformer by pasting this code:

        facts.append(
            {
                "order_id": str(order["order_id"]),
                "order_date": str(order["order_date"]),
                "customer_id": customer_id,
                "product_id": product_id,
                "quantity": quantity,
                "unit_price": unit_price,
                "revenue": round(quantity * unit_price, 2),
            }
        )

    fact_orders = sorted(facts, key=lambda row: row["order_id"])
    dim_customers = sorted(customers.values(), key=lambda row: row["customer_id"])
    dim_products = sorted(products.values(), key=lambda row: row["product_id"])

    return {
        "fact_orders": fact_orders,
        "dim_customers": dim_customers,
        "dim_products": dim_products,
        "fact_orders_jsonl": to_json_lines(fact_orders),
        "dim_customers_jsonl": to_json_lines(dim_customers),
        "dim_products_jsonl": to_json_lines(dim_products),
    }

How does the fact dataset connect?

• Each fact_orders row represents one retail order.
• The calculated revenue value multiplies quantity by unit price.
• The customer_id and product_id values act as foreign keys to the dimension datasets.
• The return value includes list versions for processing. It also includes JSON Lines strings for storage.

• Choose Deploy.

The console confirms that the complete transformer has been deployed.

Seeing a deployment problem?

• Check that facts.append() is indented inside the order loop.
• Check that fact_orders starts at the same indentation level as the loop.
• Check that every opening parenthesis and brace has a matching closing character.

Need help comparing the blocks? Help me find the mismatch in my transformer.

• Double-check the complete lambda_function.py file using the tabs below:

✔️ Awesome, I've got everything!

Your deployed file contains the JSON Lines helper and the complete transformation handler.

ⓧ I'd like to double check the full code

import json


def to_json_lines(rows):
    return "\n".join(
        json.dumps(row, separators=(",", ":"), sort_keys=True)
        for row in rows
    )


def lambda_handler(event, context):
    orders = event["orders"]
    customers = {}
    products = {}
    facts = []

    for order in orders:
        customer_id = str(order["customer_id"])
        product_id = str(order["product_id"])
        quantity = int(order["quantity"])
        unit_price = float(order["unit_price"])

        customers[customer_id] = {
            "customer_id": customer_id,
            "customer_name": str(order["customer_name"]),
            "segment": str(order["segment"]),
        }
        products[product_id] = {
            "product_id": product_id,
            "product_name": str(order["product_name"]),
            "category": str(order["category"]),
            "unit_price": unit_price,
        }
        facts.append(
            {
                "order_id": str(order["order_id"]),
                "order_date": str(order["order_date"]),
                "customer_id": customer_id,
                "product_id": product_id,
                "quantity": quantity,
                "unit_price": unit_price,
                "revenue": round(quantity * unit_price, 2),
            }
        )

    fact_orders = sorted(facts, key=lambda row: row["order_id"])
    dim_customers = sorted(customers.values(), key=lambda row: row["customer_id"])
    dim_products = sorted(products.values(), key=lambda row: row["product_id"])

    return {
        "fact_orders": fact_orders,
        "dim_customers": dim_customers,
        "dim_products": dim_products,
        "fact_orders_jsonl": to_json_lines(fact_orders),
        "dim_customers_jsonl": to_json_lines(dim_customers),
        "dim_products_jsonl": to_json_lines(dim_products),
    }

How to compare the file

Check the indentation around the order loop first. Then compare the six keys in the returned dictionary.

Test the modeled datasets

The corrected test event contains six valid orders. Several orders repeat the same customers and products, which tests whether the dimensions remain unique.

• Choose Test.
• Choose Create test event.
• Name the event valid-orders.
• Replace the contents of Event JSON with this payload:

{
  "orders": [
    {
      "order_id": "O1001",
      "order_date": "2026-08-20",
      "customer_id": "C01",
      "customer_name": "Ava Patel",
      "segment": "Consumer",
      "product_id": "P01",
      "product_name": "Travel Mug",
      "category": "Home",
      "quantity": 2,
      "unit_price": 25.0
    },
    {
      "order_id": "O1002",
      "order_date": "2026-08-20",
      "customer_id": "C02",
      "customer_name": "Noah Kim",
      "segment": "Small Business",
      "product_id": "P02",
      "product_name": "Wireless Headphones",
      "category": "Electronics",
      "quantity": 1,
      "unit_price": 80.0
    },
    {
      "order_id": "O1003",
      "order_date": "2026-08-21",
      "customer_id": "C03",
      "customer_name": "Mia Garcia",
      "segment": "Consumer",
      "product_id": "P03",
      "product_name": "Hardcover Notebook",
      "category": "Stationery",
      "quantity": 3,
      "unit_price": 15.0
    },
    {
      "order_id": "O1004",
      "order_date": "2026-08-22",
      "customer_id": "C01",
      "customer_name": "Ava Patel",
      "segment": "Consumer",
      "product_id": "P02",
      "product_name": "Wireless Headphones",
      "category": "Electronics",
      "quantity": 2,
      "unit_price": 80.0
    },
    {
      "order_id": "O1005",
      "order_date": "2026-08-22",
      "customer_id": "C02",
      "customer_name": "Noah Kim",
      "segment": "Small Business",
      "product_id": "P01",
      "product_name": "Travel Mug",
      "category": "Home",
      "quantity": 4,
      "unit_price": 25.0
    },
    {
      "order_id": "O1006",
      "order_date": "2026-08-23",
      "customer_id": "C03",
      "customer_name": "Mia Garcia",
      "segment": "Consumer",
      "product_id": "P03",
      "product_name": "Hardcover Notebook",
      "category": "Stationery",
      "quantity": 2,
      "unit_price": 15.0
    }
  ]
}

What does this sample test?

The six orders contain three unique customers. They also contain three unique products.

The repeated IDs prove whether the transformation removes duplicated dimension details while preserving every sale.

• Finish creating the event by choosing Create test event.

Before you run the event, how many fact rows should six valid orders produce? How many unique customer and product rows should remain?

• Run the transformation by choosing Test.

In Execution result, you will see six rows in fact_orders. You will also see three rows in each dimension dataset.

The result also contains fact_orders_jsonl, dim_customers_jsonl, and dim_products_jsonl as three JSON Lines strings.

• Inspect fact_orders to confirm that it contains six rows.
• Inspect dim_customers to confirm that it contains three rows.
• Inspect dim_products to confirm that it contains three rows.
• Trace C01 from a fact row to its matching customer dimension row.
• Trace P02 from a fact row to its matching product dimension row.

That is the star schema working. Every sale remains available while repeated customer and product details collapse into reusable dimensions.

Do the row counts look wrong?

• Confirm that the selected test event is valid-orders.
• Confirm that all six order objects are present in Event JSON.
• Check that customers uses customer_id as its dictionary key.

Need help reading the result? Help me diagnose my star schema output.

Your clean orders now form a reusable data model. Next, you will orchestrate validation and transformation before storing the outputs in Amazon S3 and generating the dashboard.

Orchestrate the ETL and Dashboard

Your two AWS Lambda functions now validate orders and transform clean records into a star schema. Manual invocations leave storage and dashboard generation as separate jobs.

AWS Step Functions will turn those jobs into one repeatable ETL workflow. Each execution will move through an observable graph before writing its results to Amazon S3.

In this step, get ready to:

• Deploy a function that converts the transformed datasets into an HTML dashboard.
• Create a Standard workflow that coordinates storage, validation, transformation, and dashboard generation.
• Run invalid and valid samples to observe both branches of the pipeline.

Create the dashboard function

The transformed arrays contain everything needed for a standalone HTML dashboard. A third Lambda function will calculate the headline totals before building the page.

• Return to the AWS Lambda console from earlier.
• Click Create function.
• Select Author from scratch.
• Enter generate-retail-dashboard in the Function name field.
• Select Python 3.14 from the Runtime menu.
• Keep lambda_function.lambda_handler as the default handler.
• Click Create function.

• Replace the contents of lambda_function.py with the complete file in the double-check tab below.

✔️ Awesome, I've got everything!

Check that your dashboard function contains the complete source before you deploy it.

ⓧ I'd like to double check the full code

from html import escape


def money(value):
    return f"${value:,.2f}"


def lambda_handler(event, context):
    fact_orders = event.get("fact_orders", [])
    dim_customers = event.get("dim_customers", [])
    dim_products = event.get("dim_products", [])

    product_lookup = {
        product["product_id"]: product
        for product in dim_products
    }
    product_revenue = {}

    for order in fact_orders:
        product_id = order["product_id"]
        product_revenue[product_id] = round(
            product_revenue.get(product_id, 0) + float(order["revenue"]),
            2,
        )

    total_revenue = round(
        sum(float(order["revenue"]) for order in fact_orders),
        2,
    )
    ranked_products = sorted(
        product_revenue.items(),
        key=lambda item: item[1],
        reverse=True,
    )
    largest_revenue = max((value for _, value in ranked_products), default=1)

    product_rows = []
    for product_id, revenue in ranked_products:
        product = product_lookup[product_id]
        width = round((revenue / largest_revenue) * 100, 1)
        product_rows.append(
            f"""
            <tr>
              <td>{escape(product['product_name'])}</td>
              <td>{escape(product['category'])}</td>
              <td>{money(revenue)}</td>
              <td><div class="bar" style="width:{width}%"></div></td>
            </tr>
            """
        )

    html_document = f"""<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>Retail ETL Dashboard</title>
  <style>
    body {{ margin: 0; font-family: Arial, sans-serif; background: #f4f7fb; color: #172033; }}
    main {{ max-width: 980px; margin: 0 auto; padding: 32px 20px; }}
    h1 {{ margin-bottom: 4px; }}
    .subtitle {{ color: #5c667a; margin-top: 0; }}
    .cards {{ display: grid; grid-template-columns: repeat(3, 1fr); gap: 16px; margin: 24px 0; }}
    .card {{ background: white; border-radius: 12px; padding: 20px; box-shadow: 0 5px 18px rgba(23,32,51,.08); }}
    .label {{ color: #667085; font-size: 14px; }}
    .value {{ font-size: 30px; font-weight: 700; margin-top: 8px; }}
    table {{ width: 100%; border-collapse: collapse; background: white; border-radius: 12px; overflow: hidden; }}
    th, td {{ padding: 14px; text-align: left; border-bottom: 1px solid #e8ecf2; }}
    th {{ background: #172033; color: white; }}
    .bar {{ height: 12px; min-width: 4px; border-radius: 8px; background: #4f46e5; }}
    @media (max-width: 700px) {{ .cards {{ grid-template-columns: 1fr; }} }}
  </style>
</head>
<body>
  <main>
    <h1>Retail ETL Dashboard</h1>
    <p class="subtitle">Generated automatically from the curated star schema.</p>
    <section class="cards">
      <article class="card"><div class="label">Revenue</div><div class="value">{money(total_revenue)}</div></article>
      <article class="card"><div class="label">Orders</div><div class="value">{len(fact_orders)}</div></article>
      <article class="card"><div class="label">Customers</div><div class="value">{len(dim_customers)}</div></article>
    </section>
    <h2>Product performance</h2>
    <table>
      <thead><tr><th>Product</th><th>Category</th><th>Revenue</th><th>Relative revenue</th></tr></thead>
      <tbody>{''.join(product_rows)}</tbody>
    </table>
  </main>
</body>
</html>"""

    return {
        "html": html_document,
        "summary": {
            "revenue": total_revenue,
            "orders": len(fact_orders),
            "customers": len(dim_customers),
            "products": len(dim_products),
        },
    }

How Does the Dashboard Function Work?

• The lookup connects each fact row to its product name and category.
• The revenue calculations produce the headline total and ranked product results.
• The HTML document combines summary cards with a product performance table.
• The returned summary keeps the dashboard totals visible inside the workflow output.

• Click Deploy in the Code section.

That completes the presentation layer. Your third Lambda function can now turn curated records into a dashboard document.

• Copy the displayed ARN for generate-retail-dashboard and record it here: your generate-retail-dashboard function ARN.
• Return to the validate-retail-orders function.
• Copy its displayed ARN and record it here: your validate-retail-orders function ARN.
• Return to the transform-retail-orders function.
• Copy its displayed ARN and record it here: your transform-retail-orders function ARN.

Dashboard Function Not Deploying?

Check that lambda_function.py contains the full source from the double-check tab. Confirm that the handler remains lambda_function.lambda_handler.

Need another pair of eyes? Help me troubleshoot why my generate-retail-dashboard Lambda function is not deploying.

Build the workflow and its permissions

The workflow uses JSONPath to pass order data and function results between states. A Choice state reads the validation result before allowing transformation to begin.

• Switch to the AWS Step Functions console.
• Choose State machines.
• Click Create state machine.
• Select Create from blank.
• Click Continue.
• Switch to Code.
• Remove the starter definition from the code editor.
• Paste the complete state-machine.json definition from the double-check tab below.

✔️ Awesome, I've got everything!

Check that the workflow contains every state from raw storage through dashboard delivery.

ⓧ I'd like to double check the full code

{
  "Comment": "Validate and transform retail orders, store modeled data, and generate a dashboard",
  "QueryLanguage": "JSONPath",
  "StartAt": "Store Raw Orders",
  "States": {
    "Store Raw Orders": {
      "Type": "Task",
      "Resource": "arn:aws:states:::aws-sdk:s3:putObject",
      "Parameters": {
        "Bucket": "BUCKET_NAME",
        "Key": "raw/orders.json",
        "Body.$": "States.JsonToString($.orders)",
        "ContentType": "application/json"
      },
      "ResultPath": null,
      "Next": "Validate Orders"
    },
    "Validate Orders": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "VALIDATE_FUNCTION_ARN",
        "Payload.$": "$"
      },
      "ResultPath": "$.validation",
      "Next": "Orders Valid"
    },
    "Orders Valid": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.validation.Payload.valid",
          "BooleanEquals": true,
          "Next": "Transform Orders"
        }
      ],
      "Default": "Invalid Orders"
    },
    "Invalid Orders": {
      "Type": "Fail",
      "Error": "OrderValidationFailed",
      "Cause": "One or more retail orders failed validation"
    },
    "Transform Orders": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "TRANSFORM_FUNCTION_ARN",
        "Payload.$": "$"
      },
      "ResultPath": "$.transformed",
      "Next": "Store Fact Orders"
    },
    "Store Fact Orders": {
      "Type": "Task",
      "Resource": "arn:aws:states:::aws-sdk:s3:putObject",
      "Parameters": {
        "Bucket": "BUCKET_NAME",
        "Key": "curated/fact_orders/data.jsonl",
        "Body.$": "$.transformed.Payload.fact_orders_jsonl",
        "ContentType": "application/json"
      },
      "ResultPath": null,
      "Next": "Store Customer Dimension"
    },
    "Store Customer Dimension": {
      "Type": "Task",
      "Resource": "arn:aws:states:::aws-sdk:s3:putObject",
      "Parameters": {
        "Bucket": "BUCKET_NAME",
        "Key": "curated/dim_customers/data.jsonl",
        "Body.$": "$.transformed.Payload.dim_customers_jsonl",
        "ContentType": "application/json"
      },
      "ResultPath": null,
      "Next": "Store Product Dimension"
    },
    "Store Product Dimension": {
      "Type": "Task",
      "Resource": "arn:aws:states:::aws-sdk:s3:putObject",
      "Parameters": {
        "Bucket": "BUCKET_NAME",
        "Key": "curated/dim_products/data.jsonl",
        "Body.$": "$.transformed.Payload.dim_products_jsonl",
        "ContentType": "application/json"
      },
      "ResultPath": null,
      "Next": "Generate Dashboard"
    },
    "Generate Dashboard": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "DASHBOARD_FUNCTION_ARN",
        "Payload.$": "$.transformed.Payload"
      },
      "ResultPath": "$.dashboard",
      "Next": "Store Dashboard"
    },
    "Store Dashboard": {
      "Type": "Task",
      "Resource": "arn:aws:states:::aws-sdk:s3:putObject",
      "Parameters": {
        "Bucket": "BUCKET_NAME",
        "Key": "dashboard/index.html",
        "Body.$": "$.dashboard.Payload.html",
        "ContentType": "text/html"
      },
      "ResultPath": null,
      "Next": "Pipeline Succeeded"
    },
    "Pipeline Succeeded": {
      "Type": "Succeed"
    }
  }
}

How Does the Workflow Move Data?

• The first state stores the incoming orders at raw/orders.json.
• The validation result decides whether the workflow stops or reaches transformation.
• The storage states write one fact dataset and two dimension datasets as JSON Lines.
• The final states generate dashboard/index.html before marking the execution successful.

• Replace BUCKET_NAME with retail-etl-your-initials-randomdigits everywhere in the definition.
• Replace VALIDATE_FUNCTION_ARN with your validate-retail-orders function ARN.
• Replace TRANSFORM_FUNCTION_ARN with your transform-retail-orders function ARN.
• Replace DASHBOARD_FUNCTION_ARN with your generate-retail-dashboard function ARN.

• Open the Config panel.
• Enter retail-etl-pipeline as the state machine name.
• Select Standard as the workflow type.
• Select Create new role for the execution role.
• Click Create.
• Click Confirm if AWS asks you to approve the generated role.

Your workflow graph now connects all three functions. The generated role still needs permission to write pipeline objects into your private bucket.

AWS Identity and Access Management controls what the execution role can do. The inline policy will grant only the S3 object-write action used by this workflow.

• Click the displayed execution role ARN on the state machine details page.
• Open the Permissions section.
• Click Add permissions.
• Select Create inline policy.
• Choose the JSON editor.
• Replace the policy editor contents with this policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "WriteRetailEtlObjects",
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::BUCKET_NAME/*"
    }
  ]
}

Why Add This Policy Manually?

Step Functions cannot generate complete permissions for the S3 AWS SDK integration. This statement grants the execution role permission to write objects inside your project bucket.

The resource ends with /* because the workflow writes objects under several prefixes. The policy does not grant public access to the bucket.

• Replace BUCKET_NAME with retail-etl-your-initials-randomdigits in the policy.
• Click Next.
• Enter RetailEtlS3Write in the Policy Name field.
• Click Create policy.

That closes the permissions gap. The role now lists RetailEtlS3Write as an inline policy for your bucket.

Policy Not Appearing?

Confirm that you added the policy to the role linked from retail-etl-pipeline. Check that the resource ends with retail-etl-your-initials-randomdigits/*.

Still stuck? Help me troubleshoot my RetailEtlS3Write inline policy.

Run both validation paths

The state machine has two meaningful outcomes. Invalid data should stop at the validation branch while corrected data should reach the success state.

• Return to the validate-retail-orders function.
• Open its existing invalid-orders test event.
• Copy the complete contents of Event JSON.
• Return to the retail-etl-pipeline state machine.
• Click Start execution.
• Paste the invalid sample into Input.

Before you start the execution, where do you think the invalid branch will stop?

• Click Start Execution.

The Execution Details page reports Failed. The graph reaches Invalid Orders after storing the raw input.

This Failure Is Intentional

The failed execution proves that the validation gate blocks malformed orders from reaching transformation. The raw input remains available for investigation.

No curated datasets or dashboard are produced by this invalid execution. The workflow has protected the trusted side of the pipeline.

• Return to the transform-retail-orders function.
• Open its existing valid-orders test event.
• Copy the complete contents of Event JSON.
• Return to the retail-etl-pipeline state machine.
• Click Start execution.
• Paste the valid sample into Input.

Before you start this execution, do you expect every state on the valid path to complete?

• Click Start Execution.

The Execution Details page reports Succeeded. The graph reaches Pipeline Succeeded after storing the dashboard.

• Select Generate Dashboard in Graph view.
• Open Step details.
• Inspect the state output for the dashboard summary.

You should see orders set to 6. You should also see revenue set to 465.0.

Before you inspect the bucket, which raw, curated, and dashboard objects do you expect the successful execution to have written?

• Return to Amazon S3.
• Choose General purpose buckets.
• Select retail-etl-your-initials-randomdigits.
• Refresh the Objects view.

You should find raw/orders.json, curated/fact_orders/data.jsonl, curated/dim_customers/data.jsonl, curated/dim_products/data.jsonl, and dashboard/index.html.

Valid Execution Not Succeeding?

• Confirm that the state machine and all three Lambda functions use the same AWS Region.
• Check that every function ARN placeholder was replaced with the matching displayed ARN.
• Confirm that RetailEtlS3Write grants s3:PutObject for your bucket objects.

Need help reading the failed state? Help me troubleshoot my retail-etl-pipeline execution.

Your ETL stages now run as one observable pipeline. Next, you will register the three curated datasets as catalog tables before opening the generated dashboard.

Catalog Data and Open the Dashboard

Your workflow now turns clean orders into curated files. It also generates a working dashboard.

Files in Amazon S3 hold the results. Analytics tools still need metadata that describes each dataset.

Inside the AWS Management Console, the AWS Glue Data Catalog supplies that metadata. You will register the star schema before opening the dashboard in Microsoft Edge.

In this step, get ready to:

• Create a metadata database for the retail datasets.
• Register the three curated datasets as Data Catalog tables.
• View the generated retail dashboard in Microsoft Edge.

Create the metadata database

A Data Catalog database groups related table definitions. The retail_etl database becomes the metadata home for your retail star schema.

• Return to the AWS Management Console tab in Microsoft Edge.
• Use the search field in the console header to find AWS Glue.
• Select AWS Glue from the search results.
• Confirm the Region selector matches the Region containing retail-etl-pipeline.

The Region should match the one you selected for every earlier project resource.

• Choose Databases under Data Catalog in the left navigation.
• Choose Add database.
• Enter retail_etl as the database name.
• Leave Location - optional empty.
• Choose Create database.

You should see retail_etl in the database list with no location configured.

Your curated datasets now have a metadata home that analytics tools can discover.

Register the star schema tables

A central fact table stores measurable order events. Reusable customer and product details live in dimension tables.

Each table points to one curated S3 prefix. Its format is JavaScript Object Notation.

Table Schemas

Use these columns for fact_orders:

• Set order_id to string.
• Set order_date to string.
• Set customer_id to string.
• Set product_id to string.
• Set quantity to bigint.
• Set unit_price to double.
• Set revenue to double.

Use these columns for dim_customers:

• Set customer_id to string.
• Set customer_name to string.
• Set segment to string.

Use these columns for dim_products:

• Set product_id to string.
• Set product_name to string.
• Set category to string.
• Set unit_price to double.

• Choose Tables under Data Catalog.
• Choose Add tables.
• Choose Add table manually.
• Enter fact_orders as the table name.
• Select retail_etl as the database.

The wizard should now show fact_orders inside the retail_etl database.

• Enter s3://retail-etl-your-initials-randomdigits/curated/fact_orders/ as the S3 path.
• Choose JavaScript Object Notation as the data format.

The table source should now point to the curated fact dataset in your private bucket.

• Add every fact_orders column from the Table Schemas callout.
• Finish the Add table wizard to create fact_orders.

You should see fact_orders in the table list.

• Return to Tables under Data Catalog.
• Choose Add tables.
• Choose Add table manually.
• Enter dim_customers as the table name.
• Select retail_etl as the database.

The wizard should now show dim_customers inside the retail_etl database.

• Enter s3://retail-etl-your-initials-randomdigits/curated/dim_customers/ as the S3 path.
• Choose JavaScript Object Notation as the data format.

The table source should now point to the curated customer dimension in your private bucket.

• Add every dim_customers column from the Table Schemas callout.
• Finish the Add table wizard to create dim_customers.

You should see fact_orders plus dim_customers in the table list.

• Return to Tables under Data Catalog.
• Choose Add tables.
• Choose Add table manually.
• Enter dim_products as the table name.
• Select retail_etl as the database.

The wizard should now show dim_products inside the retail_etl database.

• Enter s3://retail-etl-your-initials-randomdigits/curated/dim_products/ as the S3 path.
• Choose JavaScript Object Notation as the data format.

The table source should now point to the curated product dimension in your private bucket.

• Add every dim_products column from the Table Schemas callout.
• Finish the Add table wizard to create dim_products.

Your star schema is registered. AWS Glue now lists one fact table plus two dimensions.

Tables Missing?

• Confirm AWS Glue is using the same Region as your S3 bucket.
• Open retail_etl to confirm each table belongs to the correct database.
• Compare each table location with its matching curated/ prefix in Amazon S3.

Help me troubleshoot missing AWS Glue Data Catalog tables.

Open the generated dashboard

The dashboard is a standalone HTML file generated from the curated star schema. Downloading it lets you view the result while the S3 bucket stays private.

• Use the console search field to return to Amazon S3.
• Choose General purpose buckets.
• Choose retail-etl-your-initials-randomdigits.
• Open the dashboard/ prefix.
• Select index.html.
• Choose Download.

Microsoft Edge should show the completed index.html download.

Before you open the downloaded file, what revenue and record counts do you expect to see?

• Select the downloaded index.html file in Microsoft Edge.

You should see $465.00 under Revenue. You should also see 6 under Orders.

You should see 3 under Customers. The Product performance table shows revenue by product.

You now have a dashboard you can demo locally while every source object stays private in S3.

Secret mission

Quarantine Rejected Orders

Your validation gate stops bad orders, but its report disappears when the workflow fails. Extend the failure branch so every rejected report lands in a private S3 quarantine object before the execution ends.

Clean Up Your Resources

Clean Up Your Resources

Your active AWS Free Plan keeps this small project at $0 while the plan remains active. The downloaded dashboard has no ongoing cloud cost.

Choose whether to keep the AWS resources available, pause new executions, or delete everything.

Cost warning

The Free Plan ends after six months or when its credits are exhausted. Amazon S3 can incur standard storage and request charges on a Paid Plan.

• Delete the cloud resources before the plan ends if absolute zero cost matters.

Resources you used:

• One private Amazon S3 general purpose bucket named retail-etl-your-initials-randomdigits. It holds the raw/, curated/, dashboard/, and quarantine/ data.
• Three AWS Lambda functions named validate-retail-orders, transform-retail-orders, and generate-retail-dashboard.
• One Standard AWS Step Functions state machine named retail-etl-pipeline.
• One AWS Identity and Access Management (IAM) execution role with the inline RetailEtlS3Write policy.
• One AWS Glue Data Catalog database named retail_etl with the fact_orders, dim_customers, and dim_products tables.
• One downloaded dashboard/index.html file available locally through Microsoft Edge.

Keep everything running

No action is needed. Choose this if you want to demonstrate the pipeline again while your Free Plan remains active.

• Keep retail-etl-pipeline plus its supporting AWS resources in the selected Region.
• Keep the downloaded dashboard/index.html file for local demonstrations.
• Avoid starting unnecessary workflow executions.
• Confirm that your Free Plan is still active before running the workflow again.

Pause - I'll come back to this later

The pipeline has no continuously running server. Pausing means stopping new executions while leaving the stored data and metadata in place.

• Stop starting new executions of retail-etl-pipeline.
• Leave every listed cloud resource unchanged.
• Keep the downloaded dashboard/index.html file for your next session.
• Return before the Free Plan ends to keep or delete the remaining resources.

Delete - I don't want to use this again

This option removes every cloud resource from the project. Your local copy disappears only when you delete it in the final cleanup group.

Deletion is permanent. The ordered cleanup groups below help you avoid leaving connected resources behind.

Remove the Glue Catalog

• Return to the AWS Glue Data Catalog in your selected Region to remove the metadata layer.
• Choose Tables.
• Select fact_orders.
• Select dim_customers.
• Select dim_products.
• Delete the selected tables.
• Complete the confirmation shown by AWS.

You should no longer see any tables inside the retail_etl database.

• Choose Databases.
• Select retail_etl.
• Delete the selected database.
• Complete the confirmation shown by AWS.

You should no longer see retail_etl in the database list.

Remove the Workflow and Role

• Return to State machines in AWS Step Functions to remove the orchestration layer.
• Select retail-etl-pipeline.
• Use the displayed IAM role ARN to open its role in a new Microsoft Edge tab.
• Keep the IAM role tab open.
• Return to the state machine tab.
• Delete retail-etl-pipeline.
• Complete the confirmation shown by AWS.

You should no longer see retail-etl-pipeline in the state machine list.

• Switch back to the IAM role tab.
• Choose Permissions.
• Select RetailEtlS3Write.
• Delete the inline policy.
• Complete the confirmation shown by AWS.
• Delete the current execution role.
• Complete the confirmation shown by AWS.

You should no longer find the RetailEtlS3Write policy or its execution role.

Remove the Lambda Functions

• Return to AWS Lambda in your selected Region to remove the pipeline code.
• Select validate-retail-orders.
• Select transform-retail-orders.
• Select generate-retail-dashboard.
• Delete the selected functions.
• Complete the confirmation shown by AWS.

You should no longer see any of the three retail ETL functions in AWS Lambda.

Empty and Delete the S3 Bucket

• Return to Amazon S3 in your selected Region to remove the stored pipeline data.
• Choose retail-etl-your-initials-randomdigits.
• Choose Objects.
• Select every item in the bucket.
• Delete the selected objects.
• Complete the confirmation shown by AWS.

You should see an empty bucket with no raw, curated, dashboard, or quarantine objects.

• Return to General purpose buckets.
• Select retail-etl-your-initials-randomdigits.
• Delete the selected bucket.
• Complete the confirmation shown by AWS.

You should no longer see retail-etl-your-initials-randomdigits in the bucket list.

Remove the Local Dashboard Copy

• Return to Microsoft Edge's download history to locate the downloaded dashboard/index.html file.
• Show the file in its local folder.
• Delete the downloaded dashboard/index.html file.

Your cloud resources and local dashboard copy are now removed.

Nice Work!

Nice Work!

You did it! Your serverless ETL pipeline now turns validated orders into modeled retail data and a working sales dashboard.

You've learned how to:

• Build a private validation gate with AWS Lambda that stores incoming orders in Amazon S3 and rejects malformed records with a structured report.
• Transform clean orders into a star schema containing fact_orders, dim_customers, and dim_products. Register all three datasets in the AWS Glue Data Catalog for discovery.
• Orchestrate the complete ETL workflow with AWS Step Functions, JSONPath, and least-privilege IAM. Open the generated dashboard to see $465.00 revenue across 6 orders.
• Secret Mission: Extend the invalid branch to preserve validation errors in quarantine/rejected-orders.json before OrderValidationFailed ends the execution.

Ready to quiz yourself?