Deploy a Containerized Data Pipeline

Build and deploy a medallion ETL pipeline on AWS Lambda using Docker.

Introduction

30 Second Summary

Every time you file an insurance claim, your data passes through layers of processing before anyone sees a report. Behind the scenes, data pipelines clean, validate, and transform raw records into the neat summaries that analysts use to make decisions.

In this project, you will build a containerized data pipeline that processes insurance claims through three layers, package it with Docker, and deploy it to AWS Lambda so you can run it on demand from the command line.

What You'll Build

A serverless ETL pipeline that ingests raw JSON claims data, validates and cleans it, then aggregates it into analytics-ready Parquet files, all running inside a Docker container on AWS Lambda.

By the end of this project, you'll have:

  • A Python ETL pipeline containerized with Docker that processes insurance claims through Bronze, Silver, and Gold data layers.
  • A Lambda function deployed from your container image in Amazon ECR that writes structured data to S3 buckets.
  • A queryable Gold-layer dataset accessible via Amazon Athena using SQL.
  • Secret Mission: Add automated daily scheduling with EventBridge and data quality gates that reject anomalous claims.

Are there any prerequisites?

You should have basic Python and SQL knowledge before starting. You also need Docker Desktop and VS Code installed on your machine.

Set Up AWS Account and CLI

To deploy your data pipeline to the cloud, you need an AWS account and a way to interact with AWS services from your terminal.

In this step, you will create a free-tier AWS account, set up an IAM user with the permissions your pipeline needs, and install the AWS CLI so you can deploy resources from your command line.

In this step, get ready to:
  • Create an AWS account with free-tier credits.
  • Create an IAM user with the permissions needed for S3, ECR, Lambda, Athena, and Glue.
  • Install and configure the AWS CLI on your machine.
Create an AWS account

AWS offers up to $200 in credits for new accounts ($100 at sign-up, plus up to $100 more as you explore services). This covers the resources you will create in this project.

  • Open https://aws.amazon.com/free/ in your browser.
  • Click Create a Free Account (or Create an AWS Account).
  • Follow the sign-up prompts to enter your email, create a password, and provide billing information.
  • When asked to choose a support plan, select the Paid plan to get full access to all services (you still receive free-tier credits).
  • Complete the account setup and sign in to the AWS Management Console.

Why the Paid plan?

The Paid plan gives you unrestricted access to every AWS service (ECR, Lambda, Athena, Glue) that this project uses.

New accounts receive up to $200 in credits that cover your usage. The free-tier credit window lasts 6 months or until credits are exhausted, whichever comes first.

  • Once signed in, search for Billing in the top search bar and open the Billing and Cost Management console.
  • Set up a billing alert or budget to notify you if costs approach your credit limit.

Set a budget alert early

Creating a $5 or $10 budget alert in the Billing console gives you peace of mind. You will get an email if your usage ever approaches that threshold.

Create an IAM user with programmatic access

Your root AWS account has full access to everything, which is risky for day-to-day work. An IAM user is a separate identity with only the permissions you explicitly grant.

  • In the AWS Console, search for IAM in the top search bar and open the IAM console.
  • In the left navigation pane, click Users.
  • Click Create user.
  • On the Specify user details page, enter a user name (for example, pipeline-deployer).
  • Click Next.
  • On the Set permissions page, choose Attach policies directly.
  • In the search box, search for and select each of the following five policies:
  • AmazonS3FullAccess
  • AmazonEC2ContainerRegistryFullAccess
  • AWSLambda_FullAccess
  • AmazonAthenaFullAccess
  • AWSGlueConsoleFullAccess
  • After selecting all five policies, click Next.
  • Review the summary and click Create user.

What are these policies?

Each policy grants your IAM user access to a specific AWS service. S3 stores your pipeline data. ECR stores your Docker container image. Lambda runs your pipeline. Athena lets you query your data. Glue manages the metadata catalog.

Now generate an access key so the AWS CLI can authenticate as this user.

  • Click View user to go to the user details page.
  • Select the Security credentials tab.
  • In the Access keys section, click Create access key.
  • For the use case, select Command Line Interface (CLI) and click Next.
  • Click Create access key.
  • Click Download .csv file to save your access key ID and secret access key to a secure location.
  • Click Done.

Save your credentials now

This is your only chance to view or download the secret access key. If you lose it, you will need to create a new access key. Store the .csv file somewhere secure and do not share it.

Install and configure the AWS CLI

The AWS CLI lets you create and manage AWS resources directly from your terminal. You will use it throughout this project to deploy your pipeline.

  • Download the AWS CLI v2 installer for Windows from https://awscli.amazonaws.com/AWSCLIV2.msi.
  • Run the downloaded AWSCLIV2.msi installer and follow the prompts to complete the installation.
  • Close and reopen your terminal (PowerShell) after installation so the new path takes effect.
  • Verify the installation by running:
aws --version

✔️ I see a version number starting with aws-cli/2

The AWS CLI is installed and ready. Continue to the next step below.

ⓧ Command not found

The installer may not have added the CLI to your system path yet.

  • Close your terminal completely and open a new one.
  • Try running aws --version again.
  • If it still fails, verify that C:\Program Files\Amazon\AWSCLIV2\ exists and is in your system PATH.

Now configure the CLI with your IAM credentials.

  • Connect the AWS CLI to your IAM user by running:
aws configure

What will aws configure ask?

The command prompts you for four values. Enter them as follows:

  • AWS Access Key ID [None]: paste your access key ID from the downloaded .csv file.
  • AWS Secret Access Key [None]: paste your secret access key from the .csv file.
  • Default region name [None]: enter us-east-1.
  • Default output format [None]: enter json.

Next, verify that Python 3.12 is available on your machine. Your pipeline container uses Python 3.12, so you need it locally for testing.

  • Check your Python version by running:
python --version

✔️ I see Python 3.12 or higher

Your Python version is compatible. Continue to verifying your AWS setup below.

ⓧ I see a version lower than 3.12

  • Download Python 3.12 from https://www.python.org/downloads/.
  • Run the installer. Make sure to check Add python.exe to PATH during installation.
  • Close and reopen your terminal.
  • Run python --version again to confirm it shows 3.12 or higher.

Windows may still show the old version

If your system still reports an older version after installing 3.12, try running py -3.12 --version instead. Windows sometimes keeps an older Python on the default path.

ⓧ Command not found

  • Download Python 3.12 from https://www.python.org/downloads/.
  • Run the installer. Make sure to check Add python.exe to PATH during installation.
  • Close and reopen your terminal.
  • Run python --version again to confirm it shows 3.12 or higher.

Finally, verify your full AWS setup is working.

  • Confirm the AWS CLI is connected to your account by running:
aws sts get-caller-identity

What should I see?

You should see JSON output containing your Account ID, UserId, and Arn (which includes your IAM user name). This confirms the CLI can authenticate as your IAM user.

Seeing an error?

  • If you see a message about being unable to locate credentials, run aws configure again and double-check you pasted the access key ID and secret access key correctly.
  • If you see a message about an invalid client token, your access key may have been deactivated. Go back to the IAM console and verify the key status is Active.

Still stuck? Help me troubleshoot my aws sts get-caller-identity error

Your AWS environment is ready. Next up, you will build the Python ETL pipeline that processes insurance claims through the Medallion architecture.

Build the Python ETL Pipeline

Your AWS environment is configured and ready to receive data. Now you need the actual pipeline code that processes insurance claims through three layers of transformation.

Data pipelines in production follow a Medallion architecture. Raw data lands in a Bronze layer, gets validated and cleaned in a Silver layer, then gets aggregated into analytics-ready output in a Gold layer. You'll write a Python script that handles all three stages.

In this step, get ready to:
  • Create your project directory and write the sample data generation code.
  • Build the ETL processing functions for Bronze, Silver, and Gold layers.
  • Install dependencies and verify the pipeline runs locally.
Create the project directory and sample data

Before writing any pipeline logic, you need a project folder to hold your code. The pipeline will process synthetic insurance claims data that simulates what a real system would receive from source applications.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir claims-pipeline
cd claims-pipeline
  • Confirm the folder exists by running:
ls

You should see an empty directory. This is where all your pipeline files will live.

  • Open the claims-pipeline folder in VS Code by running:
code .
  • In VS Code, create a new file called lambda_function.py by clicking File in the top menu bar, then New File.
  • Save it immediately with Ctrl+S and name it lambda_function.py.

Now you'll add the imports, configuration, and data generation code. This sets up the foundation that the rest of the pipeline builds on.

  • Copy the following code into lambda_function.py:
import json
import boto3
import os
from datetime import datetime, date

s3_client = boto3.client('s3')

BUCKET_NAME = os.environ.get('BUCKET_NAME', 'claims-pipeline-data-YOURACCOUNTID')


def generate_sample_claims():
    """Generate synthetic insurance claims data."""
    claims = [
        {"claim_id": "CLM001", "policy_number": "POL-2024-001", "claim_date": "2024-11-15", "claim_amount": 2500.00, "claim_type": "auto", "status": "approved", "provider_name": "City Auto Repair"},
        {"claim_id": "CLM002", "policy_number": "POL-2024-002", "claim_date": "2024-11-16", "claim_amount": 15000.00, "claim_type": "health", "status": "pending", "provider_name": "General Hospital"},
        {"claim_id": "CLM003", "policy_number": "POL-2024-003", "claim_date": "2024-11-17", "claim_amount": 750.00, "claim_type": "home", "status": "approved", "provider_name": "HomeFixers LLC"},
        {"claim_id": "CLM004", "policy_number": "POL-2024-004", "claim_date": "2024-11-18", "claim_amount": 3200.00, "claim_type": "auto", "status": "denied", "provider_name": "QuickFix Auto"},
        {"claim_id": "CLM005", "policy_number": "POL-2024-005", "claim_date": "2024-11-19", "claim_amount": 8900.00, "claim_type": "health", "status": "approved", "provider_name": "Sunrise Medical"},
        {"claim_id": "CLM006", "policy_number": "POL-2024-006", "claim_date": "2024-11-20", "claim_amount": 450.00, "claim_type": "auto", "status": "approved", "provider_name": "City Auto Repair"},
        {"claim_id": "CLM007", "policy_number": "POL-2024-007", "claim_date": "2024-11-21", "claim_amount": 22000.00, "claim_type": "home", "status": "pending", "provider_name": "BuildRight Construction"},
        {"claim_id": "CLM008", "policy_number": "POL-2024-008", "claim_date": "2024-11-22", "claim_amount": 1100.00, "claim_type": "health", "status": "approved", "provider_name": "Downtown Clinic"},
        {"claim_id": "CLM009", "policy_number": "POL-2024-009", "claim_date": "2024-11-23", "claim_amount": 5600.00, "claim_type": "auto", "status": "approved", "provider_name": "Premium Auto Body"},
        {"claim_id": "CLM010", "policy_number": "POL-2024-010", "claim_date": "2024-11-24", "claim_amount": 18500.00, "claim_type": "health", "status": "denied", "provider_name": "General Hospital"},
    ]
    return claims


def write_to_s3(data, key):
    """Write data to S3."""
    if isinstance(data, bytes):
        s3_client.put_object(Bucket=BUCKET_NAME, Key=key, Body=data)
    else:
        s3_client.put_object(Bucket=BUCKET_NAME, Key=key, Body=json.dumps(data, default=str))

What does this code do?

This foundation code sets up the building blocks for your entire pipeline.

  • The imports bring in boto3 (AWS SDK for Python), json for data serialization, and datetime for timestamps.
  • BUCKET_NAME reads from an environment variable so you can configure it at deployment time without changing code.
  • generate_sample_claims() creates 10 synthetic insurance claims with fields like claim_id, claim_amount, and claim_type. This simulates data arriving from source systems.
  • write_to_s3() is a helper that handles both raw bytes (for Parquet files) and JSON-serializable data.
  • Save lambda_function.py with Ctrl+S.

Seeing red squiggly lines under imports?

VS Code may show warnings because boto3 and pyarrow are not installed yet. You will install them in the final substep. The warnings will disappear after that.

Help me fix import warnings in my lambda_function.py

Write the ETL processing functions

With the data generation in place, you need the three processing stages that transform raw claims into analytics-ready output. Each stage writes its results to S3 under a different prefix.

  • Add the Bronze and Silver processing functions below the write_to_s3 function in lambda_function.py:
def process_bronze(claims):
    """Bronze layer: store raw data as-is."""
    timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
    key = f"bronze/claims_{timestamp}.json"
    write_to_s3(claims, key)
    return claims


def process_silver(claims):
    """Silver layer: validate, clean, and standardize."""
    valid_claims = []
    for claim in claims:
        if not all(k in claim for k in ['claim_id', 'claim_amount', 'claim_type', 'status', 'claim_date']):
            continue
        if claim['claim_amount'] is None or claim['claim_amount'] <= 0:
            continue
        claim['claim_amount'] = float(claim['claim_amount'])
        claim['claim_type'] = claim['claim_type'].lower().strip()
        claim['status'] = claim['status'].lower().strip()
        claim['processed_at'] = datetime.now().isoformat()
        valid_claims.append(claim)

    timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
    key = f"silver/claims_{timestamp}.json"
    write_to_s3(valid_claims, key)
    return valid_claims

What do Bronze and Silver do?

Bronze is the simplest layer. It takes whatever data comes in and stores it exactly as-is in S3. This creates an immutable audit trail of raw data.

Silver is where data quality happens. It checks that required fields exist, removes claims with invalid amounts, casts types to consistent formats, and adds a processed_at timestamp. Any record that fails validation is silently dropped.

  • Add the Gold processing function below process_silver:
def process_gold(claims):
    """Gold layer: aggregate into analytics-ready Parquet."""
    import pyarrow as pa
    import pyarrow.parquet as pq
    import io

    aggregations = {}
    for claim in claims:
        group_key = (claim['claim_type'], claim['status'])
        if group_key not in aggregations:
            aggregations[group_key] = {'total_amount': 0.0, 'claim_count': 0}
        aggregations[group_key]['total_amount'] += claim['claim_amount']
        aggregations[group_key]['claim_count'] += 1

    rows = []
    for (claim_type, status), agg in aggregations.items():
        rows.append({
            'claim_type': claim_type,
            'status': status,
            'total_amount': agg['total_amount'],
            'claim_count': agg['claim_count'],
            'avg_amount': agg['total_amount'] / agg['claim_count']
        })

    table = pa.table({
        'claim_type': [r['claim_type'] for r in rows],
        'status': [r['status'] for r in rows],
        'total_amount': [r['total_amount'] for r in rows],
        'claim_count': [r['claim_count'] for r in rows],
        'avg_amount': [r['avg_amount'] for r in rows],
    })

    buffer = io.BytesIO()
    pq.write_table(table, buffer)
    buffer.seek(0)

    timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
    key = f"gold/claims_summary_{timestamp}.parquet"
    write_to_s3(buffer.read(), key)
    return rows

What does the Gold layer do?

The Gold layer groups claims by claim_type and status, then computes summary statistics (total amount, count, average). This is the kind of pre-aggregated data that analysts query directly.

The output is written as Parquet using the pyarrow library. Parquet is a columnar format that makes analytical queries dramatically faster and cheaper than scanning raw JSON rows.

  • Add the Lambda handler function below process_gold. This orchestrates the entire pipeline:
def handler(event, context):
    """Lambda handler: orchestrate the full pipeline."""
    claims = generate_sample_claims()

    bronze_result = process_bronze(claims)
    print(f"Bronze: wrote {len(bronze_result)} raw claims")

    silver_result = process_silver(bronze_result)
    print(f"Silver: {len(silver_result)} valid claims after cleaning")

    gold_result = process_gold(silver_result)
    print(f"Gold: {len(gold_result)} aggregated groups written as Parquet")

    return {
        'statusCode': 200,
        'body': json.dumps({
            'message': 'Pipeline completed successfully',
            'bronze_count': len(bronze_result),
            'silver_count': len(silver_result),
            'gold_groups': len(gold_result)
        })
    }

What does the handler do?

The handler function is what AWS Lambda invokes when your pipeline runs. It calls each processing stage in order (Bronze, Silver, Gold), logs progress, and returns a success response with counts for each layer.

The event and context parameters are required by Lambda's interface. You don't use them here because the pipeline generates its own sample data.

  • Save lambda_function.py with Ctrl+S.

✔️ Awesome, I've got everything!

Great. Double check you have saved the file before moving on.

ⓧ I'd like to double check the full code

Here is the complete lambda_function.py file. Make sure your code matches this exactly:

import json
import boto3
import os
from datetime import datetime, date

s3_client = boto3.client('s3')

BUCKET_NAME = os.environ.get('BUCKET_NAME', 'claims-pipeline-data-YOURACCOUNTID')


def generate_sample_claims():
    """Generate synthetic insurance claims data."""
    claims = [
        {"claim_id": "CLM001", "policy_number": "POL-2024-001", "claim_date": "2024-11-15", "claim_amount": 2500.00, "claim_type": "auto", "status": "approved", "provider_name": "City Auto Repair"},
        {"claim_id": "CLM002", "policy_number": "POL-2024-002", "claim_date": "2024-11-16", "claim_amount": 15000.00, "claim_type": "health", "status": "pending", "provider_name": "General Hospital"},
        {"claim_id": "CLM003", "policy_number": "POL-2024-003", "claim_date": "2024-11-17", "claim_amount": 750.00, "claim_type": "home", "status": "approved", "provider_name": "HomeFixers LLC"},
        {"claim_id": "CLM004", "policy_number": "POL-2024-004", "claim_date": "2024-11-18", "claim_amount": 3200.00, "claim_type": "auto", "status": "denied", "provider_name": "QuickFix Auto"},
        {"claim_id": "CLM005", "policy_number": "POL-2024-005", "claim_date": "2024-11-19", "claim_amount": 8900.00, "claim_type": "health", "status": "approved", "provider_name": "Sunrise Medical"},
        {"claim_id": "CLM006", "policy_number": "POL-2024-006", "claim_date": "2024-11-20", "claim_amount": 450.00, "claim_type": "auto", "status": "approved", "provider_name": "City Auto Repair"},
        {"claim_id": "CLM007", "policy_number": "POL-2024-007", "claim_date": "2024-11-21", "claim_amount": 22000.00, "claim_type": "home", "status": "pending", "provider_name": "BuildRight Construction"},
        {"claim_id": "CLM008", "policy_number": "POL-2024-008", "claim_date": "2024-11-22", "claim_amount": 1100.00, "claim_type": "health", "status": "approved", "provider_name": "Downtown Clinic"},
        {"claim_id": "CLM009", "policy_number": "POL-2024-009", "claim_date": "2024-11-23", "claim_amount": 5600.00, "claim_type": "auto", "status": "approved", "provider_name": "Premium Auto Body"},
        {"claim_id": "CLM010", "policy_number": "POL-2024-010", "claim_date": "2024-11-24", "claim_amount": 18500.00, "claim_type": "health", "status": "denied", "provider_name": "General Hospital"},
    ]
    return claims


def write_to_s3(data, key):
    """Write data to S3."""
    if isinstance(data, bytes):
        s3_client.put_object(Bucket=BUCKET_NAME, Key=key, Body=data)
    else:
        s3_client.put_object(Bucket=BUCKET_NAME, Key=key, Body=json.dumps(data, default=str))


def process_bronze(claims):
    """Bronze layer: store raw data as-is."""
    timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
    key = f"bronze/claims_{timestamp}.json"
    write_to_s3(claims, key)
    return claims


def process_silver(claims):
    """Silver layer: validate, clean, and standardize."""
    valid_claims = []
    for claim in claims:
        if not all(k in claim for k in ['claim_id', 'claim_amount', 'claim_type', 'status', 'claim_date']):
            continue
        if claim['claim_amount'] is None or claim['claim_amount'] <= 0:
            continue
        claim['claim_amount'] = float(claim['claim_amount'])
        claim['claim_type'] = claim['claim_type'].lower().strip()
        claim['status'] = claim['status'].lower().strip()
        claim['processed_at'] = datetime.now().isoformat()
        valid_claims.append(claim)

    timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
    key = f"silver/claims_{timestamp}.json"
    write_to_s3(valid_claims, key)
    return valid_claims


def process_gold(claims):
    """Gold layer: aggregate into analytics-ready Parquet."""
    import pyarrow as pa
    import pyarrow.parquet as pq
    import io

    aggregations = {}
    for claim in claims:
        group_key = (claim['claim_type'], claim['status'])
        if group_key not in aggregations:
            aggregations[group_key] = {'total_amount': 0.0, 'claim_count': 0}
        aggregations[group_key]['total_amount'] += claim['claim_amount']
        aggregations[group_key]['claim_count'] += 1

    rows = []
    for (claim_type, status), agg in aggregations.items():
        rows.append({
            'claim_type': claim_type,
            'status': status,
            'total_amount': agg['total_amount'],
            'claim_count': agg['claim_count'],
            'avg_amount': agg['total_amount'] / agg['claim_count']
        })

    table = pa.table({
        'claim_type': [r['claim_type'] for r in rows],
        'status': [r['status'] for r in rows],
        'total_amount': [r['total_amount'] for r in rows],
        'claim_count': [r['claim_count'] for r in rows],
        'avg_amount': [r['avg_amount'] for r in rows],
    })

    buffer = io.BytesIO()
    pq.write_table(table, buffer)
    buffer.seek(0)

    timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
    key = f"gold/claims_summary_{timestamp}.parquet"
    write_to_s3(buffer.read(), key)
    return rows


def handler(event, context):
    """Lambda handler: orchestrate the full pipeline."""
    claims = generate_sample_claims()

    bronze_result = process_bronze(claims)
    print(f"Bronze: wrote {len(bronze_result)} raw claims")

    silver_result = process_silver(bronze_result)
    print(f"Silver: {len(silver_result)} valid claims after cleaning")

    gold_result = process_gold(silver_result)
    print(f"Gold: {len(gold_result)} aggregated groups written as Parquet")

    return {
        'statusCode': 200,
        'body': json.dumps({
            'message': 'Pipeline completed successfully',
            'bronze_count': len(bronze_result),
            'silver_count': len(silver_result),
            'gold_groups': len(gold_result)
        })
    }
Install dependencies and test locally

Your pipeline code imports boto3 and pyarrow. You need to declare these in a requirements.txt file so they can be installed both locally and inside your Docker container later.

  • In VS Code, create a new file called requirements.txt in the same claims-pipeline folder.
  • Paste the following dependencies into requirements.txt:
boto3>=1.35.0
pyarrow>=17.0.0

What are these packages?

boto3 is the AWS SDK for Python. It lets your code interact with S3, Lambda, and other AWS services.

pyarrow is the standard Python library for reading and writing Parquet files. It provides the columnar format that makes your Gold layer query-efficient.

  • Save requirements.txt with Ctrl+S.

Now create a virtual environment and install the dependencies so you can test your code locally.

  • Open a terminal in VS Code by pressing Ctrl+` (backtick key).
  • Create a virtual environment by running:
python -m venv .venv
  • Activate the virtual environment by running:
.venv\Scripts\activate

You should see (.venv) appear at the start of your terminal prompt. This means packages will install into your project's isolated environment.

  • Install the project dependencies by running:
pip install -r requirements.txt

You should see pip downloading and installing boto3, pyarrow, and their sub-dependencies. This may take a minute.

pip not found or installation failed?

If pip is not recognized, try python -m pip install -r requirements.txt instead. Make sure you activated the virtual environment first (the (.venv) prefix should be visible).

Help me troubleshoot my pip installation issue

Now verify that your pipeline logic works by testing the data generation and import structure.

  • Test that your code loads correctly by running:
python -c "from lambda_function import generate_sample_claims, process_silver; claims = generate_sample_claims(); print(f'Generated {len(claims)} claims'); print(claims[0])"

You should see output showing 10 generated claims and the first claim's details (CLM001, auto, 2500.00).

ModuleNotFoundError or ImportError?

Make sure your terminal is in the claims-pipeline folder (where lambda_function.py lives). Run ls to confirm you can see the file listed.

Help me fix my ModuleNotFoundError

Your pipeline code is complete and tested. Next up, you'll package it into a Docker container so it can run on AWS Lambda.

Containerize with Docker

Your pipeline logic is working. You have a handler that processes claims through Bronze, Silver, and Gold layers. The next challenge is packaging this pipeline so it can run anywhere, not just on your local machine.

To deploy your pipeline to AWS Lambda, you need to package it inside a Docker container. This gives Lambda everything it needs to run your code: the Python runtime, your dependencies like pyarrow, and your application code, all bundled together.

In this step, get ready to:
  • Create a Dockerfile that packages your pipeline for Lambda.
  • Build and test the container image locally.
Create the Dockerfile

A Dockerfile is a recipe that tells Docker how to build your container image. For Lambda, AWS provides official base images that include the Lambda runtime interface. Your Dockerfile layers your code and dependencies on top of that base.

  • In VS Code, create a new file called Dockerfile in your project folder (the same folder that contains lambda_function.py).
  • Paste the following code into your Dockerfile:
FROM public.ecr.aws/lambda/python:3.12

COPY requirements.txt ${LAMBDA_TASK_ROOT}

RUN pip install -r requirements.txt

COPY lambda_function.py ${LAMBDA_TASK_ROOT}

CMD ["lambda_function.handler"]

What does this Dockerfile do?

  • FROM public.ecr.aws/lambda/python:3.12 starts from the official AWS Lambda Python 3.12 base image, which includes the Lambda runtime interface emulator for local testing.
  • COPY requirements.txt ${LAMBDA_TASK_ROOT} copies your dependency file into the Lambda working directory.
  • RUN pip install -r requirements.txt installs boto3 and pyarrow inside the container.
  • COPY lambda_function.py ${LAMBDA_TASK_ROOT} adds your pipeline code to the image.
  • CMD ["lambda_function.handler"] tells Lambda which function to invoke when the container runs.

Why containerize instead of a zip upload?

Lambda supports two deployment methods: zip files and container images. Container images are the industry standard for production pipelines because they bundle all dependencies (including native libraries like pyarrow) reliably. Zip deployments often break when packages have compiled components that differ between your machine and Lambda's runtime.

Container images also support up to 10 GB, giving you room for large ML libraries or datasets that zip's 250 MB limit cannot handle.

  • Save Dockerfile.
  • Confirm the file exists by running:
ls

You should see Dockerfile, lambda_function.py, and requirements.txt listed.

Don't see the Dockerfile?

Make sure you created the file in the same folder as lambda_function.py. The file must be named exactly Dockerfile with a capital D and no file extension.

Help me find my Dockerfile

✔️ Awesome, I've got everything!

Great. Double check you've saved your Dockerfile.

ⓧ I'd like to double check the full code

FROM public.ecr.aws/lambda/python:3.12

COPY requirements.txt ${LAMBDA_TASK_ROOT}

RUN pip install -r requirements.txt

COPY lambda_function.py ${LAMBDA_TASK_ROOT}

CMD ["lambda_function.handler"]
Build and test the container locally

Now you will build the container image and run it on your machine. The Lambda base image includes a runtime interface emulator that lets you invoke the handler locally, exactly as Lambda would in the cloud.

  • Make sure Docker Desktop is running. You can check the system tray for the Docker whale icon.
  • Build your container image by running this command in your terminal (from your project folder):
docker buildx build --platform linux/amd64 --provenance=false -t claims-pipeline:latest .

What does this command do?

  • --platform linux/amd64 builds the image for the same architecture Lambda uses, regardless of your local machine's CPU.
  • --provenance=false ensures the image format is compatible with Lambda. Without this flag, the image may fail to deploy.
  • -t claims-pipeline:latest tags the image with a name you can reference later.

The build takes 1-2 minutes as Docker downloads the base image and installs your Python packages. You should see a success message at the end.

Build failed with an error?

If you see a message about connecting to the Docker daemon, make sure Docker Desktop is running. Look for the whale icon in your system tray.

If you see a permissions error, try running your terminal as Administrator.

Help me troubleshoot my Docker build error

  • Start the container locally by running:
docker run --platform linux/amd64 -p 9000:8080 claims-pipeline:latest

What does this command do?

This starts your container and maps port 9000 on your machine to port 8080 inside the container. The Lambda runtime interface emulator listens on port 8080. Your terminal will stay open showing the container logs.

  • Open a second terminal window. Press Cmd+Space (macOS) or the Windows key (Windows), type PowerShell, and press Enter.
  • Invoke your Lambda handler by running this command in the new terminal:
Invoke-WebRequest -Uri "http://localhost:9000/2015-03-31/functions/function/invocations" -Method Post -Body '{}' -ContentType "application/json"

You should see a response from the container confirming the pipeline was invoked.

What should I expect from the response?

A successful response means the container started, received the event, and executed your handler function. The response body will contain a JSON payload with information about the pipeline layers processed.

If you see an error about S3 access or credentials in the response body, that is expected at this stage. The S3 bucket has not been created yet, and the container does not have AWS credentials. The important thing is that the container accepted the invocation and ran your handler. The full end-to-end test happens after deployment in the next step.

  • Return to the first terminal (where the container is running) and press Ctrl+C to stop the container.

Not getting any response?

Make sure the container is still running in your first terminal. If it exited immediately, check the error output for missing files or import errors.

Confirm you are using port 9000 in the URL, not 8080. Port 9000 is the one mapped to your machine.

Help me troubleshoot my local Lambda container connection

Your pipeline is now containerized and ready for the cloud. Next up, you will push this image to Amazon ECR and deploy it as a live Lambda function.

Deploy to AWS

Your pipeline works locally inside a Docker container. Now it is time to deploy it to AWS so it runs as a real serverless function in the cloud.

In this step, you will push your container image to Amazon ECR (Elastic Container Registry), create an IAM execution role, and deploy the image as a Lambda function. When you invoke the function, data will flow through all three medallion layers and land in S3.

In this step, get ready to:
  • Create an S3 bucket and ECR repository for your pipeline.
  • Push your Docker image to ECR.
  • Deploy and invoke the Lambda function from the CLI.
Create the S3 bucket and ECR repository

Your pipeline needs somewhere to store its data and somewhere to store its container image. S3 holds the pipeline output. ECR holds the Docker image that Lambda will pull from.

  • Create an S3 bucket for your pipeline data by running this command:
aws s3 mb s3://claims-pipeline-data-[[ACCOUNTID="your 12-digit AWS account ID"]] --region us-east-1

What does this command do?

The aws s3 mb command creates ("makes") a new S3 bucket. S3 bucket names must be globally unique across all AWS accounts, which is why you include your account ID in the name.

  • Create the ECR repository by running:
aws ecr create-repository --repository-name claims-pipeline --region us-east-1 --image-scanning-configuration scanOnPush=true --image-tag-mutability MUTABLE

What does this command do?

This creates a private container registry called claims-pipeline in your AWS account. The scanOnPush=true flag tells ECR to automatically scan your image for known vulnerabilities when you push it.

The MUTABLE tag setting means you can overwrite the latest tag with updated images in the future.

  • Confirm the repository was created. You should see JSON output containing a repositoryUri that looks like YOURACCOUNTID.dkr.ecr.us-east-1.amazonaws.com/claims-pipeline.

Seeing an error about the repository already existing?

If you previously ran this command, the repository already exists. You can safely continue to the next substep.

If you see an access denied error, confirm your IAM user has the AmazonEC2ContainerRegistryFullAccess policy attached from Step 1.

Ask for help: Why am I getting an error when creating my ECR repository?

Authenticate Docker to ECR and push the image

ECR is a private registry. Docker needs to authenticate before it can push images to it. AWS provides a token-based login that lasts 12 hours.

  • Authenticate Docker to your ECR registry by running:
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin [[ACCOUNTID="your 12-digit AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com

What does this command do?

The first half (aws ecr get-login-password) fetches a temporary authentication token from AWS. The pipe (|) sends that token directly to docker login, which stores the credentials so Docker can push images to your private registry.

  • You should see Login Succeeded in your terminal.
  • Tag your local image with the ECR repository URI by running:
docker tag claims-pipeline:latest [[ACCOUNTID="your 12-digit AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com/claims-pipeline:latest
  • Push the image to ECR by running:
docker push [[ACCOUNTID="your 12-digit AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com/claims-pipeline:latest

The push uploads each layer of your Docker image. This may take a minute depending on your internet speed.

  • Confirm the push completed. You should see multiple Pushed messages followed by the image digest.

Seeing a denied or authorization error?

Make sure you ran the aws ecr get-login-password command first. The login token expires after 12 hours, so re-run it if it has been a while.

Also confirm the account ID in your tag command matches your actual AWS account ID. You can double-check by running aws sts get-caller-identity.

Ask for help: Why is my Docker push to ECR failing with an authorization error?

Create the Lambda function and invoke it

Lambda needs an execution role that grants it permission to access your S3 bucket. You will create this role, then deploy the Lambda function from the container image you just pushed.

  • Create a trust policy file that allows Lambda to assume the role. Run this command to create trust-policy.json:
echo '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"lambda.amazonaws.com"},"Action":"sts:AssumeRole"}]}' > trust-policy.json
  • Create the IAM role using that trust policy by running:
aws iam create-role --role-name lambda-pipeline-role --assume-role-policy-document file://trust-policy.json
  • Attach the S3 full access policy to the role by running:
aws iam attach-role-policy --role-name lambda-pipeline-role --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess

What is an IAM execution role?

When Lambda runs your function, it needs permission to interact with other AWS services. The execution role is an IAM role that Lambda "assumes" at runtime. The trust policy you created tells AWS that the Lambda service is allowed to use this role.

You attached AmazonS3FullAccess so the function can read and write objects in your S3 bucket across the bronze, silver, and gold layers.

  • Wait about 10 seconds for the role to propagate across AWS. Then create the Lambda function from your container image by running:
aws lambda create-function --function-name claims-pipeline --package-type Image --code ImageUri=[[ACCOUNTID="your 12-digit AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com/claims-pipeline:latest --role arn:aws:iam::[[ACCOUNTID="your 12-digit AWS account ID"]]:role/lambda-pipeline-role --timeout 60 --memory-size 512 --environment Variables={BUCKET_NAME=claims-pipeline-data-[[ACCOUNTID="your 12-digit AWS account ID"]]}

What do these flags mean?

  • --package-type Image tells Lambda this function runs from a container image (not a zip file).
  • --code ImageUri=... points Lambda to your ECR image.
  • --timeout 60 gives the function up to 60 seconds to complete. Processing 10 claims through all three layers takes a few seconds.
  • --memory-size 512 allocates 512 MB of memory. Pyarrow needs this to write Parquet files.
  • --environment passes your S3 bucket name as an environment variable so the function knows where to write data.
  • Wait about 30 seconds for the function to become active, then invoke it by running:
aws lambda invoke --function-name claims-pipeline response.json
  • Check the response by running:
cat response.json

You should see a JSON response with statusCode of 200 and a body containing Pipeline completed successfully, along with counts for bronze, silver, and gold layers.

  • Verify the data landed in S3 by listing your bucket contents:
aws s3 ls s3://claims-pipeline-data-[[ACCOUNTID="your 12-digit AWS account ID"]] --recursive

You should see files under bronze/, silver/, and gold/ prefixes. This confirms your pipeline ran end-to-end in the cloud.

Seeing a ResourceNotFoundException or function state error?

Lambda takes time to pull and unpack your container image. If you see a state error, wait 30 seconds and try the invoke command again.

If you see a role error, the IAM role may not have finished propagating. Wait 10 seconds and retry the create-function command.

If the function invokes but response.json shows an error about S3 access, confirm you attached AmazonS3FullAccess to lambda-pipeline-role and that your bucket name matches the one in the --environment flag.

Ask for help: Why is my Lambda function failing to invoke or access S3?

Your containerized data pipeline is now running in production on AWS. Next up, you will query the Gold layer data using SQL in Amazon Athena.

Query Gold Layer with Athena

Your pipeline is deployed and producing data across all three layers. The Gold layer contains analytics-ready Parquet files with aggregated claims data.

Now you need a way to actually query that data using SQL. Amazon Athena is a serverless query engine that lets you run SQL directly against data stored in S3. It integrates with the Glue Data Catalog to understand the schema of your Parquet files.

In this step, get ready to:
  • Create a Glue Data Catalog database and external table pointing to your Gold layer.
  • Run analytical SQL queries against your processed claims data.
Create the Glue Data Catalog database and table

Before you can query your Parquet files, Athena needs to know where the data lives and what columns it contains. You define this by creating a database and an external table in the Glue Data Catalog.

  • In your browser, go to the AWS Console and type Athena in the search bar at the top.
  • Click on Athena in the search results to open the service.
  • If this is your first time visiting Athena, click Explore the query editor to open it.

What is Athena?

Athena is a serverless query service. You do not need to manage any infrastructure. You simply point it at data in S3, define the schema, and run standard SQL queries.

It charges $5 per terabyte of data scanned. On your small Parquet dataset (a few kilobytes), each query costs a fraction of a cent.

Before running any queries, Athena requires a location in S3 to store query results. You will see an Edit Settings prompt at the top of the editor.

  • Click Edit Settings.
  • In the Location of query result box, enter a path for your results bucket. Use the format:
s3://claims-pipeline-data-[[ACCOUNTID="your 12-digit AWS account ID"]]-athena-results/

Reuse your existing bucket

You can also use your existing pipeline bucket by entering s3://claims-pipeline-data-YOURACCOUNTID/athena-results/ (replacing YOURACCOUNTID with your actual account ID). Athena creates the prefix automatically.

  • Click Save.
  • Click Editor in the navigation bar to switch back to the query editor.

Now you are ready to create the database. The query editor has a text area where you type SQL statements.

  • Create the Glue Data Catalog database by pasting this SQL into the query editor and clicking Run:
CREATE DATABASE claims_db;

What does this do?

This creates a new database in the Glue Data Catalog. The database acts as a namespace to organize your tables. Athena uses the Glue Data Catalog as its metadata store.

  • Confirm the query completes successfully. You should see a green success message below the editor.
  • From the Database dropdown on the left side of the query editor, select claims_db to set it as your current database.

Next, create an external table that maps to your Gold layer Parquet data in S3.

  • Clear the query editor, then paste this SQL and click Run:
CREATE EXTERNAL TABLE claims_db.gold_claims_summary (
  claim_type string,
  status string,
  total_amount double,
  claim_count bigint,
  avg_amount double
)
STORED AS PARQUET
LOCATION 's3://claims-pipeline-data-[[ACCOUNTID="your 12-digit AWS account ID"]]/gold/';

What does this table definition do?

This tells Athena where to find your data and what shape it has. The LOCATION points to the S3 prefix where your Gold layer Parquet files live.

The column names and types match exactly what process_gold() writes in your pipeline. Athena reads the Parquet metadata to confirm the schema matches at query time.

  • Confirm you see a green success message after running the CREATE EXTERNAL TABLE statement.

Seeing a permission error?

Make sure your IAM user has the AmazonAthenaFullAccess and AWSGlueConsoleFullAccess policies attached. Also check that the S3 bucket path in your LOCATION matches your actual bucket name exactly.

help me fix a permission error when creating a table in Athena

Run analytical queries against the Gold layer

Your table is ready. Now you can run SQL queries against your Gold layer data just like you would with a traditional database.

  • Clear the query editor, paste this query, and click Run:
SELECT * FROM claims_db.gold_claims_summary ORDER BY total_amount DESC;

What does this query show?

This retrieves all rows from your Gold summary table and orders them by total claim amount, highest first. You should see each combination of claim type and status with its aggregated totals.

  • Check the results panel below the editor. You should see rows with columns: claim_type, status, total_amount, claim_count, and avg_amount.

Now try a more targeted analytical query that groups claims by type.

  • Clear the editor, paste this query, and click Run:
SELECT claim_type, SUM(claim_count) as total_claims FROM claims_db.gold_claims_summary GROUP BY claim_type;

What does this query show?

This aggregates across all statuses to show the total number of claims per type (auto, health, home). In a real pipeline, this is the kind of query a data analyst would run daily to track claim volumes.

  • Confirm the results show a row for each claim type (auto, health, home) with the total number of claims in each category.

Getting zero rows or an error?

If the query returns zero rows, your Gold layer may not have data. Go back to Step 4 and re-invoke the Lambda function. Then check your S3 bucket for files under the gold/ prefix.

If you see a schema mismatch error, double-check that the column names and types in your CREATE EXTERNAL TABLE match the ones shown above exactly.

help me debug why my Athena query returns no results from my Gold layer Parquet data

Your pipeline is now fully deployed and queryable. You have data flowing from raw ingestion (Bronze) through validation (Silver) to analytics-ready aggregations (Gold), all accessible through standard SQL.

Secret mission

Automated Scheduling and Data Quality Gates

Your pipeline works, but it trusts every record blindly. In production, bad data costs money. What if a $5 million claim slips through? What if someone submits a future-dated record? Add a data quality gate that catches anomalies before they reach your Gold layer, then wire up an EventBridge schedule so the pipeline runs itself every day without you lifting a finger.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

This project uses AWS services that fall within the Free Tier for normal usage. Your Lambda function only costs money when invoked, and S3 storage for this project's data is pennies per month.

Resources you used:

  • S3 bucket (claims-pipeline-data-YOURACCOUNTID) storing Bronze, Silver, and Gold layer data.
  • ECR repository (claims-pipeline) storing your container image.
  • Lambda function (claims-pipeline) deployed from the container image.
  • IAM roles (lambda-pipeline-role and eventbridge-lambda-role).
  • Glue Data Catalog database (claims_db) and table (gold_claims_summary).
  • EventBridge Scheduler rule (claims-pipeline-daily) if you completed the Secret Mission.
  • Local project files (lambda_function.py, requirements.txt, Dockerfile).

Keep everything running

No action needed. Choose this if you are still actively building or want to invoke your pipeline again later.

  • Your Lambda function only costs money when invoked. With the daily EventBridge schedule (Secret Mission), you use 30 invocations per month, well within the 1,000,000 free invocations per month.
  • S3 storage for this project's data files is a few pennies per month.
  • ECR storage is free for up to 500 MB under Free Tier credits.
  • Total ongoing cost is effectively $0.

Pause - I'll come back to this later

Delete the EventBridge schedule to stop automatic daily invocations. Everything else stays dormant at no cost until you invoke it manually.

  • Delete the EventBridge schedule (Secret Mission only) by running:
  • Everything else (Lambda, S3, ECR, Glue) remains in place with no ongoing charges beyond minimal storage.
aws scheduler delete-schedule --name claims-pipeline-daily

Your Lambda function, S3 data, and ECR image remain available whenever you want to resume.

Delete - I don't want to use this again

Remove all project resources and start fresh. Run the following commands in your terminal to fully clean up.

  • Delete the Lambda function:
aws lambda delete-function --function-name claims-pipeline
  • Delete the ECR repository and its images:
aws ecr delete-repository --repository-name claims-pipeline --force
  • Delete the S3 bucket and all its contents:
aws s3 rb s3://[[BUCKET_NAME="claims-pipeline-data-YOURACCOUNTID"]] --force
  • Delete the Glue Data Catalog database and table by opening the Athena query editor in the AWS Console and running:
DROP DATABASE claims_db CASCADE;
  • Delete the EventBridge schedule (Secret Mission only):
aws scheduler delete-schedule --name claims-pipeline-daily
  • Delete the IAM roles. You must detach policies before deleting each role. For lambda-pipeline-role:
aws iam detach-role-policy --role-name lambda-pipeline-role --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess
aws iam delete-role --role-name lambda-pipeline-role
  • For eventbridge-lambda-role (Secret Mission only):
aws iam detach-role-policy --role-name eventbridge-lambda-role --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaRole
aws iam delete-role --role-name eventbridge-lambda-role

What about the Athena query results bucket?

Athena automatically created a results bucket when you first ran a query. You can find and delete it in the S3 console by searching for buckets with "athena" in the name. This bucket may contain query result CSV files from your sessions.

  • Delete your local project files by running this in your terminal:
cd ~/Desktop
rm -rf claims-pipeline

All resources have been removed from your AWS account and local machine.

Nice Work!

Nice Work!

Nice work. You just built, containerized, and deployed a production-grade data pipeline to AWS.

You've learned how to:

  • Build a Python ETL pipeline that processes insurance claims through a Medallion architecture (Bronze, Silver, and Gold layers).
  • Containerize the pipeline with Docker and deploy it to AWS Lambda via Amazon ECR.
  • Create a Glue Data Catalog table and query your Gold layer with Amazon Athena using SQL.
  • Secret Mission: Add automated daily scheduling with EventBridge Scheduler and data quality gates that reject anomalous claims.

Ready to quiz yourself?