Serverless Data Lake Pipeline with Terraform

Build a Terraform-managed AWS pipeline from DynamoDB to Redshift and QuickSight.

Introduction

30 Second Summary

Every time you place an online order, that transaction kicks off a chain of data movement. The order gets recorded, transformed, cataloged, and eventually shows up as a dot on a revenue chart. Building the infrastructure behind that journey is what data engineers do every day.

In this project, you will build a serverless data lake pipeline on AWS that captures e-commerce order data, streams it into a data lake, catalogs it automatically, loads it into a data warehouse, and visualizes business insights. You will provision the entire stack using Terraform.

What You'll Build

A fully automated, serverless analytics pipeline on AWS that moves data from an operational database through a data lake to a data warehouse and BI dashboard.

By the end of this project, you'll have:

  • A DynamoDB table with Streams enabled that captures order data, a Lambda function that streams records into an S3 data lake with raw and processed zones.
  • A Glue Crawler that automatically discovers schemas in your data lake, a Redshift Serverless warehouse for analytics queries, and a CloudWatch dashboard monitoring pipeline health.
  • A QuickSight dashboard connected to Redshift that visualizes business insights, proving end-to-end data flow from source to BI layer.
  • Secret Mission: Add a second Lambda function that transforms raw JSON into Parquet format in the processed zone and configure the Glue Crawler to catalog both zones.

Are there any prerequisites?

You will need an AWS account with admin access and basic familiarity with Terraform and AWS console navigation. This project also requires signing up for QuickSight Standard Edition ($9/month) and uses the Redshift Serverless $300 free trial credit.

Provision the Data Source Layer

Your environment is set up with Terraform and the AWS CLI ready to go. Now it's time to define the foundational resources your pipeline depends on.

Every data pipeline starts with a source and a destination. In this step, you'll provision the two resources that form the base of your architecture: a DynamoDB table to capture e-commerce orders and an S3 bucket structured as a data lake with separate zones for raw and processed data.

In this step, get ready to:
  • Write Terraform provider configuration for AWS.
  • Create an S3 bucket with raw and processed zone prefixes.
  • Create a DynamoDB table with Streams enabled and deploy the infrastructure.
Configure the AWS provider

Terraform needs to know which cloud provider and region to target. The terraform block pins the provider version, and the provider block sets the AWS region.

  • Create a file called main.tf in your project directory.
  • Create a subdirectory called lambda by running this command:
mkdir lambda
  • Add the following Terraform and provider configuration to main.tf:
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
  required_version = ">= 1.5.0"
}

provider "aws" {
  region = "us-east-1"
}

What does this code do?

The terraform block declares that your project uses the official AWS provider from HashiCorp, pinned to version ~> 6.0. This constraint means Terraform will accept any 6.x release but won't jump to 7.0.

The provider block tells Terraform to deploy resources in the us-east-1 region. This matches the region your AWS CLI is configured for.

  • Save main.tf.
  • You won't see any AWS resources yet. The next chunk adds the S3 bucket.
Create the S3 data lake bucket

A data lake organizes data into zones. The raw/ zone stores data exactly as it arrives from the source. The processed/ zone holds transformed, analysis-ready data.

  • In main.tf, add the following S3 resources below the provider block:
# ---------- S3 Data Lake ----------
resource "aws_s3_bucket" "data_lake" {
  bucket_prefix = "data-lake-pipeline-"
  force_destroy = true
}

resource "aws_s3_object" "raw_prefix" {
  bucket  = aws_s3_bucket.data_lake.id
  key     = "raw/"
  content = ""
}

resource "aws_s3_object" "processed_prefix" {
  bucket  = aws_s3_bucket.data_lake.id
  key     = "processed/"
  content = ""
}

What does this code do?

The aws_s3_bucket resource creates a bucket with a randomized suffix (thanks to bucket_prefix). Setting force_destroy = true lets Terraform delete the bucket even if it contains objects, which makes cleanup easier later.

The two aws_s3_object resources create empty objects with trailing slashes. These act as folder markers, establishing the raw/ and processed/ prefixes that organize your data lake into zones.

  • Save main.tf.
  • You won't see any change in AWS yet. The next chunk adds the DynamoDB table, then we'll deploy everything together.
Create the DynamoDB table and deploy

DynamoDB Streams capture every change made to a table and emit it as an event. Your Lambda function (built in the next step) will consume these events to feed data into the lake.

  • In main.tf, add the DynamoDB table resource below the S3 resources:
# ---------- DynamoDB ----------
resource "aws_dynamodb_table" "orders" {
  name             = "orders"
  billing_mode     = "PAY_PER_REQUEST"
  hash_key         = "order_id"
  stream_enabled   = true
  stream_view_type = "NEW_AND_OLD_IMAGES"

  attribute {
    name = "order_id"
    type = "S"
  }
}

What does this code do?

This creates a DynamoDB table called orders with on-demand billing (no capacity planning needed). The order_id attribute is the partition key, typed as a string (S).

Setting stream_enabled = true activates DynamoDB Streams on this table. The NEW_AND_OLD_IMAGES view type means each stream event includes both the previous and updated state of the record. This gives your pipeline the full picture of every change.

Why PAY_PER_REQUEST instead of provisioned capacity?

On-demand billing charges per request with no upfront capacity planning. For a learning project with low traffic, this keeps costs near zero while staying within the AWS free tier.

  • Save main.tf.

✔️ Awesome, I've got everything!

Great. Double check you've saved main.tf before moving on.

ⓧ I'd like to double check the full code

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
  required_version = ">= 1.5.0"
}

provider "aws" {
  region = "us-east-1"
}

# ---------- S3 Data Lake ----------
resource "aws_s3_bucket" "data_lake" {
  bucket_prefix = "data-lake-pipeline-"
  force_destroy = true
}

resource "aws_s3_object" "raw_prefix" {
  bucket  = aws_s3_bucket.data_lake.id
  key     = "raw/"
  content = ""
}

resource "aws_s3_object" "processed_prefix" {
  bucket  = aws_s3_bucket.data_lake.id
  key     = "processed/"
  content = ""
}

# ---------- DynamoDB ----------
resource "aws_dynamodb_table" "orders" {
  name             = "orders"
  billing_mode     = "PAY_PER_REQUEST"
  hash_key         = "order_id"
  stream_enabled   = true
  stream_view_type = "NEW_AND_OLD_IMAGES"

  attribute {
    name = "order_id"
    type = "S"
  }
}

Now create the outputs file so Terraform prints useful values after deployment.

  • Create a new file called outputs.tf in the same directory as main.tf.
  • Add the following output definitions to outputs.tf:
output "s3_bucket_name" {
  description = "Name of the data lake S3 bucket"
  value       = aws_s3_bucket.data_lake.id
}

output "dynamodb_table_name" {
  description = "Name of the DynamoDB orders table"
  value       = aws_dynamodb_table.orders.name
}

What do outputs do?

Terraform outputs print important values to your terminal after a successful apply. Here you're surfacing the generated S3 bucket name (which includes a random suffix) and the DynamoDB table name. You'll reference these values in later steps.

  • Save outputs.tf.

✔️ Awesome, I've got everything!

Great. Double check you've saved outputs.tf before moving on.

ⓧ I'd like to double check the full code

output "s3_bucket_name" {
  description = "Name of the data lake S3 bucket"
  value       = aws_s3_bucket.data_lake.id
}

output "dynamodb_table_name" {
  description = "Name of the DynamoDB orders table"
  value       = aws_dynamodb_table.orders.name
}

Time to deploy. Terraform uses a two-step process: init downloads the provider plugins, and apply creates the actual AWS resources.

  • Initialize the Terraform working directory by running this command:
terraform init
  • You should see a message that says Terraform has been successfully initialized! in the output.

Seeing a provider error?

If terraform init fails with a provider version error, make sure your main.tf has version = "~> 6.0" inside the required_providers block. Also confirm your Terraform CLI version is 1.5.0 or higher by running terraform --version.

help me debug a terraform init provider error

  • Deploy your infrastructure by running:
terraform apply

What happens during apply?

Terraform shows you a plan of what it will create, then asks for confirmation. Type yes and press Enter to approve.

  • After the apply finishes, you should see two outputs printed in your terminal: s3_bucket_name and dynamodb_table_name.

Apply failed with an access denied error?

This usually means your AWS CLI credentials don't have sufficient permissions. Confirm your IAM user has the AdministratorAccess policy attached, or at minimum permissions for S3 and DynamoDB.

Run aws sts get-caller-identity to verify which IAM identity Terraform is using.

help me fix a terraform apply access denied error

Your data source layer is live. Next up, you'll build a Lambda function that listens to DynamoDB Streams and writes incoming records into the S3 data lake.

Build the Streaming Ingestion Layer

Your data lake bucket and DynamoDB table are provisioned. Now you need a way to automatically move data from DynamoDB into the S3 raw zone whenever a new order arrives.

In this step, you will create a Lambda function that listens to DynamoDB Streams and writes each change event as a JSON file to S3. This is the streaming ingestion layer that turns your static data lake into a living, continuously updated system.

In this step, get ready to:
  • Create the IAM role and policies for the Lambda function.
  • Write the Lambda function code that reads DynamoDB Stream events and writes JSON to S3.
  • Deploy the Lambda function and connect it to the DynamoDB Stream.
Create the IAM role and policy for Lambda

Before the Lambda function can read from DynamoDB Streams and write to S3, it needs an IAM role that grants exactly those permissions.

  • In main.tf, add the following IAM configuration below the DynamoDB resource block:
# ---------- IAM for Lambda ----------
resource "aws_iam_role" "lambda_role" {
  name = "data-lake-lambda-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = {
        Service = "lambda.amazonaws.com"
      }
    }]
  })
}

resource "aws_iam_role_policy" "lambda_policy" {
  name = "data-lake-lambda-policy"
  role = aws_iam_role.lambda_role.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "dynamodb:GetRecords",
          "dynamodb:GetShardIterator",
          "dynamodb:DescribeStream",
          "dynamodb:ListStreams"
        ]
        Resource = aws_dynamodb_table.orders.stream_arn
      },
      {
        Effect = "Allow"
        Action = [
          "s3:PutObject"
        ]
        Resource = "${aws_s3_bucket.data_lake.arn}/raw/*"
      },
      {
        Effect = "Allow"
        Action = [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents"
        ]
        Resource = "arn:aws:logs:*:*:*"
      }
    ]
  })
}

What does this code do?

  • aws_iam_role.lambda_role creates a role that only the Lambda service can assume. This is the identity your function runs under.
  • The first policy statement grants the four DynamoDB Streams permissions Lambda needs to poll and read stream records.
  • The second statement allows writing objects only to the raw/ prefix. This is the principle of least privilege in action.
  • The third statement grants CloudWatch Logs access so Lambda can write execution logs for debugging.
  • Save main.tf.

You won't see a change yet. The next chunk adds the Lambda function that uses this role.

Seeing a syntax error?

  • Make sure the IAM block is placed after the closing brace of the aws_dynamodb_table resource.
  • Check that every opening brace { has a matching closing brace }.
  • help me fix a Terraform syntax error in my IAM role configuration
Write the Lambda function code

The Lambda function receives batches of DynamoDB Stream events. For each INSERT or MODIFY event, it converts the DynamoDB JSON format into standard JSON and writes it to the S3 raw zone.

  • Create a new file called stream_to_s3.py inside the lambda/ directory of your project, and paste the following code:
import json
import os
import boto3
from datetime import datetime

s3 = boto3.client("s3")

BUCKET_NAME = os.environ["BUCKET_NAME"]
PREFIX = os.environ["PREFIX"]


def handler(event, context):
    records = event.get("Records", [])
    if not records:
        return {"statusCode": 200, "body": "No records to process"}

    for record in records:
        if record["eventName"] in ("INSERT", "MODIFY"):
            new_image = record["dynamodb"].get("NewImage", {})
            # Convert DynamoDB JSON to standard JSON
            item = deserialize_dynamodb_item(new_image)
            item["_stream_timestamp"] = datetime.utcnow().isoformat()
            item["_event_name"] = record["eventName"]

            # Write to S3
            timestamp = datetime.utcnow().strftime("%Y/%m/%d/%H")
            key = f"{PREFIX}{timestamp}/{item.get('order_id', 'unknown')}_{datetime.utcnow().strftime('%Y%m%d%H%M%S%f')}.json"

            s3.put_object(
                Bucket=BUCKET_NAME,
                Key=key,
                Body=json.dumps(item),
                ContentType="application/json",
            )

    return {"statusCode": 200, "body": f"Processed {len(records)} records"}


def deserialize_dynamodb_item(dynamodb_item):
    """Convert DynamoDB JSON format to standard Python dict."""
    result = {}
    for key, value in dynamodb_item.items():
        if "S" in value:
            result[key] = value["S"]
        elif "N" in value:
            result[key] = float(value["N"])
        elif "BOOL" in value:
            result[key] = value["BOOL"]
        elif "NULL" in value:
            result[key] = None
        elif "L" in value:
            result[key] = [deserialize_dynamodb_item({"v": v})["v"] for v in value["L"]]
        elif "M" in value:
            result[key] = deserialize_dynamodb_item(value["M"])
        else:
            result[key] = str(value)
    return result

What does this code do?

The handler function is the entry point Lambda invokes. It loops over each stream record, filters for INSERT and MODIFY events, and writes the item data as JSON to S3.

  • BUCKET_NAME and PREFIX come from environment variables that Terraform sets on the function.
  • The S3 key uses a date-partitioned path (raw/2026/05/20/14/) so files are organized by time. This makes downstream querying more efficient.
  • deserialize_dynamodb_item converts DynamoDB's typed JSON format (e.g. {"S": "value"}) into plain Python dictionaries.
  • Save lambda/stream_to_s3.py.
  • Confirm the file exists by running:
ls lambda/

You should see stream_to_s3.py in the output.

Don't see the file?

  • Make sure you created the file inside the lambda/ subdirectory, not in the project root.
  • Check that the filename is exactly stream_to_s3.py with underscores, not hyphens.
  • help me find where I saved my Lambda function file
Deploy the Lambda and connect the stream

Now you will define the Lambda function resource itself, package the Python code into a zip, and wire it up to the DynamoDB Stream so it triggers automatically.

  • In main.tf, add the following block below the IAM policy you just created:
# ---------- Lambda Function ----------
data "archive_file" "lambda_zip" {
  type        = "zip"
  source_file = "${path.module}/lambda/stream_to_s3.py"
  output_path = "${path.module}/lambda/stream_to_s3.zip"
}

resource "aws_lambda_function" "stream_to_s3" {
  filename         = data.archive_file.lambda_zip.output_path
  function_name    = "stream-to-s3"
  role             = aws_iam_role.lambda_role.arn
  handler          = "stream_to_s3.handler"
  runtime          = "python3.12"
  source_code_hash = data.archive_file.lambda_zip.output_base64sha256
  timeout          = 60

  environment {
    variables = {
      BUCKET_NAME = aws_s3_bucket.data_lake.id
      PREFIX      = "raw/"
    }
  }
}

resource "aws_lambda_event_source_mapping" "dynamodb_stream" {
  event_source_arn  = aws_dynamodb_table.orders.stream_arn
  function_name     = aws_lambda_function.stream_to_s3.arn
  starting_position = "LATEST"
  batch_size        = 10
}

What does this code do?

  • archive_file is a Terraform data source that zips your Python file at plan time. You do not need to manually zip it before deploying.
  • aws_lambda_function.stream_to_s3 creates the function with the python3.12 runtime. The source_code_hash ensures Terraform redeploys whenever your Python code changes.
  • aws_lambda_event_source_mapping.dynamodb_stream connects the Lambda to the DynamoDB Stream. starting_position = "LATEST" means it only processes new records, not historical ones.
  • batch_size = 10 means Lambda receives up to 10 stream records per invocation, balancing throughput and cost.
  • Now add the Lambda output to outputs.tf. Add the following block at the end of the file:
output "lambda_function_name" {
  description = "Name of the stream-to-S3 Lambda function"
  value       = aws_lambda_function.stream_to_s3.function_name
}
  • Save both main.tf and outputs.tf.
  • Deploy all the new resources by running:
terraform apply -auto-approve

Terraform will create the IAM role, Lambda function, and event source mapping. You should see output ending with Apply complete! and a count of resources added.

Seeing an error during apply?

  • If you see a file-not-found error for stream_to_s3.py, make sure the file is at lambda/stream_to_s3.py relative to your main.tf.
  • If you see an IAM permission error, wait 10-15 seconds and run terraform apply again. IAM role propagation can take a moment.
  • help me debug my terraform apply error for the Lambda function

Now test the pipeline end-to-end by inserting a record into DynamoDB.

  • Insert a test order into the orders table by running:
aws dynamodb put-item --table-name orders --item '{"order_id": {"S": "order-001"}, "status": {"S": "pending"}, "amount": {"N": "49.99"}}'

What does this command do?

This inserts a sample e-commerce order with an ID, status, and amount. Because DynamoDB Streams is enabled, this INSERT event triggers your Lambda function, which writes the record to the S3 raw zone.

  • Wait about 10 seconds for the stream event to trigger Lambda, then verify the file landed in S3 by running:
aws s3 ls s3://$(terraform output -raw s3_bucket_name)/raw/ --recursive

You should see at least one .json file listed under the raw/ prefix with today's date in the path.

No files in the raw/ prefix?

  • Wait 30 seconds and try the aws s3 ls command again. The first Lambda cold start can take a few seconds.
  • Check the Lambda logs for errors by running aws logs tail /aws/lambda/stream-to-s3 --since 5m to see if the function encountered a permissions issue.
  • Confirm the event source mapping is active by running aws lambda list-event-source-mappings --function-name stream-to-s3 and checking the State field shows Enabled.
  • help me debug why my Lambda is not writing to S3

✔️ Awesome, I've got everything!

Great. Double check you have saved both main.tf and outputs.tf.

ⓧ I'd like to double check the full code

Here is the full main.tf at this point:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
  required_version = ">= 1.5.0"
}

provider "aws" {
  region = "us-east-1"
}

# ---------- S3 Data Lake ----------
resource "aws_s3_bucket" "data_lake" {
  bucket_prefix = "data-lake-pipeline-"
  force_destroy = true
}

resource "aws_s3_object" "raw_prefix" {
  bucket  = aws_s3_bucket.data_lake.id
  key     = "raw/"
  content = ""
}

resource "aws_s3_object" "processed_prefix" {
  bucket  = aws_s3_bucket.data_lake.id
  key     = "processed/"
  content = ""
}

# ---------- DynamoDB ----------
resource "aws_dynamodb_table" "orders" {
  name             = "orders"
  billing_mode     = "PAY_PER_REQUEST"
  hash_key         = "order_id"
  stream_enabled   = true
  stream_view_type = "NEW_AND_OLD_IMAGES"

  attribute {
    name = "order_id"
    type = "S"
  }
}

# ---------- IAM for Lambda ----------
resource "aws_iam_role" "lambda_role" {
  name = "data-lake-lambda-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = {
        Service = "lambda.amazonaws.com"
      }
    }]
  })
}

resource "aws_iam_role_policy" "lambda_policy" {
  name = "data-lake-lambda-policy"
  role = aws_iam_role.lambda_role.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "dynamodb:GetRecords",
          "dynamodb:GetShardIterator",
          "dynamodb:DescribeStream",
          "dynamodb:ListStreams"
        ]
        Resource = aws_dynamodb_table.orders.stream_arn
      },
      {
        Effect = "Allow"
        Action = [
          "s3:PutObject"
        ]
        Resource = "${aws_s3_bucket.data_lake.arn}/raw/*"
      },
      {
        Effect = "Allow"
        Action = [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents"
        ]
        Resource = "arn:aws:logs:*:*:*"
      }
    ]
  })
}

# ---------- Lambda Function ----------
data "archive_file" "lambda_zip" {
  type        = "zip"
  source_file = "${path.module}/lambda/stream_to_s3.py"
  output_path = "${path.module}/lambda/stream_to_s3.zip"
}

resource "aws_lambda_function" "stream_to_s3" {
  filename         = data.archive_file.lambda_zip.output_path
  function_name    = "stream-to-s3"
  role             = aws_iam_role.lambda_role.arn
  handler          = "stream_to_s3.handler"
  runtime          = "python3.12"
  source_code_hash = data.archive_file.lambda_zip.output_base64sha256
  timeout          = 60

  environment {
    variables = {
      BUCKET_NAME = aws_s3_bucket.data_lake.id
      PREFIX      = "raw/"
    }
  }
}

resource "aws_lambda_event_source_mapping" "dynamodb_stream" {
  event_source_arn  = aws_dynamodb_table.orders.stream_arn
  function_name     = aws_lambda_function.stream_to_s3.arn
  starting_position = "LATEST"
  batch_size        = 10
}

Here is the full outputs.tf:

output "s3_bucket_name" {
  description = "Name of the data lake S3 bucket"
  value       = aws_s3_bucket.data_lake.id
}

output "dynamodb_table_name" {
  description = "Name of the DynamoDB orders table"
  value       = aws_dynamodb_table.orders.name
}

output "lambda_function_name" {
  description = "Name of the stream-to-S3 Lambda function"
  value       = aws_lambda_function.stream_to_s3.function_name
}

Here is the full lambda/stream_to_s3.py:

import json
import os
import boto3
from datetime import datetime

s3 = boto3.client("s3")

BUCKET_NAME = os.environ["BUCKET_NAME"]
PREFIX = os.environ["PREFIX"]


def handler(event, context):
    records = event.get("Records", [])
    if not records:
        return {"statusCode": 200, "body": "No records to process"}

    for record in records:
        if record["eventName"] in ("INSERT", "MODIFY"):
            new_image = record["dynamodb"].get("NewImage", {})
            # Convert DynamoDB JSON to standard JSON
            item = deserialize_dynamodb_item(new_image)
            item["_stream_timestamp"] = datetime.utcnow().isoformat()
            item["_event_name"] = record["eventName"]

            # Write to S3
            timestamp = datetime.utcnow().strftime("%Y/%m/%d/%H")
            key = f"{PREFIX}{timestamp}/{item.get('order_id', 'unknown')}_{datetime.utcnow().strftime('%Y%m%d%H%M%S%f')}.json"

            s3.put_object(
                Bucket=BUCKET_NAME,
                Key=key,
                Body=json.dumps(item),
                ContentType="application/json",
            )

    return {"statusCode": 200, "body": f"Processed {len(records)} records"}


def deserialize_dynamodb_item(dynamodb_item):
    """Convert DynamoDB JSON format to standard Python dict."""
    result = {}
    for key, value in dynamodb_item.items():
        if "S" in value:
            result[key] = value["S"]
        elif "N" in value:
            result[key] = float(value["N"])
        elif "BOOL" in value:
            result[key] = value["BOOL"]
        elif "NULL" in value:
            result[key] = None
        elif "L" in value:
            result[key] = [deserialize_dynamodb_item({"v": v})["v"] for v in value["L"]]
        elif "M" in value:
            result[key] = deserialize_dynamodb_item(value["M"])
        else:
            result[key] = str(value)
    return result

Your streaming ingestion layer is live. Every new or modified order in DynamoDB now flows automatically into your S3 data lake. Next up, you will configure a Glue Crawler to catalog the data so it can be queried.

Configure the Glue Data Catalog and Crawler

Your Lambda function is now streaming order data into the S3 data lake's raw zone. The next challenge is discoverability. Without a catalog, downstream services like Redshift have no way of knowing what schemas exist in your data lake or where to find them.

The AWS Glue Data Catalog solves this by acting as a central metadata repository. A Glue Crawler automatically scans your S3 data, infers the schema, and registers tables in the catalog. In this step, you will create the catalog database, set up IAM permissions for the crawler, and deploy the crawler itself.

In this step, get ready to:
  • Create a Glue Catalog Database to hold your data lake table definitions.
  • Create an IAM role for the Glue Crawler with the required policies.
  • Deploy the Glue Crawler targeting the S3 raw/ prefix and run it to verify schema discovery.
Create the Glue Catalog Database

A Glue Catalog Database is a logical grouping of table definitions. Think of it as a namespace that keeps your data lake's metadata organized.

  • In main.tf, add the following code at the bottom of the file to create the Glue Data Catalog section:
# ---------- Glue Data Catalog ----------
resource "aws_glue_catalog_database" "data_lake_db" {
  name = "data_lake_db"
}

What does this code do?

This creates a Glue Catalog Database named data_lake_db. All tables that the crawler discovers will be registered under this database name.

Glue database names must be lowercase letters, numbers, and underscores only.

Create the IAM role for the Glue Crawler

The Glue Crawler needs permission to access the Glue service APIs and read objects in your S3 bucket. You will create an IAM role with two permission sources: the AWS-managed AWSGlueServiceRole policy and a custom inline policy for S3 access.

  • Add the IAM role and policies below the Glue Catalog Database resource in main.tf:
resource "aws_iam_role" "glue_role" {
  name = "data-lake-glue-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = {
        Service = "glue.amazonaws.com"
      }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "glue_service" {
  role       = aws_iam_role.glue_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole"
}

resource "aws_iam_role_policy" "glue_s3_policy" {
  name = "glue-s3-access"
  role = aws_iam_role.glue_role.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Action = [
        "s3:GetObject",
        "s3:ListBucket"
      ]
      Resource = [
        aws_s3_bucket.data_lake.arn,
        "${aws_s3_bucket.data_lake.arn}/*"
      ]
    }]
  })
}

What does this code do?

The IAM role uses a two-part permission strategy:

  • AWSGlueServiceRole is an AWS-managed policy that grants the crawler permission to write metadata to the Glue Data Catalog and emit logs to CloudWatch.
  • The inline glue-s3-access policy grants s3:GetObject and s3:ListBucket so the crawler can read and list files in your data lake bucket.
  • The assume_role_policy allows the Glue service (glue.amazonaws.com) to assume this role when running the crawler.

Why two separate policies?

AWS-managed policies like AWSGlueServiceRole cover the baseline Glue permissions that every crawler needs. The inline policy adds only the specific S3 bucket access your project requires.

This pattern keeps permissions minimal and auditable. If you later add a second bucket, you update one inline policy rather than creating a broad custom role from scratch.

  • Save main.tf.
Create the Glue Crawler and deploy

The crawler is the component that actually scans your S3 data and populates the catalog. You will point it at the raw/ prefix so it discovers the JSON files your Lambda function writes.

  • Add the Glue Crawler resource below the IAM policies in main.tf:
resource "aws_glue_crawler" "raw_crawler" {
  database_name = aws_glue_catalog_database.data_lake_db.name
  name          = "raw-zone-crawler"
  role          = aws_iam_role.glue_role.arn

  s3_target {
    path = "s3://${aws_s3_bucket.data_lake.id}/raw/"
  }
}

What does this code do?

  • database_name tells the crawler which catalog database to register discovered tables in.
  • s3_target.path points the crawler at the raw/ prefix of your data lake bucket. It will scan all JSON files under this path.
  • No schedule is defined, so the crawler runs on-demand only. You will trigger it manually to verify it works.
  • Now add a Terraform output so you can reference the Glue database name later. Open outputs.tf and add the following at the bottom:
output "glue_database_name" {
  description = "Glue Catalog database name"
  value       = aws_glue_catalog_database.data_lake_db.name
}
  • Save outputs.tf.
  • Deploy the new resources by running:
terraform apply -auto-approve
  • Wait for Terraform to finish. You should see 4 new resources added (the catalog database, IAM role, policy attachment, inline policy, and crawler).

Seeing an access denied error?

  • Make sure your AWS CLI credentials have permission to create Glue and IAM resources.
  • Check that the policy_arn for the Glue service role attachment matches arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole exactly.

Still stuck? Help me debug a Terraform apply error for Glue resources.

  • Now run the crawler manually to verify it discovers schemas in your data lake. Start it by running:
aws glue start-crawler --name raw-zone-crawler

How long does the crawler take?

The first run typically takes 1-2 minutes for a small dataset. The crawler reads the JSON files, infers the schema (column names and types), and registers a table in the Glue Data Catalog.

  • Wait about 2 minutes, then verify that the crawler created a table by running:
aws glue get-tables --database-name data_lake_db
  • You should see a TableList array containing at least one table. The table name is auto-generated from the S3 path (typically something like raw).

TableList is empty?

  • Make sure the crawler has finished running. You can check its status in the AWS Glue console under **Crawlers**.
  • If the crawler succeeded but found no tables, verify that JSON files exist in your S3 bucket under the raw/ prefix. Go back to the previous step and insert a test item into DynamoDB if needed.
  • Check the crawler's IAM role has the s3:GetObject and s3:ListBucket permissions pointing at the correct bucket ARN.

Help me figure out why my Glue Crawler isn't finding tables.

✔️ Awesome, I've got everything!

Great. Double check you have saved both main.tf and outputs.tf.

ⓧ I'd like to double check the full code

Here is the complete main.tf up to this point:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
  required_version = ">= 1.5.0"
}

provider "aws" {
  region = "us-east-1"
}

# ---------- S3 Data Lake ----------
resource "aws_s3_bucket" "data_lake" {
  bucket_prefix = "data-lake-pipeline-"
  force_destroy = true
}

resource "aws_s3_object" "raw_prefix" {
  bucket  = aws_s3_bucket.data_lake.id
  key     = "raw/"
  content = ""
}

resource "aws_s3_object" "processed_prefix" {
  bucket  = aws_s3_bucket.data_lake.id
  key     = "processed/"
  content = ""
}

# ---------- DynamoDB ----------
resource "aws_dynamodb_table" "orders" {
  name             = "orders"
  billing_mode     = "PAY_PER_REQUEST"
  hash_key         = "order_id"
  stream_enabled   = true
  stream_view_type = "NEW_AND_OLD_IMAGES"

  attribute {
    name = "order_id"
    type = "S"
  }
}

# ---------- IAM for Lambda ----------
resource "aws_iam_role" "lambda_role" {
  name = "data-lake-lambda-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = {
        Service = "lambda.amazonaws.com"
      }
    }]
  })
}

resource "aws_iam_role_policy" "lambda_policy" {
  name = "data-lake-lambda-policy"
  role = aws_iam_role.lambda_role.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "dynamodb:GetRecords",
          "dynamodb:GetShardIterator",
          "dynamodb:DescribeStream",
          "dynamodb:ListStreams"
        ]
        Resource = aws_dynamodb_table.orders.stream_arn
      },
      {
        Effect = "Allow"
        Action = [
          "s3:PutObject"
        ]
        Resource = "${aws_s3_bucket.data_lake.arn}/raw/*"
      },
      {
        Effect = "Allow"
        Action = [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents"
        ]
        Resource = "arn:aws:logs:*:*:*"
      }
    ]
  })
}

# ---------- Lambda Function ----------
data "archive_file" "lambda_zip" {
  type        = "zip"
  source_file = "${path.module}/lambda/stream_to_s3.py"
  output_path = "${path.module}/lambda/stream_to_s3.zip"
}

resource "aws_lambda_function" "stream_to_s3" {
  filename         = data.archive_file.lambda_zip.output_path
  function_name    = "stream-to-s3"
  role             = aws_iam_role.lambda_role.arn
  handler          = "stream_to_s3.handler"
  runtime          = "python3.12"
  source_code_hash = data.archive_file.lambda_zip.output_base64sha256
  timeout          = 60

  environment {
    variables = {
      BUCKET_NAME = aws_s3_bucket.data_lake.id
      PREFIX      = "raw/"
    }
  }
}

resource "aws_lambda_event_source_mapping" "dynamodb_stream" {
  event_source_arn  = aws_dynamodb_table.orders.stream_arn
  function_name     = aws_lambda_function.stream_to_s3.arn
  starting_position = "LATEST"
  batch_size        = 10
}

# ---------- Glue Data Catalog ----------
resource "aws_glue_catalog_database" "data_lake_db" {
  name = "data_lake_db"
}

resource "aws_iam_role" "glue_role" {
  name = "data-lake-glue-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = {
        Service = "glue.amazonaws.com"
      }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "glue_service" {
  role       = aws_iam_role.glue_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole"
}

resource "aws_iam_role_policy" "glue_s3_policy" {
  name = "glue-s3-access"
  role = aws_iam_role.glue_role.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Action = [
        "s3:GetObject",
        "s3:ListBucket"
      ]
      Resource = [
        aws_s3_bucket.data_lake.arn,
        "${aws_s3_bucket.data_lake.arn}/*"
      ]
    }]
  })
}

resource "aws_glue_crawler" "raw_crawler" {
  database_name = aws_glue_catalog_database.data_lake_db.name
  name          = "raw-zone-crawler"
  role          = aws_iam_role.glue_role.arn

  s3_target {
    path = "s3://${aws_s3_bucket.data_lake.id}/raw/"
  }
}

And here is the complete outputs.tf:

output "s3_bucket_name" {
  description = "Name of the data lake S3 bucket"
  value       = aws_s3_bucket.data_lake.id
}

output "dynamodb_table_name" {
  description = "Name of the DynamoDB orders table"
  value       = aws_dynamodb_table.orders.name
}

output "lambda_function_name" {
  description = "Name of the stream-to-S3 Lambda function"
  value       = aws_lambda_function.stream_to_s3.function_name
}

output "glue_database_name" {
  description = "Glue Catalog database name"
  value       = aws_glue_catalog_database.data_lake_db.name
}

Your data lake now has automatic schema discovery. Next up, you will set up Redshift Serverless to query this cataloged data at warehouse scale.

Set Up Redshift Serverless and Load Data

Your data lake now has raw JSON files flowing in from DynamoDB, and the Glue Crawler has cataloged their schema. The next piece of the puzzle is a data warehouse that can run fast analytical queries across all that data.

In this step, you will provision Redshift Serverless as the analytics engine. Redshift Serverless auto-scales compute and pauses when idle, so you only pay for queries you actually run. You will create the warehouse with Terraform, then use the Redshift Query Editor v2 to create a table and load data from S3.

In this step, get ready to:
  • Create an IAM role for Redshift with S3 read access.
  • Provision a Redshift Serverless namespace and workgroup.
  • Load data from S3 into Redshift using the COPY command.
Create the Redshift IAM role and S3 policy

Redshift needs permission to read objects from your S3 data lake bucket. You will create a dedicated IAM role that Redshift assumes, with an inline policy granting s3:GetObject and s3:ListBucket on the data lake bucket.

  • In main.tf, add the following Redshift IAM role and policy below the Glue Crawler block:
# ---------- Redshift Serverless ----------
resource "aws_iam_role" "redshift_role" {
  name = "data-lake-redshift-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = {
        Service = "redshift.amazonaws.com"
      }
    }]
  })
}

resource "aws_iam_role_policy" "redshift_s3_policy" {
  name = "redshift-s3-read"
  role = aws_iam_role.redshift_role.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Action = [
        "s3:GetObject",
        "s3:ListBucket"
      ]
      Resource = [
        aws_s3_bucket.data_lake.arn,
        "${aws_s3_bucket.data_lake.arn}/*"
      ]
    }]
  })
}

What does this code do?

The aws_iam_role resource creates a role that only the Redshift service can assume (via the redshift.amazonaws.com trust policy).

The inline policy grants read-only access to your S3 data lake bucket. Redshift will use this role when running COPY commands to pull data from S3.

Provision the Redshift Serverless namespace and workgroup

Redshift Serverless organizes resources into two layers: a namespace (the database and admin credentials) and a workgroup (the compute that runs queries).

  • Below the Redshift IAM policy you just added, add the namespace and workgroup resources:
resource "aws_redshiftserverless_namespace" "analytics" {
  namespace_name   = "analytics-ns"
  admin_username   = "admin"
  admin_user_password = var.redshift_admin_password
  db_name          = "analyticsdb"
  iam_roles        = [aws_iam_role.redshift_role.arn]
  default_iam_role_arn = aws_iam_role.redshift_role.arn
}

resource "aws_redshiftserverless_workgroup" "analytics" {
  namespace_name = aws_redshiftserverless_namespace.analytics.namespace_name
  workgroup_name = "analytics-wg"
  base_capacity  = 4

  publicly_accessible = true
}

What does this code do?

The namespace analytics-ns creates a database called analyticsdb with an admin user. The password comes from a Terraform variable marked as sensitive, so it never appears in plain text in your state file.

The workgroup analytics-wg uses 4 RPU (Redshift Processing Units) as its base capacity. This is the minimum supported capacity. Setting publicly_accessible = true allows you to connect from the Query Editor in the AWS console.

What is Redshift Serverless?

Redshift Serverless is a data warehouse that scales compute automatically and charges per-second with no charge when idle. It starts at $1.50/hour when active and includes a $300 free trial credit valid for 90 days for new accounts.

  • Now add the Terraform variable that holds the Redshift admin password. Add this block at the end of main.tf:
# ---------- Variables ----------
variable "redshift_admin_password" {
  description = "Admin password for Redshift Serverless namespace"
  type        = string
  sensitive   = true
}

Why a sensitive variable?

Marking the variable as sensitive = true tells Terraform to redact its value from CLI output and plan logs. Terraform will prompt you for the password at apply time.

  • Next, add a Redshift endpoint output to outputs.tf. Add this block at the bottom of the file:
output "redshift_workgroup_endpoint" {
  description = "Redshift Serverless workgroup endpoint"
  value       = aws_redshiftserverless_workgroup.analytics.endpoint
}
  • Save both main.tf and outputs.tf.
  • Apply your Terraform configuration by running:
terraform apply
  • When prompted for var.redshift_admin_password, enter a strong password (at least 8 characters, with uppercase, lowercase, and a number).
  • Type yes to confirm the apply.

How long does this take?

Provisioning the Redshift Serverless namespace and workgroup typically takes 3-5 minutes. Wait for the apply to complete before continuing.

  • Confirm the output shows Apply complete! with the new resources created.

Seeing a VPC subnet error?

Redshift Serverless requires at least 3 subnets across 3 different Availability Zones. If your default VPC does not meet this requirement, check that your AWS region has a default VPC with subnets in at least 3 AZs.

help me fix this Redshift Serverless VPC subnet error

✔️ Awesome, I've got everything!

Great. Double check you have saved both files before continuing.

ⓧ I'd like to double check the full code

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
  required_version = ">= 1.5.0"
}

provider "aws" {
  region = "us-east-1"
}

# ---------- S3 Data Lake ----------
resource "aws_s3_bucket" "data_lake" {
  bucket_prefix = "data-lake-pipeline-"
  force_destroy = true
}

resource "aws_s3_object" "raw_prefix" {
  bucket  = aws_s3_bucket.data_lake.id
  key     = "raw/"
  content = ""
}

resource "aws_s3_object" "processed_prefix" {
  bucket  = aws_s3_bucket.data_lake.id
  key     = "processed/"
  content = ""
}

# ---------- DynamoDB ----------
resource "aws_dynamodb_table" "orders" {
  name             = "orders"
  billing_mode     = "PAY_PER_REQUEST"
  hash_key         = "order_id"
  stream_enabled   = true
  stream_view_type = "NEW_AND_OLD_IMAGES"

  attribute {
    name = "order_id"
    type = "S"
  }
}

# ---------- IAM for Lambda ----------
resource "aws_iam_role" "lambda_role" {
  name = "data-lake-lambda-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = {
        Service = "lambda.amazonaws.com"
      }
    }]
  })
}

resource "aws_iam_role_policy" "lambda_policy" {
  name = "data-lake-lambda-policy"
  role = aws_iam_role.lambda_role.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "dynamodb:GetRecords",
          "dynamodb:GetShardIterator",
          "dynamodb:DescribeStream",
          "dynamodb:ListStreams"
        ]
        Resource = aws_dynamodb_table.orders.stream_arn
      },
      {
        Effect = "Allow"
        Action = [
          "s3:PutObject"
        ]
        Resource = "${aws_s3_bucket.data_lake.arn}/raw/*"
      },
      {
        Effect = "Allow"
        Action = [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents"
        ]
        Resource = "arn:aws:logs:*:*:*"
      }
    ]
  })
}

# ---------- Lambda Function ----------
data "archive_file" "lambda_zip" {
  type        = "zip"
  source_file = "${path.module}/lambda/stream_to_s3.py"
  output_path = "${path.module}/lambda/stream_to_s3.zip"
}

resource "aws_lambda_function" "stream_to_s3" {
  filename         = data.archive_file.lambda_zip.output_path
  function_name    = "stream-to-s3"
  role             = aws_iam_role.lambda_role.arn
  handler          = "stream_to_s3.handler"
  runtime          = "python3.12"
  source_code_hash = data.archive_file.lambda_zip.output_base64sha256
  timeout          = 60

  environment {
    variables = {
      BUCKET_NAME = aws_s3_bucket.data_lake.id
      PREFIX      = "raw/"
    }
  }
}

resource "aws_lambda_event_source_mapping" "dynamodb_stream" {
  event_source_arn  = aws_dynamodb_table.orders.stream_arn
  function_name     = aws_lambda_function.stream_to_s3.arn
  starting_position = "LATEST"
  batch_size        = 10
}

# ---------- Glue Data Catalog ----------
resource "aws_glue_catalog_database" "data_lake_db" {
  name = "data_lake_db"
}

resource "aws_iam_role" "glue_role" {
  name = "data-lake-glue-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = {
        Service = "glue.amazonaws.com"
      }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "glue_service" {
  role       = aws_iam_role.glue_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole"
}

resource "aws_iam_role_policy" "glue_s3_policy" {
  name = "glue-s3-access"
  role = aws_iam_role.glue_role.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Action = [
        "s3:GetObject",
        "s3:ListBucket"
      ]
      Resource = [
        aws_s3_bucket.data_lake.arn,
        "${aws_s3_bucket.data_lake.arn}/*"
      ]
    }]
  })
}

resource "aws_glue_crawler" "raw_crawler" {
  database_name = aws_glue_catalog_database.data_lake_db.name
  name          = "raw-zone-crawler"
  role          = aws_iam_role.glue_role.arn

  s3_target {
    path = "s3://${aws_s3_bucket.data_lake.id}/raw/"
  }
}

# ---------- Redshift Serverless ----------
resource "aws_redshiftserverless_namespace" "analytics" {
  namespace_name   = "analytics-ns"
  admin_username   = "admin"
  admin_user_password = var.redshift_admin_password
  db_name          = "analyticsdb"
  iam_roles        = [aws_iam_role.redshift_role.arn]
  default_iam_role_arn = aws_iam_role.redshift_role.arn
}

resource "aws_redshiftserverless_workgroup" "analytics" {
  namespace_name = aws_redshiftserverless_namespace.analytics.namespace_name
  workgroup_name = "analytics-wg"
  base_capacity  = 4

  publicly_accessible = true
}

resource "aws_iam_role" "redshift_role" {
  name = "data-lake-redshift-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = {
        Service = "redshift.amazonaws.com"
      }
    }]
  })
}

resource "aws_iam_role_policy" "redshift_s3_policy" {
  name = "redshift-s3-read"
  role = aws_iam_role.redshift_role.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Action = [
        "s3:GetObject",
        "s3:ListBucket"
      ]
      Resource = [
        aws_s3_bucket.data_lake.arn,
        "${aws_s3_bucket.data_lake.arn}/*"
      ]
    }]
  })
}

# ---------- Variables ----------
variable "redshift_admin_password" {
  description = "Admin password for Redshift Serverless namespace"
  type        = string
  sensitive   = true
}
output "s3_bucket_name" {
  description = "Name of the data lake S3 bucket"
  value       = aws_s3_bucket.data_lake.id
}

output "dynamodb_table_name" {
  description = "Name of the DynamoDB orders table"
  value       = aws_dynamodb_table.orders.name
}

output "lambda_function_name" {
  description = "Name of the stream-to-S3 Lambda function"
  value       = aws_lambda_function.stream_to_s3.function_name
}

output "glue_database_name" {
  description = "Glue Catalog database name"
  value       = aws_glue_catalog_database.data_lake_db.name
}

output "redshift_workgroup_endpoint" {
  description = "Redshift Serverless workgroup endpoint"
  value       = aws_redshiftserverless_workgroup.analytics.endpoint
}
Load data into Redshift using the Query Editor

Now that the warehouse is running, you will connect to it and create a table to hold your order data. Then you will use the Redshift COPY command to bulk-load JSON data directly from S3.

  • Open the Amazon Redshift console at https://console.aws.amazon.com/redshiftv2/.
  • From the left navigation menu, choose **Editor**, then **Query editor V2**. A new browser tab opens.
  • In the tree-view panel on the left, click the workgroup name analytics-wg.
  • When prompted for authentication, select **Database user name and password**.
  • Enter admin as the user name and the password you provided during terraform apply.
  • For the database name, enter analyticsdb.
  • Click **Create connection**.

You are now connected to the analyticsdb database in your Redshift Serverless workgroup.

  • Create a schema and orders table by pasting the following SQL into the query editor and clicking **Run**:
CREATE SCHEMA IF NOT EXISTS pipeline;

CREATE TABLE pipeline.orders (
  order_id VARCHAR(50),
  customer_name VARCHAR(100),
  product VARCHAR(200),
  quantity DECIMAL(10,2),
  status VARCHAR(20),
  order_date VARCHAR(30),
  _stream_timestamp VARCHAR(30),
  _event_name VARCHAR(10)
);

What does this SQL do?

It creates a pipeline schema to keep your analytics tables organized. The pipeline.orders table mirrors the JSON fields that Lambda writes to S3, plus the metadata fields _stream_timestamp and _event_name.

  • Confirm you see a success message indicating the table was created.
  • Now load data from S3 into the table. Replace the bucket name placeholder with your actual S3 bucket name (you can find it by running terraform output s3_bucket_name in your terminal). Replace the IAM role ARN placeholder with your Redshift role ARN (found by running terraform output or checking the IAM console). Paste and run the following COPY command:
COPY pipeline.orders
FROM 's3://[[DATALAKE_BUCKET="your-data-lake-bucket-name"]]/raw/'
IAM_ROLE '[[REDSHIFT_ROLE_ARN="arn:aws:iam::123456789012:role/data-lake-redshift-role"]]'
FORMAT AS JSON 'auto'
REGION 'us-east-1';

What does the COPY command do?

COPY is Redshift's bulk-load command. It reads all JSON files under the raw/ prefix in your S3 bucket and maps JSON keys to table columns automatically (FORMAT AS JSON 'auto').

The IAM_ROLE parameter tells Redshift which role to assume when reading from S3. This is the data-lake-redshift-role you provisioned earlier.

  • Verify the data loaded successfully by running this query:
SELECT * FROM pipeline.orders LIMIT 10;
  • You should see rows of order data in the results panel. This confirms end-to-end data flow from DynamoDB through S3 into Redshift.

COPY command returned 0 rows?

If no rows loaded, make sure you have inserted test items into DynamoDB in a previous step so that the Lambda function has written JSON files to the raw/ prefix in S3.

Check the S3 console to confirm .json files exist under the raw/ folder. If not, insert a test record into DynamoDB, wait 30 seconds for the stream to trigger Lambda, then retry the COPY.

help me debug why no data loaded into Redshift from S3

Your data warehouse is live and holding data from the pipeline. Next up, you will set up CloudWatch monitoring to keep an eye on the health of your Lambda function and Glue Crawler.

Monitor with CloudWatch

Your pipeline now moves data from DynamoDB all the way through to Redshift. But how do you know if something breaks at 2 AM?

In this step, you will add observability to your pipeline using CloudWatch. You will create a dashboard that displays Lambda invocation and error metrics alongside Glue Crawler activity, plus an alarm that fires whenever Lambda encounters an error.

In this step, get ready to:
  • Create a CloudWatch dashboard with Lambda and Glue Crawler widgets.
  • Create a CloudWatch alarm for Lambda function errors.
  • Verify metrics appear after inserting test data into DynamoDB.
Create the CloudWatch dashboard

A CloudWatch dashboard gives you a single pane of glass to monitor multiple AWS services at once. You will define two widgets: one tracking your Lambda function's invocations and errors, and another tracking how many records your Glue Crawler processes.

  • In main.tf, scroll to the bottom of the file (below the # ---------- Variables ---------- section).
  • Add the following CloudWatch section above the Variables section by pasting this code:
# ---------- CloudWatch ----------
resource "aws_cloudwatch_dashboard" "pipeline_dashboard" {
  dashboard_name = "DataLakePipeline"

  dashboard_body = jsonencode({
    widgets = [
      {
        type   = "metric"
        x      = 0
        y      = 0
        width  = 12
        height = 6
        properties = {
          metrics = [
            ["AWS/Lambda", "Invocations", "FunctionName", aws_lambda_function.stream_to_s3.function_name],
            ["AWS/Lambda", "Errors", "FunctionName", aws_lambda_function.stream_to_s3.function_name]
          ]
          period = 300
          stat   = "Sum"
          region = "us-east-1"
          title  = "Lambda Stream-to-S3 Metrics"
        }
      },
      {
        type   = "metric"
        x      = 12
        y      = 0
        width  = 12
        height = 6
        properties = {
          metrics = [
            ["AWS/Glue", "glue.driver.aggregate.recordsRead", "JobName", "raw-zone-crawler", "JobRunId", "ALL", "Type", "count"]
          ]
          period = 300
          stat   = "Sum"
          region = "us-east-1"
          title  = "Glue Crawler Records"
        }
      }
    ]
  })
}

What does this code do?

The aws_cloudwatch_dashboard resource creates a dashboard named DataLakePipeline in the CloudWatch console.

The dashboard_body is a JSON definition of widget layouts. The first widget (left half, 12 columns wide) plots Lambda Invocations and Errors for your stream-to-s3 function. The second widget (right half) plots how many records your Glue Crawler reads.

Both use a 300-second (5-minute) period and Sum statistic, which matches CloudWatch's default aggregation window for Lambda metrics.

  • Save main.tf.

Seeing a JSON encoding error?

Make sure you have matching opening and closing braces inside the jsonencode() block. A missing comma between widget objects or an extra trailing comma after the last widget will cause a parse failure.

help me fix a JSON encoding error in my Terraform CloudWatch dashboard definition

Create the CloudWatch alarm

A dashboard shows you what happened. An alarm tells you when something goes wrong in real time. You will create a metric alarm that triggers whenever Lambda reports any errors within a 5-minute window.

  • In main.tf, add the following alarm resource directly below the dashboard resource you just added:
resource "aws_cloudwatch_metric_alarm" "lambda_errors" {
  alarm_name          = "stream-to-s3-errors"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 1
  metric_name         = "Errors"
  namespace           = "AWS/Lambda"
  period              = 300
  statistic           = "Sum"
  threshold           = 0
  alarm_description   = "Alarm when Lambda stream-to-s3 has errors"

  dimensions = {
    FunctionName = aws_lambda_function.stream_to_s3.function_name
  }
}

What does this alarm do?

This alarm watches the Errors metric in the AWS/Lambda namespace, scoped to the stream-to-s3 function.

If the sum of errors in any single 5-minute period (period = 300) is greater than 0, the alarm transitions to the ALARM state. In production, you would attach an SNS topic to send email or Slack notifications.

  • Save main.tf.
  • Now add an output so Terraform prints the dashboard URL after applying. Open outputs.tf and add this block at the bottom:
output "cloudwatch_dashboard_url" {
  description = "URL to the CloudWatch dashboard"
  value       = "https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#dashboards:name=${aws_cloudwatch_dashboard.pipeline_dashboard.dashboard_name}"
}
  • Save outputs.tf.
  • Apply the changes by running:
terraform apply -var="redshift_admin_password=your-redshift-password-here"
  • When prompted to confirm, type yes and press Enter.

Terraform will create 2 new resources: the CloudWatch dashboard and the metric alarm. You should see the cloudwatch_dashboard_url output printed at the end.

Apply failed with a reference error?

Make sure the # ---------- CloudWatch ---------- section is placed above the # ---------- Variables ---------- section in main.tf. The variable block must remain at the bottom for clean file organization.

help me fix a Terraform apply error for my CloudWatch resources

Verify metrics by inserting test data

Your dashboard and alarm are deployed, but they need data flowing through the pipeline to display anything useful. Inserting a record into DynamoDB will trigger the stream, invoke Lambda, and generate CloudWatch metrics you can observe.

  • Insert a test order into your DynamoDB table by running:
aws dynamodb put-item \
    --table-name orders \
    --item '{"order_id": {"S": "order-101"}, "customer": {"S": "Jane Doe"}, "amount": {"N": "49.99"}, "status": {"S": "pending"}}'

What does this command do?

This inserts a single item into the orders table with four attributes: order_id (partition key), customer, amount, and status. Because DynamoDB Streams is enabled, this INSERT event triggers your Lambda function automatically.

  • Insert a second test order to generate more data:
aws dynamodb put-item \
    --table-name orders \
    --item '{"order_id": {"S": "order-102"}, "customer": {"S": "John Smith"}, "amount": {"N": "129.99"}, "status": {"S": "shipped"}}'
  • Wait 3-5 minutes for CloudWatch metrics to populate.
  • Open the CloudWatch dashboard URL that Terraform printed, or navigate to the CloudWatch console in your browser.
  • Click Dashboards in the left sidebar.
  • Click DataLakePipeline.

You should see the Lambda Stream-to-S3 Metrics widget showing 2 invocations and 0 errors. The Glue Crawler Records widget will be empty until you run the crawler.

Why is the Glue Crawler widget empty?

The Glue Crawler only emits metrics when it runs. Since your crawler is configured for on-demand execution, you will see data in that widget after you manually start the crawler (or schedule it). The Lambda widget, on the other hand, populates automatically because the DynamoDB Stream triggers invocations in real time.

  • Verify the alarm state by clicking Alarms in the left sidebar, then All alarms.
  • Confirm the stream-to-s3-errors alarm shows OK state (green). This means Lambda processed your test records without errors.

Dashboard showing no data points?

CloudWatch metrics can take up to 5 minutes to appear after a Lambda invocation. If you still see no data after 5 minutes, check the Lambda function logs in CloudWatch Logs to confirm the function was invoked by the stream.

Also verify that the DynamoDB table name in your put-item command matches the table Terraform created (orders).

help me troubleshoot why my CloudWatch dashboard is not showing Lambda metrics

✔️ Awesome, I've got everything!

Great. Double check you have saved both main.tf and outputs.tf.

ⓧ I'd like to double check the full code

Here is the full main.tf at this point:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
  required_version = ">= 1.5.0"
}

provider "aws" {
  region = "us-east-1"
}

# ---------- S3 Data Lake ----------
resource "aws_s3_bucket" "data_lake" {
  bucket_prefix = "data-lake-pipeline-"
  force_destroy = true
}

resource "aws_s3_object" "raw_prefix" {
  bucket  = aws_s3_bucket.data_lake.id
  key     = "raw/"
  content = ""
}

resource "aws_s3_object" "processed_prefix" {
  bucket  = aws_s3_bucket.data_lake.id
  key     = "processed/"
  content = ""
}

# ---------- DynamoDB ----------
resource "aws_dynamodb_table" "orders" {
  name             = "orders"
  billing_mode     = "PAY_PER_REQUEST"
  hash_key         = "order_id"
  stream_enabled   = true
  stream_view_type = "NEW_AND_OLD_IMAGES"

  attribute {
    name = "order_id"
    type = "S"
  }
}

# ---------- IAM for Lambda ----------
resource "aws_iam_role" "lambda_role" {
  name = "data-lake-lambda-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = {
        Service = "lambda.amazonaws.com"
      }
    }]
  })
}

resource "aws_iam_role_policy" "lambda_policy" {
  name = "data-lake-lambda-policy"
  role = aws_iam_role.lambda_role.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "dynamodb:GetRecords",
          "dynamodb:GetShardIterator",
          "dynamodb:DescribeStream",
          "dynamodb:ListStreams"
        ]
        Resource = aws_dynamodb_table.orders.stream_arn
      },
      {
        Effect = "Allow"
        Action = [
          "s3:PutObject"
        ]
        Resource = "${aws_s3_bucket.data_lake.arn}/raw/*"
      },
      {
        Effect = "Allow"
        Action = [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents"
        ]
        Resource = "arn:aws:logs:*:*:*"
      }
    ]
  })
}

# ---------- Lambda Function ----------
data "archive_file" "lambda_zip" {
  type        = "zip"
  source_file = "${path.module}/lambda/stream_to_s3.py"
  output_path = "${path.module}/lambda/stream_to_s3.zip"
}

resource "aws_lambda_function" "stream_to_s3" {
  filename         = data.archive_file.lambda_zip.output_path
  function_name    = "stream-to-s3"
  role             = aws_iam_role.lambda_role.arn
  handler          = "stream_to_s3.handler"
  runtime          = "python3.12"
  source_code_hash = data.archive_file.lambda_zip.output_base64sha256
  timeout          = 60

  environment {
    variables = {
      BUCKET_NAME = aws_s3_bucket.data_lake.id
      PREFIX      = "raw/"
    }
  }
}

resource "aws_lambda_event_source_mapping" "dynamodb_stream" {
  event_source_arn  = aws_dynamodb_table.orders.stream_arn
  function_name     = aws_lambda_function.stream_to_s3.arn
  starting_position = "LATEST"
  batch_size        = 10
}

# ---------- Glue Data Catalog ----------
resource "aws_glue_catalog_database" "data_lake_db" {
  name = "data_lake_db"
}

resource "aws_iam_role" "glue_role" {
  name = "data-lake-glue-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = {
        Service = "glue.amazonaws.com"
      }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "glue_service" {
  role       = aws_iam_role.glue_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole"
}

resource "aws_iam_role_policy" "glue_s3_policy" {
  name = "glue-s3-access"
  role = aws_iam_role.glue_role.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Action = [
        "s3:GetObject",
        "s3:ListBucket"
      ]
      Resource = [
        aws_s3_bucket.data_lake.arn,
        "${aws_s3_bucket.data_lake.arn}/*"
      ]
    }]
  })
}

resource "aws_glue_crawler" "raw_crawler" {
  database_name = aws_glue_catalog_database.data_lake_db.name
  name          = "raw-zone-crawler"
  role          = aws_iam_role.glue_role.arn

  s3_target {
    path = "s3://${aws_s3_bucket.data_lake.id}/raw/"
  }
}

# ---------- Redshift Serverless ----------
resource "aws_redshiftserverless_namespace" "analytics" {
  namespace_name   = "analytics-ns"
  admin_username   = "admin"
  admin_user_password = var.redshift_admin_password
  db_name          = "analyticsdb"
  iam_roles        = [aws_iam_role.redshift_role.arn]
  default_iam_role_arn = aws_iam_role.redshift_role.arn
}

resource "aws_redshiftserverless_workgroup" "analytics" {
  namespace_name = aws_redshiftserverless_namespace.analytics.namespace_name
  workgroup_name = "analytics-wg"
  base_capacity  = 4

  publicly_accessible = true
}

resource "aws_iam_role" "redshift_role" {
  name = "data-lake-redshift-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = {
        Service = "redshift.amazonaws.com"
      }
    }]
  })
}

resource "aws_iam_role_policy" "redshift_s3_policy" {
  name = "redshift-s3-read"
  role = aws_iam_role.redshift_role.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Action = [
        "s3:GetObject",
        "s3:ListBucket"
      ]
      Resource = [
        aws_s3_bucket.data_lake.arn,
        "${aws_s3_bucket.data_lake.arn}/*"
      ]
    }]
  })
}

# ---------- CloudWatch ----------
resource "aws_cloudwatch_dashboard" "pipeline_dashboard" {
  dashboard_name = "DataLakePipeline"

  dashboard_body = jsonencode({
    widgets = [
      {
        type   = "metric"
        x      = 0
        y      = 0
        width  = 12
        height = 6
        properties = {
          metrics = [
            ["AWS/Lambda", "Invocations", "FunctionName", aws_lambda_function.stream_to_s3.function_name],
            ["AWS/Lambda", "Errors", "FunctionName", aws_lambda_function.stream_to_s3.function_name]
          ]
          period = 300
          stat   = "Sum"
          region = "us-east-1"
          title  = "Lambda Stream-to-S3 Metrics"
        }
      },
      {
        type   = "metric"
        x      = 12
        y      = 0
        width  = 12
        height = 6
        properties = {
          metrics = [
            ["AWS/Glue", "glue.driver.aggregate.recordsRead", "JobName", "raw-zone-crawler", "JobRunId", "ALL", "Type", "count"]
          ]
          period = 300
          stat   = "Sum"
          region = "us-east-1"
          title  = "Glue Crawler Records"
        }
      }
    ]
  })
}

resource "aws_cloudwatch_metric_alarm" "lambda_errors" {
  alarm_name          = "stream-to-s3-errors"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 1
  metric_name         = "Errors"
  namespace           = "AWS/Lambda"
  period              = 300
  statistic           = "Sum"
  threshold           = 0
  alarm_description   = "Alarm when Lambda stream-to-s3 has errors"

  dimensions = {
    FunctionName = aws_lambda_function.stream_to_s3.function_name
  }
}

# ---------- Variables ----------
variable "redshift_admin_password" {
  description = "Admin password for Redshift Serverless namespace"
  type        = string
  sensitive   = true
}

Here is the full outputs.tf at this point:

output "s3_bucket_name" {
  description = "Name of the data lake S3 bucket"
  value       = aws_s3_bucket.data_lake.id
}

output "dynamodb_table_name" {
  description = "Name of the DynamoDB orders table"
  value       = aws_dynamodb_table.orders.name
}

output "lambda_function_name" {
  description = "Name of the stream-to-S3 Lambda function"
  value       = aws_lambda_function.stream_to_s3.function_name
}

output "glue_database_name" {
  description = "Glue Catalog database name"
  value       = aws_glue_catalog_database.data_lake_db.name
}

output "redshift_workgroup_endpoint" {
  description = "Redshift Serverless workgroup endpoint"
  value       = aws_redshiftserverless_workgroup.analytics.endpoint
}

output "cloudwatch_dashboard_url" {
  description = "URL to the CloudWatch dashboard"
  value       = "https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#dashboards:name=${aws_cloudwatch_dashboard.pipeline_dashboard.dashboard_name}"
}

Your pipeline now has full observability. Next up, you will connect QuickSight to Redshift and build a visualization that turns your warehouse data into business insights.

Connect QuickSight to Redshift

Your data pipeline is running end to end. Orders flow from DynamoDB through Lambda into the S3 data lake, get cataloged by Glue, and land in Redshift Serverless for analytics queries.

The final piece is making that warehouse data visible to business users through a dashboard. QuickSight is AWS's managed BI service that connects directly to Redshift and turns query results into interactive visualizations.

In this step, get ready to:
  • Sign up for QuickSight Standard Edition.
  • Create a QuickSight data source connected to your Redshift Serverless workgroup.
  • Build a bar chart visualization showing orders by status.
Sign up for QuickSight

QuickSight requires a separate subscription before you can use it. The Standard Edition costs $9/month and provides everything you need to connect to Redshift and build visualizations.

Cost warning

QuickSight Standard Edition charges $9/user/month. Make sure to cancel your subscription in the Before You Go section when you finish this project to avoid ongoing charges.

  • In the AWS Management Console search bar, type QuickSight and select the service from the results.
  • Click Sign up for QuickSight.
  • Select Standard as the edition.
  • Enter a QuickSight account name (any unique name works, such as data-lake-pipeline).
  • Enter your email address in the Notification email address field.
  • Under QuickSight access to AWS services, make sure Amazon Redshift and Amazon S3 are checked. These permissions allow QuickSight to read data from your Redshift workgroup and S3 bucket.
  • Click Finish.
  • Wait for the account to be created, then click Go to Amazon QuickSight.

What is QuickSight?

QuickSight is a serverless BI (Business Intelligence) service. It connects to data sources like Redshift, Athena, and S3, then lets you build charts, graphs, and dashboards without managing any visualization infrastructure.

In a production data pipeline, QuickSight is the layer where stakeholders actually see the data and make decisions from it.

Already have a QuickSight account?

If your AWS account already has QuickSight set up, skip the sign-up steps. Instead, navigate to Manage QuickSight from your profile icon, then check that Redshift and S3 permissions are enabled under Security & Permissions.

Help me configure QuickSight permissions for Redshift access.

Create a data source pointing to Redshift Serverless

A data source in QuickSight is a connection profile that stores the endpoint, database name, and credentials needed to reach your warehouse.

  • In the QuickSight console, click Datasets in the left navigation pane.
  • Click New dataset.
  • Select Redshift Auto-discovered as the data source type. QuickSight auto-discovers Redshift Serverless workgroups in the same region.
  • For Data source name, enter data-lake-redshift.
  • For Connection type, select Public network.
  • Select your workgroup analytics-wg from the dropdown.
  • For Database name, enter analyticsdb.
  • For Username, enter admin.
  • For Password, enter the Redshift admin password you set in your Terraform variable.
  • Click Validate connection. Wait for the green confirmation message.
  • Click Create data source.

Why Auto-discovered instead of Manual connect?

QuickSight can auto-discover Redshift Serverless workgroups that are publicly accessible in the same AWS region. Since your analytics-wg workgroup has publicly_accessible = true in the Terraform configuration, QuickSight finds it automatically. This saves you from manually entering the endpoint URL.

Connection validation failed?

Make sure your Redshift Serverless workgroup is set to publicly accessible in your Terraform config. Also verify the password matches the value you passed to the redshift_admin_password variable during terraform apply.

If you see a timeout error, check that the Redshift workgroup security group allows inbound traffic from QuickSight IP ranges on port 5439.

Help me troubleshoot my QuickSight to Redshift connection.

Build a bar chart of orders by status

With the data source connected, you can now create a dataset and build a visualization that proves end-to-end data flow through your pipeline.

  • After creating the data source, QuickSight shows a Choose your table dialog. Select the schema public and choose the orders table.
  • Click Select.
  • Choose Directly query your data (this queries Redshift live instead of importing a snapshot).
  • Click Visualize to create a new analysis.
  • In the Fields list on the left, drag order_status to the X axis well.
  • Drag order_id to the Value well. QuickSight automatically aggregates it as a count.
  • In the Visual types panel, select the vertical bar chart icon.

You should see a bar chart with bars for each order status value (like pending, shipped, delivered), each showing the count of orders with that status.

What does this chart prove?

This visualization confirms that your entire pipeline is working. Data entered into DynamoDB was streamed by Lambda into S3, loaded into Redshift, and is now queryable through QuickSight. That is a complete journey from operational data store to BI layer.

  • Click the Share icon in the top-right corner of the analysis.
  • Select Publish dashboard.
  • Name the dashboard Orders Pipeline Dashboard and click Publish dashboard.

Your dashboard is now published. Anyone with QuickSight access in your account can view it.

Not seeing any data in the chart?

If the chart is empty, confirm that you loaded data into the orders table in Redshift during the previous step. Switch to the Redshift query editor and run a quick SELECT COUNT(*) FROM public.orders; to verify rows exist.

Also check that you selected the public schema and the correct orders table when creating the dataset.

Help me debug why my QuickSight chart is empty.

Your serverless data lake pipeline is complete. Data flows from DynamoDB through Lambda and S3, gets cataloged by Glue, warehoused in Redshift, monitored by CloudWatch, and visualized in QuickSight. Next up, a secret mission to add a Parquet transformation layer.

Secret mission

Add a Parquet Transformation Layer

Your pipeline writes raw JSON to S3, but production data lakes compress and column-orient their data for faster queries and lower storage costs. In this secret mission, you will add a second Lambda function that transforms raw JSON into Parquet format and update the Glue Crawler to catalog both zones automatically.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project provisions multiple AWS services that can incur charges if left running.

Cost warning

Redshift Serverless charges $1.50/hour when active (it auto-pauses when idle, but any query wakes it up). QuickSight charges $9/month for Standard Edition. If you are done with this project, destroy all resources immediately to avoid unexpected charges.

Resources you used:

  • S3 data lake bucket (with raw/ and processed/ prefixes).
  • DynamoDB table (orders).
  • Lambda functions (stream-to-s3, raw-to-parquet).
  • IAM roles (data-lake-lambda-role, data-lake-glue-role, data-lake-redshift-role).
  • Glue Catalog Database (data_lake_db) and Crawler (raw-zone-crawler).
  • Redshift Serverless namespace (analytics-ns) and workgroup (analytics-wg).
  • CloudWatch dashboard (DataLakePipeline) and alarm (stream-to-s3-errors).
  • QuickSight subscription (Standard Edition).
  • Local project files (main.tf, outputs.tf, lambda/).

Keep everything running

No action needed. Choose this if you are still actively building or experimenting with your pipeline.

  • Redshift Serverless auto-pauses when idle but will charge $1.50/hour each time a query activates it.
  • QuickSight will bill $9/month until you cancel.
  • DynamoDB, Lambda, S3, Glue, and CloudWatch remain within free-tier limits for light usage.
  • Terraform CLI and AWS CLI remain installed on your machine.

Pause - I'll come back to this later

Shut down the most expensive resources to stop charges while keeping your Terraform files for later.

  • Cancel your QuickSight subscription by navigating to https://us-east-1.quicksight.aws.amazon.com/sn/console/unsubscribe and following the account termination steps. This stops the $9/month charge.
  • Leave Redshift Serverless as-is. It auto-pauses when idle and incurs no charge until you run a query.
  • Keep your local project files intact so you can re-deploy with a single command later.

Delete - I don't want to use this again

Remove all project resources and start fresh. This is the recommended option once you have completed the project.

First, cancel your QuickSight subscription (not managed by Terraform):

  • Open https://us-east-1.quicksight.aws.amazon.com/sn/console/unsubscribe in your browser.
  • Toggle off Account termination protection is on.
  • Type the confirmation word shown on screen and click Delete account.

Next, destroy all Terraform-managed AWS resources by running this command from your project directory:

terraform destroy
  • Terraform will show you a plan of every resource it will delete. Type yes when prompted.
  • This removes the S3 bucket (force_destroy is enabled so it empties automatically), DynamoDB table, both Lambda functions, all IAM roles and policies, the Glue database and crawler, the Redshift Serverless namespace and workgroup, and the CloudWatch dashboard and alarm.

Finally, remove your local project files:

cd ~/Desktop
rm -rf data-lake-pipeline
  • This deletes main.tf, outputs.tf, the lambda/ directory, and all Terraform state files from your machine.

That's a Wrap!

That's a Wrap!

Nice work! You just built a full serverless data lake pipeline on AWS, entirely managed by Terraform.

You've learned how to:

  • Provision a DynamoDB Streams and Lambda pipeline that streams operational data into an S3 data lake with structured raw and processed zones.
  • Configure a Glue Crawler to automatically discover schemas and populate the Glue Data Catalog, then load data into Redshift Serverless for analytics queries.
  • Build a CloudWatch dashboard and alarms to monitor pipeline health, and connect QuickSight to Redshift for business intelligence visualization.
  • Secret Mission: Add a Parquet transformation layer with a second Lambda function and configure dual-zone Glue cataloging.

Ready to quiz yourself?