Build a Governed Data Lakehouse on AWS

Build an end-to-end data lakehouse with S3, Glue, Redshift, and DataZone.

Introduction

30 Second Summary

Every company collects mountains of data, but turning raw data into something decision-makers can actually use requires careful architecture. The difference between a data swamp and a data platform is structure, governance, and the right tooling.

In this project, you will build a complete data lakehouse on AWS that ingests raw retail data, transforms it into optimized formats, loads it into a star schema warehouse, and governs it all through a central catalog.

What You'll Build

A mini enterprise data lakehouse on AWS that demonstrates end-to-end data platform architecture, from raw CSV ingestion through ETL transformation to a governed, queryable star schema.

By the end of this project, you'll have:

  • A 3-zone S3 data lake (raw, curated, consumption) with automated schema discovery via AWS Glue crawlers and a populated Data Catalog.
  • A PySpark ETL job that transforms raw CSV into partitioned Parquet format, plus a star schema data warehouse running on Redshift Serverless.
  • Ad-hoc analytics powered by Amazon Athena querying data directly on the lake layer, demonstrating the lakehouse pattern.
  • A governed data catalog in Amazon DataZone with published assets, a business glossary, and data access controls.
  • Secret Mission: Add event-driven pipeline automation with a Lambda function that triggers your ETL job when new data lands in S3.

Are there any prerequisites?

You should have basic familiarity with navigating the AWS Console, SQL fundamentals, and basic Python knowledge. You also need an AWS account with a paid plan (free tier covers most services, and Redshift Serverless offers a separate $300 free trial for first-time users).

Set Up Your Data Lake Foundation

Every enterprise data platform starts with a foundation. For a data lakehouse, that foundation is a well-organized data lake on S3 with clear boundaries between raw, processed, and analytics-ready data.

In this step, you'll create the S3 bucket that stores all your lakehouse data, set up IAM permissions for AWS Glue, and upload sample retail data to get your pipeline ready.

In this step, get ready to:
  • Create an S3 bucket with three data lake zones (raw, curated, consumption).
  • Create an IAM role for AWS Glue with S3 access permissions.
  • Upload sample CSV files to the raw zone.
Create your S3 bucket with zone structure

Enterprise data lakes separate data by processing stage. Raw data lands untouched, curated data has been cleaned and optimized, and consumption data is shaped for specific business needs.

  • Open your terminal application by pressing Cmd+Space (macOS) or the Windows key (Windows) and typing Terminal.
  • Create your S3 bucket by running this command:
aws s3 mb s3://[[BUCKET_NAME="your-lakehouse-bucket-name"]]

Why does my bucket name need to be unique?

S3 bucket names are globally unique across all AWS accounts. Include your name or a random string to avoid conflicts. For example: ahmed-lakehouse-2024 or my-lakehouse-demo-abc123.

Keep the name lowercase, with no spaces or underscores. Only hyphens are allowed as separators.

  • Create the three zone prefixes inside your bucket by running these commands:
aws s3api put-object --bucket [[BUCKET_NAME="your-lakehouse-bucket-name"]] --key raw/
aws s3api put-object --bucket [[BUCKET_NAME="your-lakehouse-bucket-name"]] --key curated/
aws s3api put-object --bucket [[BUCKET_NAME="your-lakehouse-bucket-name"]] --key consumption/

What is a 3-zone data lake?

The three zones mirror how enterprises manage data at scale:

  • raw/ stores data exactly as it arrived from source systems. No transformations.
  • curated/ stores cleaned, validated, and optimized data (like Parquet format with partitions).
  • consumption/ stores data shaped for specific analytics or reporting use cases.

This separation lets teams work independently on different stages without corrupting upstream data.

  • Verify your bucket and zones exist by running:
aws s3 ls s3://[[BUCKET_NAME="your-lakehouse-bucket-name"]]/

You should see three prefixes listed: raw/, curated/, and consumption/.

Seeing an error when creating the bucket?

  • If you see BucketAlreadyExists, your bucket name is taken globally. Try adding more characters to make it unique.
  • If you see InvalidBucketName, check that your name uses only lowercase letters, numbers, and hyphens.
  • If you see IllegalLocationConstraintException, your AWS CLI may be configured for a region other than us-east-1. Add --region us-east-1 to the command or add --create-bucket-configuration LocationConstraint=your-region for other regions.

Still stuck? Help me troubleshoot my S3 bucket creation error.

Create the IAM role for Glue

AWS Glue needs permission to read and write data in your S3 bucket. You'll create a dedicated IAM role that Glue assumes when running crawlers and ETL jobs.

  • Open the IAM console at https://console.aws.amazon.com/iam/.
  • In the left navigation pane, click Roles.
  • Click Create role.
  • For Trusted entity type, select AWS service.
  • Under Service or use case, search for and select Glue.
  • Click Next.
  • On the Add permissions page, search for AWSGlueServiceRole in the search box.
  • Select the checkbox next to AWSGlueServiceRole.
  • Click Next.
  • For Role name, enter GlueLakehouseRole.
  • Click Create role.

Now you need to add an inline policy that grants Glue access to your specific S3 bucket.

  • In the Roles list, click on GlueLakehouseRole to open it.
  • Click the Permissions tab.
  • Click Add permissions, then select Create inline policy.
  • Click the JSON tab in the policy editor.
  • Delete the existing content and paste the following policy:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::[[BUCKET_NAME="your-lakehouse-bucket-name"]]/*"
        },
        {
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::[[BUCKET_NAME="your-lakehouse-bucket-name"]]"
        }
    ]
}

What does this policy do?

This inline policy grants Glue three specific permissions scoped to your bucket only:

  • s3:GetObject lets Glue read files from your bucket (needed by crawlers and ETL source reads).
  • s3:PutObject lets Glue write transformed data back (needed by ETL job outputs).
  • s3:DeleteObject lets Glue overwrite data during re-processing.
  • s3:ListBucket lets Glue list the contents of your bucket to discover files.

Scoping permissions to your specific bucket follows the principle of least privilege.

  • Click Next.
  • For Policy name, enter S3LakehouseAccess.
  • Click Create policy.

You should see both AWSGlueServiceRole and S3LakehouseAccess listed under the role's permissions.

Don't see the inline policy?

  • Make sure you clicked into the GlueLakehouseRole role before adding the inline policy. If you added it to a different role, delete it and try again.
  • If the JSON editor shows a validation error, check that your bucket name in the ARN matches the bucket you created earlier.

Need help? Help me fix my IAM inline policy for the Glue role.

Upload sample retail data to the raw zone

Your data lake needs data to work with. You'll create three CSV files representing a retail dataset and upload them to the raw zone.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create a project folder and move into it by running these commands:
mkdir lakehouse-data
cd lakehouse-data
  • Open this folder in VS Code by pressing Cmd+Space (macOS) or the Windows key (Windows), typing VS Code, and pressing Enter.
  • In VS Code, click File in the top menu bar.
  • Click Open Folder.
  • Navigate to Desktop and select the lakehouse-data folder, then click Open.
  • Create a new file by pressing Cmd+N (macOS) or Ctrl+N (Windows).
  • Paste the following content into the file:
order_id,order_date,customer_id,customer_name,customer_city,product_id,product_name,product_category,quantity,unit_price
1001,2024-01-15,C001,Ahmed Khan,Lahore,P001,Laptop Pro 15,Electronics,1,1200.00
1002,2024-01-16,C002,Sara Ali,Karachi,P002,Wireless Mouse,Electronics,3,25.00
1003,2024-01-17,C003,Usman Tariq,Islamabad,P003,Office Chair,Furniture,1,350.00
1004,2024-01-20,C001,Ahmed Khan,Lahore,P004,USB-C Hub,Electronics,2,45.00
1005,2024-02-01,C004,Fatima Noor,Lahore,P005,Standing Desk,Furniture,1,600.00
1006,2024-02-03,C002,Sara Ali,Karachi,P001,Laptop Pro 15,Electronics,1,1200.00
1007,2024-02-10,C005,Hassan Raza,Islamabad,P006,Mechanical Keyboard,Electronics,2,85.00
1008,2024-02-15,C003,Usman Tariq,Islamabad,P007,Monitor 27 inch,Electronics,1,450.00
1009,2024-03-01,C006,Ayesha Malik,Karachi,P003,Office Chair,Furniture,2,350.00
1010,2024-03-05,C001,Ahmed Khan,Lahore,P006,Mechanical Keyboard,Electronics,1,85.00
1011,2024-03-10,C007,Bilal Ahmed,Lahore,P008,Webcam HD,Electronics,1,75.00
1012,2024-03-15,C004,Fatima Noor,Lahore,P005,Standing Desk,Furniture,1,600.00
  • Save the file as sample_orders.csv by pressing Cmd+S (macOS) or Ctrl+S (Windows).
  • Create another new file and paste the following content:
customer_id,customer_name,customer_city
C001,Ahmed Khan,Lahore
C002,Sara Ali,Karachi
C003,Usman Tariq,Islamabad
C004,Fatima Noor,Lahore
C005,Hassan Raza,Islamabad
C006,Ayesha Malik,Karachi
C007,Bilal Ahmed,Lahore
  • Save the file as customers.csv.
  • Create one more new file and paste the following content:
product_id,product_name,product_category
P001,Laptop Pro 15,Electronics
P002,Wireless Mouse,Electronics
P003,Office Chair,Furniture
P004,USB-C Hub,Electronics
P005,Standing Desk,Furniture
P006,Mechanical Keyboard,Electronics
P007,Monitor 27 inch,Electronics
P008,Webcam HD,Electronics
  • Save the file as products.csv.

What is this data?

This is a retail orders dataset with 12 transactions spanning January to March 2024. It includes 7 unique customers across 3 cities and 8 products in 2 categories (Electronics and Furniture).

The separate customers.csv and products.csv files will be used later to populate dimension tables in your star schema. The main orders file contains denormalized data that your ETL job will transform.

Now upload all three files to the raw zone in your S3 bucket.

  • In your terminal, make sure you are inside the lakehouse-data folder on your Desktop.
  • Upload the orders file to the raw zone by running:
aws s3 cp sample_orders.csv s3://[[BUCKET_NAME="your-lakehouse-bucket-name"]]/raw/orders/
  • Upload the dimension files by running these commands:
aws s3 cp customers.csv s3://[[BUCKET_NAME="your-lakehouse-bucket-name"]]/raw/dimensions/
aws s3 cp products.csv s3://[[BUCKET_NAME="your-lakehouse-bucket-name"]]/raw/dimensions/

Why separate the orders from dimensions?

Organizing raw data by logical type (orders/ vs dimensions/) makes it clear what each file represents. The Glue crawler will discover each path as a separate table in the Data Catalog.

  • Verify all files uploaded successfully by running:
aws s3 ls s3://[[BUCKET_NAME="your-lakehouse-bucket-name"]]/raw/ --recursive

You should see three files listed: raw/dimensions/customers.csv, raw/dimensions/products.csv, and raw/orders/sample_orders.csv.

Upload command failing?

  • If you see NoSuchBucket, double-check the bucket name matches exactly what you created earlier.
  • If you see Unable to locate credentials, your AWS CLI is not configured. Run aws configure and enter your Access Key ID and Secret Access Key.
  • If you see No such file or directory, make sure your terminal is in the lakehouse-data folder where you saved the CSV files.

Still stuck? Help me troubleshoot my S3 upload error.

Your data lake foundation is in place. You have a structured S3 bucket, a properly permissioned IAM role, and raw data ready for processing. Next up, you'll use AWS Glue to automatically discover the schema of your uploaded data.

Discover Schema with Glue Crawler

Your data lake has raw data sitting in S3, but your ETL tools have no idea what columns or data types that CSV contains.

In this step, you'll create a Glue Data Catalog database and point a Glue Crawler at your raw zone. The crawler automatically discovers the schema of your data and registers it as a table in the catalog, so downstream jobs can reference it by name instead of parsing files manually.

In this step, get ready to:
  • Create a Glue database to organize your catalog tables.
  • Create and run a crawler that scans your raw orders data.
  • Inspect the auto-discovered schema in the Data Catalog.
Create a Glue database

The Glue Data Catalog organizes metadata into databases. Each database is a logical container for the table definitions that crawlers and ETL jobs use.

  • Open the AWS Management Console and search for AWS Glue.
  • Click AWS Glue to open the Glue console.
  • In the left navigation menu, under Data Catalog, click Databases.
  • Click Add database.
  • In the Name field, enter lakehouse_db.
  • Leave the Location field empty.
  • Click Create database.

You should see lakehouse_db appear in the Databases list.

Why underscores in the name?

Glue database names are folded to lowercase and must be compatible with Amazon Athena. Athena requires only alphanumeric and underscore characters in database names.

Don't see the Add database button?

Make sure you are in the Data Catalog section of the left menu, not the ETL section. The Databases page sits directly under Data Catalog in the navigation tree.

help me find the Add database button in the AWS Glue console

Create and configure the crawler

A crawler connects to your data store, runs built-in classifiers to identify the data format, infers column names and types, and writes the resulting table definition into the Data Catalog.

  • In the left navigation menu, under Data Catalog, click Crawlers.
  • Click Create crawler.
  • In the Name field, enter raw-orders-crawler.
  • Click Next.

You are now on the data source configuration page.

  • For Is your data already mapped to AWS Glue tables?, leave Not yet selected.
  • Click Add a data source.
  • In the modal that appears, for Data source, keep S3 selected.
  • For S3 path, click Browse S3 and navigate to your bucket, then select the raw/orders/ folder.
  • The path should read s3://my-lakehouse-bucket/raw/orders/.
  • Leave all other settings at their defaults.
  • Click Add an S3 data source.
  • Click Next.

Now you configure the IAM role the crawler uses to access S3 and write to the catalog.

  • For Existing IAM role, select GlueLakehouseRole from the dropdown.
  • Click Next.

The final configuration page sets the output database and schedule.

  • For Target database, select lakehouse_db from the dropdown.
  • Leave Table name prefix empty.
  • For Crawler schedule, keep On demand selected.
  • Click Next.
  • Review your settings on the summary page.
  • Click Create crawler.

You should see a success banner confirming that raw-orders-crawler was created.

What is a Glue Crawler doing under the hood?

The crawler reads your S3 files and runs built-in classifiers (CSV, JSON, Parquet, etc.) to detect the data format. Once it identifies the format, it infers column names from headers and guesses data types from values.

The result is a table definition in the Data Catalog that any Glue job, Athena query, or Redshift Spectrum query can reference by name.

Can't find GlueLakehouseRole in the dropdown?

The Glue console only shows IAM roles that have a trust policy for the glue.amazonaws.com service principal. Go back to IAM and confirm your role's trust relationship includes glue.amazonaws.com as the allowed service.

help me fix my IAM role trust policy so it shows up in the Glue crawler dropdown

Run the crawler and inspect the catalog

With the crawler configured, it is time to let it scan your raw orders data and build the table definition.

  • On the Crawlers page, select the checkbox next to raw-orders-crawler.
  • Click Run.
  • Wait for the Status column to change from Running to Completed. This takes about 1-2 minutes.
  • Click the refresh icon if the status does not update automatically.

Once complete, the Tables added column should show 1.

  • In the left navigation, under Data Catalog, click Tables.
  • Click the table named orders to open its details.
  • Scroll down to the Schema section.

You should see 10 columns listed with their inferred data types. The Classification field at the top of the table details page should say csv.

What should the schema look like?

The crawler infers the following columns from your sample_orders.csv file:

  • order_id (bigint), order_date (string), customer_id (string).
  • customer_name (string), customer_city (string).
  • product_id (string), product_name (string), product_category (string).
  • quantity (bigint), unit_price (double).

Numeric columns like order_id and quantity are inferred as bigint because Glue defaults integers to the widest safe type. The order_date column stays as string because CSV has no native date type. You will cast it properly in the ETL step.

Crawler shows 0 tables added?

Double-check that the S3 path in your crawler points exactly to s3://your-bucket/raw/orders/ with a trailing slash. If the path points to the bucket root or a parent folder, the crawler may create multiple tables or fail to find data.

Also confirm that sample_orders.csv is actually inside the raw/orders/ prefix by checking in the S3 console.

help me debug why my Glue crawler found zero tables

Your raw data now has a registered schema in the Glue Data Catalog. Next up, you'll write a PySpark ETL job that reads from this catalog table, transforms the data, and writes optimized Parquet to your curated zone.

Transform Data with Glue ETL

Your raw CSV data is sitting in the lake and the Data Catalog knows its schema. But CSV is not an analytics-friendly format. It is slow to scan, uncompressed, and lacks the columnar structure that query engines prefer.

In this step, you will create an AWS Glue ETL job that reads your raw CSV from the Data Catalog, transforms it (casting types, adding derived columns), and writes it to the curated zone as partitioned Parquet. Parquet is a columnar format that compresses well and enables query engines to skip irrelevant data.

In this step, get ready to:
  • Create a PySpark Glue ETL job that transforms raw CSV to optimized Parquet.
  • Run the job and verify Parquet files appear in the curated zone.
  • Crawl the curated zone to register the new table in the Data Catalog.
Create the Glue ETL job

The Glue Script Editor lets you write a PySpark script that reads from the Data Catalog, applies transformations, and writes output to S3.

  • In the AWS Glue console, click ETL jobs in the left navigation pane.
  • In the Create job section, select Script editor.
  • For Engine, select Spark.
  • Under Options, select Start fresh.
  • Click Create.

The script editor opens with boilerplate code. You will replace it entirely with your ETL script.

  • Select all the boilerplate code in the Script pane and delete it.
  • Paste the following PySpark ETL script into the empty Script pane:
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.sql.functions import col, year, month, round as spark_round

args = getResolvedOptions(sys.argv, ['JOB_NAME', 'SOURCE_DATABASE', 'SOURCE_TABLE', 'TARGET_PATH'])

sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

# Read from Glue Data Catalog
dynamic_frame = glueContext.create_dynamic_frame.from_catalog(
    database=args['SOURCE_DATABASE'],
    table_name=args['SOURCE_TABLE']
)

# Convert to Spark DataFrame for transformations
df = dynamic_frame.toDF()

# Cast data types
df = df.withColumn("order_date", col("order_date").cast("date"))
df = df.withColumn("quantity", col("quantity").cast("integer"))
df = df.withColumn("unit_price", col("unit_price").cast("double"))

# Add derived columns
df = df.withColumn("total_amount", spark_round(col("quantity") * col("unit_price"), 2))
df = df.withColumn("year", year(col("order_date")).cast("string"))
df = df.withColumn("month", month(col("order_date")).cast("string"))

# Write to curated zone as partitioned Parquet
df.write.mode("overwrite") \
    .partitionBy("year", "month") \
    .parquet(args['TARGET_PATH'])

job.commit()

What does this script do?

This PySpark script follows the standard Glue ETL pattern: extract, transform, load.

  • It reads from the Glue Data Catalog using the database and table name you pass as job parameters.
  • It converts the DynamicFrame to a Spark DataFrame so you can use native Spark transformations.
  • It casts columns to proper types (order_date to date, quantity to integer, unit_price to double).
  • It adds a total_amount derived column (quantity times unit_price) and extracts year and month from the order date for partitioning.
  • It writes the result to S3 as Parquet, partitioned by year and month. Partitioning means query engines can skip entire folders of irrelevant data.

Now you need to configure the job settings. The job name, IAM role, worker configuration, and parameters all live in the Job details tab.

  • Click the Job details tab at the top of the editor.
  • For Name, replace the default with lakehouse-etl-job.
  • For IAM Role, select GlueLakehouseRole from the dropdown.
  • For Worker type, select G.1X.
  • For Requested number of workers, enter 2.
  • Scroll down to the Advanced properties section and expand it.
  • Under Job parameters, add the following three key-value pairs (click Add new parameter for each):
  • Key: --SOURCE_DATABASE Value: lakehouse_db.
  • Key: --SOURCE_TABLE Value: orders.
  • Key: --TARGET_PATH Value: s3://my-lakehouse-bucket/curated/orders/.

Why use job parameters?

Job parameters keep the script reusable. Instead of hardcoding your database name or S3 path inside the code, you pass them as arguments. This is an enterprise best practice that lets you reuse the same script across dev, staging, and production environments.

  • Click Save in the top right corner.

Your job now appears in the Glue Jobs list with the name lakehouse-etl-job.

Don't see GlueLakehouseRole in the dropdown?

Make sure the role name matches exactly what you created in Step 1. The Glue console only shows roles that have a trust policy allowing the Glue service to assume them. If the role does not appear, confirm its trust relationship includes glue.amazonaws.com as a trusted entity.

help me debug why my IAM role doesn't appear in the Glue job dropdown

Run and monitor the ETL job

With the script saved and the job configured, you can now execute it. The job spins up Spark workers, processes your raw CSV, and writes Parquet files to the curated zone.

  • On the lakehouse-etl-job page, click Run in the top right corner.
  • Click the Runs tab to monitor the job execution.

How long does the job take?

The job will show a status of Running for approximately 2-4 minutes. Most of that time is Spark cluster startup. The actual data processing finishes in seconds because your dataset is small. Wait until the status changes to Succeeded.

  • Wait for the Run status to show Succeeded. You can click the refresh icon to update the status.
  • Navigate to the S3 console and open your bucket.
  • Browse to the curated/orders/ prefix.

You should see partition folders: year=2024/month=1/, year=2024/month=2/, and year=2024/month=3/. Each folder contains Parquet files.

Job status shows Failed?

Check the error logs by clicking the Error logs link in the Runs tab. Common causes:

  • The --SOURCE_TABLE parameter does not match the table name in your Data Catalog. Verify it is orders (the table the crawler created).
  • The --TARGET_PATH has a typo in the bucket name. The S3 path must match your actual bucket.
  • The IAM role does not have S3 write permissions. Check the inline policy on GlueLakehouseRole includes s3:PutObject.

help me debug my failed Glue ETL job

Why Parquet and partitioning?

Parquet stores data in columns rather than rows. This means a query that only needs product_category and total_amount can skip reading every other column entirely.

Partitioning by year and month creates physical folder boundaries in S3. When you query for January 2024 data, the engine reads only the year=2024/month=1/ folder and skips everything else. This directly reduces cost on Athena, which charges per TB scanned.

Crawl the curated zone

The Parquet files are in S3, but the Data Catalog does not know about them yet. You need a second crawler to register this curated data so that other services (Athena, Redshift) can discover it.

  • In the AWS Glue console, click Crawlers in the left navigation pane.
  • Click Create crawler.
  • For Name, enter curated-orders-crawler.
  • Click Next.
  • For Is your data already mapped to Glue tables?, keep Not yet selected.
  • Click Add a data source.
  • For Data source, select S3.
  • For S3 path, enter s3://my-lakehouse-bucket/curated/orders/.
  • Click Add an S3 data source.
  • Click Next.
  • For Existing IAM role, select GlueLakehouseRole.
  • Click Next.
  • For Target database, select lakehouse_db.
  • Leave the table name prefix blank.
  • For Frequency, select On demand.
  • Click Next.
  • Review your configuration and click Create crawler.
  • Select curated-orders-crawler from the crawlers list and click Run.
  • Wait for the crawler status to show Completed. This takes about 1-2 minutes.

Now verify the new table in the Data Catalog.

  • In the left navigation, click Tables under Data Catalog.
  • Look for a new table in lakehouse_db with parquet classification. The table name will be orders_curated or orders (depending on the folder structure the crawler detects).
  • Click the table name to inspect its schema.

You should see columns including total_amount (the derived column your ETL added) and partition keys year and month listed at the bottom of the schema.

Don't see a new table with Parquet classification?

If the crawler completed but created no table, verify that the S3 path points exactly to s3://your-bucket/curated/orders/ (with the trailing slash). Also confirm your ETL job ran successfully by checking that Parquet files exist at that path.

help me figure out why my Glue crawler didn't create a table for my curated Parquet data

Your data lake now has two layers registered in the catalog: raw CSV and curated Parquet. Next up, you will load this curated data into a Redshift Serverless data warehouse and model it as a star schema.

Build a Star Schema in Redshift Serverless

Your curated Parquet data is sitting in the lake, neatly partitioned by year and month. Now it is time to give that data structure by loading it into a proper data warehouse.

In this step you will activate Amazon Redshift Serverless, design a star schema with dimension and fact tables, and load data from S3 using the COPY command.

In this step, get ready to:
  • Activate Redshift Serverless with the free trial.
  • Create dimension and fact tables in a star schema design.
  • Load data from S3 and run an analytical query to verify the warehouse works.
Activate Redshift Serverless free trial

Redshift Serverless automatically provisions and scales data warehouse capacity. It comes with a $300 free trial credit valid for 90 days, so you pay nothing for this project.

  • In the AWS Console, search for Redshift and open the Amazon Redshift console.
  • Click Try Redshift Serverless Free Trial.
  • Under Configuration, select Use default settings.
  • Click Save configuration.

What do the default settings create?

Redshift Serverless creates a default namespace (a container for your databases and users) and a default workgroup (the compute resources that run your queries) with 8 RPU base capacity.

It also requires a VPC with at least three subnets in three different Availability Zones. The default settings use your account's default VPC, which meets this requirement.

  • Wait for setup to complete, then click Continue to go to your Serverless dashboard.
  • Confirm that the workgroup status shows Available in the dashboard.

Workgroup not showing Available?

The setup can take 2-3 minutes. If it stays in a creating state for longer than 5 minutes, check that your default VPC has at least three subnets in different Availability Zones.

Help me troubleshoot my Redshift Serverless workgroup not becoming available.

Before you can load data from S3 into Redshift, you need an IAM role that grants Redshift read access to your bucket.

  • Open the IAM console in a new tab by searching for IAM in the AWS Console search bar.
  • Click Roles in the left sidebar, then click Create role.
  • Under Trusted entity type, select AWS service.
  • Under Use case, search for and select Redshift, then choose Redshift - Customizable.
  • Click Next.
  • Skip the permissions page for now (you will add an inline policy next). Click Next.
  • Enter RedshiftS3ReadRole as the Role name.
  • Click Create role.

Now add the inline policy that grants S3 read access.

  • Click on the RedshiftS3ReadRole you just created to open its details.
  • Click the Permissions tab, then click Add permissions and select Create inline policy.
  • Click the JSON tab and paste the following policy:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::[[BUCKET_NAME=\"my-lakehouse-bucket\"]]",
                "arn:aws:s3:::[[BUCKET_NAME=\"my-lakehouse-bucket\"]]/*"
            ]
        }
    ]
}

What does this policy do?

This grants the Redshift role permission to read objects (s3:GetObject) and list the bucket contents (s3:ListBucket) for your specific data lake bucket. The COPY command needs both permissions to locate and read Parquet and CSV files.

  • Click Next.
  • Enter S3ReadAccess as the policy name, then click Create policy.
  • Copy and save the role ARN from the Summary section at the top of the role details page. You will need it for the COPY commands later.

Now associate this IAM role with your Redshift namespace so Redshift can assume it.

  • Go back to the Amazon Redshift console.
  • In the left sidebar, click Namespace configuration.
  • Click the default namespace.
  • Click the Security and encryption tab.
  • Click Manage IAM roles.
  • Select RedshiftS3ReadRole from the dropdown and click Associate IAM role.
  • Click Save changes.

Don't see your role in the dropdown?

Make sure the role's trust policy allows redshift.amazonaws.com as a trusted entity. If you selected "Redshift - Customizable" during creation, this is set automatically.

Help me fix my Redshift IAM role association issue.

Create dimension and fact tables

A star schema organizes data into a central fact table (containing measurable events like orders) surrounded by dimension tables (containing descriptive attributes like customer details and product info).

  • In the Redshift Serverless dashboard, click Query data to open the Redshift Query Editor v2 in a new browser tab.
  • In the left tree-view panel, click the workgroup name (default) to connect.
  • When prompted for authentication, leave Federated user selected and click Create connection.

You are now connected to the dev database, which is the default database created by Redshift Serverless.

  • Create the dimension tables by pasting and running the following SQL in the query editor:
-- Create dimension tables
CREATE TABLE dim_customer (
    customer_id VARCHAR(10) PRIMARY KEY,
    customer_name VARCHAR(100),
    customer_city VARCHAR(50)
);

CREATE TABLE dim_product (
    product_id VARCHAR(10) PRIMARY KEY,
    product_name VARCHAR(100),
    product_category VARCHAR(50)
);

What are dimension tables?

Dimension tables hold descriptive attributes that provide context to your measurements. dim_customer stores who placed each order, and dim_product stores what was ordered.

By separating these attributes from the transactional data, you avoid duplication and make analytical queries faster.

  • Click Run to execute the statement.
  • Verify you see a success message in the results panel for both CREATE TABLE statements.
  • Now create the fact table by clearing the editor and pasting the following SQL:
-- Create fact table
CREATE TABLE fact_orders (
    order_id INTEGER PRIMARY KEY,
    order_date DATE,
    customer_id VARCHAR(10) REFERENCES dim_customer(customer_id),
    product_id VARCHAR(10) REFERENCES dim_product(product_id),
    quantity INTEGER,
    unit_price DECIMAL(10,2),
    total_amount DECIMAL(10,2)
);

What makes this a star schema?

The fact_orders table sits at the center of the star. It contains foreign keys (customer_id, product_id) that point to the surrounding dimension tables, plus numeric measures (quantity, unit_price, total_amount) that analysts aggregate.

When you visualize the relationships, the fact table in the center with dimension tables radiating outward looks like a star.

  • Click Run to create the fact table.
  • Verify you see a success message in the results panel.

Seeing a "relation already exists" error?

This means you ran the same CREATE TABLE statement twice. The tables already exist and you can proceed to the next substep. If you want to start fresh, run DROP TABLE fact_orders, dim_product, dim_customer; first, then re-run both CREATE blocks in order (dimensions first, then fact).

Help me fix my Redshift CREATE TABLE errors.

Load data from S3 and run an analytical query

With the schema in place, you will load data using Redshift's COPY command. This command reads files directly from S3 using the IAM role you associated earlier.

  • Load the dimension tables first. Clear the editor and paste the following COPY commands, replacing the bucket name and account ID with your own values:
-- Load dimensions from raw CSV
COPY dim_customer (customer_id, customer_name, customer_city)
FROM 's3://[[BUCKET_NAME="my-lakehouse-bucket"]]/raw/dimensions/customers.csv'
IAM_ROLE 'arn:aws:iam::[[ACCOUNT_ID="your 12-digit AWS account ID"]]:role/RedshiftS3ReadRole'
FORMAT AS CSV
IGNOREHEADER 1;

COPY dim_product (product_id, product_name, product_category)
FROM 's3://[[BUCKET_NAME="my-lakehouse-bucket"]]/raw/dimensions/products.csv'
IAM_ROLE 'arn:aws:iam::[[ACCOUNT_ID="your 12-digit AWS account ID"]]:role/RedshiftS3ReadRole'
FORMAT AS CSV
IGNOREHEADER 1;

What does the COPY command do?

COPY is Redshift's bulk-loading command. It reads files directly from S3 using the specified IAM role for authentication.

FORMAT AS CSV tells Redshift to parse CSV format. IGNOREHEADER 1 skips the first row (the column header line) so it is not loaded as data.

  • Click Run to load both dimension tables.
  • Verify the results panel shows success messages for both COPY commands.
  • Now load the fact table from the curated Parquet files. Clear the editor and paste this COPY command:
-- Load fact table from S3 curated zone (Parquet)
COPY fact_orders (order_id, order_date, customer_id, product_id, quantity, unit_price, total_amount)
FROM 's3://[[BUCKET_NAME="my-lakehouse-bucket"]]/curated/orders/'
IAM_ROLE 'arn:aws:iam::[[ACCOUNT_ID="your 12-digit AWS account ID"]]:role/RedshiftS3ReadRole'
FORMAT AS PARQUET;

Why Parquet for the fact table?

Your Glue ETL job already transformed the raw CSV into optimized Parquet in the curated zone. FORMAT AS PARQUET tells Redshift to read the columnar Parquet format directly. This is the standard pattern for loading curated data lake content into a warehouse.

  • Click Run to load the fact table.
  • Verify you see a success message confirming 12 rows loaded.

Seeing "S3ServiceException: Access Denied"?

Check that the IAM role ARN in your COPY command exactly matches the role you created. Also confirm that you associated RedshiftS3ReadRole with your Redshift namespace in the previous substep.

If the bucket name is wrong, you will see a "The specified bucket does not exist" error instead. Double-check the S3 path matches your actual bucket name.

Help me fix my Redshift COPY command S3 access error.

Finally, run an analytical query that joins the fact and dimension tables to prove your star schema is working.

  • Clear the editor and paste this analytical query:
-- Sample analytical query: Revenue by product category and city
SELECT
    dp.product_category,
    dc.customer_city,
    SUM(f.total_amount) AS total_revenue,
    COUNT(f.order_id) AS order_count
FROM fact_orders f
JOIN dim_customer dc ON f.customer_id = dc.customer_id
JOIN dim_product dp ON f.product_id = dp.product_id
GROUP BY dp.product_category, dc.customer_city
ORDER BY total_revenue DESC;

What does this query show?

This query demonstrates the power of the star schema pattern. It joins the central fact table to both dimension tables, then aggregates revenue by product category and customer city.

You should see Electronics in Lahore at the top with the highest revenue. This is exactly the type of analytical query a data warehouse is designed to handle efficiently.

  • Click Run to execute the query.
  • Verify you see a results table with product categories, cities, total revenue, and order counts.

✔️ I see query results with revenue data

Your star schema is fully operational. The data flows from your S3 data lake through the curated zone and into a structured warehouse.

ⓧ I see an error or no results

If you see zero rows, the COPY commands may not have loaded data successfully. Run a quick check:

SELECT COUNT(*) FROM fact_orders;
SELECT COUNT(*) FROM dim_customer;
SELECT COUNT(*) FROM dim_product;

You should see 12 rows in fact_orders, 7 in dim_customer, and 8 in dim_product. If any table shows 0, re-run the corresponding COPY command from earlier in this substep.

Help me debug my Redshift star schema query returning no results.

✔️ Awesome, I've got everything!

Great. Double check that you have saved or noted the IAM role ARN for reference.

ⓧ I'd like to double check the full code

Here is the complete SQL you should have run in this step:

-- Create dimension tables
CREATE TABLE dim_customer (
    customer_id VARCHAR(10) PRIMARY KEY,
    customer_name VARCHAR(100),
    customer_city VARCHAR(50)
);

CREATE TABLE dim_product (
    product_id VARCHAR(10) PRIMARY KEY,
    product_name VARCHAR(100),
    product_category VARCHAR(50)
);

-- Create fact table
CREATE TABLE fact_orders (
    order_id INTEGER PRIMARY KEY,
    order_date DATE,
    customer_id VARCHAR(10) REFERENCES dim_customer(customer_id),
    product_id VARCHAR(10) REFERENCES dim_product(product_id),
    quantity INTEGER,
    unit_price DECIMAL(10,2),
    total_amount DECIMAL(10,2)
);

-- Load fact table from S3 curated zone (Parquet)
COPY fact_orders (order_id, order_date, customer_id, product_id, quantity, unit_price, total_amount)
FROM 's3://YOUR-BUCKET-NAME/curated/orders/'
IAM_ROLE 'arn:aws:iam::YOUR-ACCOUNT-ID:role/RedshiftS3ReadRole'
FORMAT AS PARQUET;

-- Load dimensions from raw CSV
COPY dim_customer (customer_id, customer_name, customer_city)
FROM 's3://YOUR-BUCKET-NAME/raw/dimensions/customers.csv'
IAM_ROLE 'arn:aws:iam::YOUR-ACCOUNT-ID:role/RedshiftS3ReadRole'
FORMAT AS CSV
IGNOREHEADER 1;

COPY dim_product (product_id, product_name, product_category)
FROM 's3://YOUR-BUCKET-NAME/raw/dimensions/products.csv'
IAM_ROLE 'arn:aws:iam::YOUR-ACCOUNT-ID:role/RedshiftS3ReadRole'
FORMAT AS CSV
IGNOREHEADER 1;

-- Sample analytical query: Revenue by product category and city
SELECT
    dp.product_category,
    dc.customer_city,
    SUM(f.total_amount) AS total_revenue,
    COUNT(f.order_id) AS order_count
FROM fact_orders f
JOIN dim_customer dc ON f.customer_id = dc.customer_id
JOIN dim_product dp ON f.product_id = dp.product_id
GROUP BY dp.product_category, dc.customer_city
ORDER BY total_revenue DESC;

Your data warehouse is live and queryable. Next up, you will query the same data directly from the lake layer using Athena, demonstrating the core lakehouse pattern where the same data is accessible through both a warehouse engine and a serverless lake engine.

Query the Lake Layer with Athena

Your data now lives in two places: the curated Parquet layer in S3 and the star schema in Redshift. A real lakehouse gives analysts the choice of where to query depending on the use case.

Amazon Athena lets you run SQL directly on data sitting in S3. No server provisioning, no data loading. You pay only for the bytes scanned. In this step, you will query the curated layer to prove the lakehouse pattern works.

In this step, get ready to:
  • Run your first Athena query against the curated Parquet table in S3.
  • Compare lake vs warehouse query patterns using partition pruning.
Run your first Athena query

Athena reads directly from the Glue Data Catalog tables you already have. Because the curated table points at Parquet files in S3, Athena can query them without any data movement.

  • In the AWS Console search bar, type Athena and select the Athena service.
  • Click **Query editor** in the left sidebar if you are not already there.
  • In the **Database** dropdown on the left panel, select lakehouse_db.

You should see your tables listed below the dropdown, including orders_curated.

What about query results storage?

Athena uses managed query results by default. This means your results are stored and managed automatically at no additional cost. You do not need to create a separate S3 bucket for query output.

If Athena prompts you to set up a query result location, click **Edit settings**, then select **Manage query results via Athena** and click **Save**.

  • Paste this query into the query editor to calculate total revenue by product category:
-- Query 1: Total revenue by product category (runs on curated Parquet in S3)
SELECT
    product_category,
    SUM(quantity * unit_price) AS total_revenue,
    COUNT(*) AS order_count
FROM lakehouse_db.orders_curated
GROUP BY product_category
ORDER BY total_revenue DESC;

What does this query do?

This query aggregates all orders by product_category, calculating total revenue and order count. It runs directly on the Parquet files stored in your S3 curated zone.

No data was loaded into Athena. It scans the S3 objects in place, which is the core value of the lakehouse pattern.

  • Click **Run** (or press Ctrl+Enter).

You should see results showing Electronics with higher total revenue than Furniture. Note the **Data scanned** value shown below the results. It will be very small (a few KB) because your dataset is small.

Query failed or no results?

Make sure you selected lakehouse_db in the Database dropdown. If you see a permissions error, confirm your AWS user has AmazonAthenaFullAccess and AmazonS3ReadOnlyAccess policies attached.

If you see "Table not found", go back to Glue and verify that the curated crawler completed successfully and created the orders_curated table.

Still stuck? Help me debug my Athena query that's failing against the curated Parquet table in lakehouse_db.

Why Athena alongside Redshift?

Redshift is optimized for complex joins and repeated dashboards. Athena is optimized for ad-hoc exploration and one-off questions where you do not want to load data first.

In a real lakehouse, data engineers use Redshift for predictable reporting workloads and Athena for exploratory analysis. Same data, different engines, different cost profiles.

Compare lake vs warehouse query patterns

Partitioned Parquet lets Athena skip entire folders of data that do not match your filter. This reduces both scan time and cost. Your curated zone is already partitioned by year and month, so filtering on those columns triggers partition pruning.

  • Clear the query editor and paste this partition-pruned query that only scans January 2024 data:
-- Query 2: Partition-pruned query (only scans January 2024 partition)
SELECT
    customer_name,
    product_name,
    total_amount
FROM lakehouse_db.orders_curated
WHERE year = '2024' AND month = '1'
ORDER BY total_amount DESC;

What is partition pruning?

When you filter on partition columns (year and month), Athena only reads the matching S3 prefixes. It skips February and March entirely. With large datasets, this can reduce data scanned (and cost) by orders of magnitude.

Athena charges $5 per TB of data scanned. Partition pruning is the primary cost optimization lever for lake queries.

  • Click **Run**.

You should see only 4 rows (the January 2024 orders). Note the Data scanned value. It is even smaller than the previous query because Athena only read the year=2024/month=1/ partition folder.

  • Clear the editor and paste this final query to see monthly revenue trends:
-- Query 3: Monthly revenue trend
SELECT
    year,
    month,
    SUM(total_amount) AS monthly_revenue,
    COUNT(*) AS order_volume
FROM lakehouse_db.orders_curated
GROUP BY year, month
ORDER BY year, month;
  • Click **Run**.

You should see 3 rows, one for each month (January, February, March 2024), with the monthly revenue and order volume for each.

What should I see?

January should show the highest revenue (driven by a Laptop Pro 15 order). February is close behind. March has fewer high-value items. The exact numbers depend on your sample_orders.csv data.

Seeing zero results or wrong numbers?

Check that the partition column values match your filter exactly. The month column stores values as '1', '2', '3' (not '01', '02'). Using month = '01' returns nothing.

If total_amount shows NULL, confirm your Glue ETL job ran successfully and the curated Parquet files contain the derived total_amount column.

Help me troubleshoot why my Athena partition-pruned query returns zero results or unexpected values.

✔️ Awesome, I've got everything!

Great. Double check you ran all three queries successfully and noted the Data scanned values.

ⓧ I'd like to double check the full code

Here are all three Athena queries you should have run:

-- Query 1: Total revenue by product category (runs on curated Parquet in S3)
SELECT
    product_category,
    SUM(quantity * unit_price) AS total_revenue,
    COUNT(*) AS order_count
FROM lakehouse_db.orders_curated
GROUP BY product_category
ORDER BY total_revenue DESC;

-- Query 2: Partition-pruned query (only scans January 2024 partition)
SELECT
    customer_name,
    product_name,
    total_amount
FROM lakehouse_db.orders_curated
WHERE year = '2024' AND month = '1'
ORDER BY total_amount DESC;

-- Query 3: Monthly revenue trend
SELECT
    year,
    month,
    SUM(total_amount) AS monthly_revenue,
    COUNT(*) AS order_volume
FROM lakehouse_db.orders_curated
GROUP BY year, month
ORDER BY year, month;

You just proved the lakehouse pattern in action. The same curated data is queryable through Redshift (for structured reporting) and Athena (for ad-hoc exploration). Next up, you will layer governance on top of everything with Amazon DataZone.

Govern with Amazon DataZone

Your data is now queryable through both Redshift and Athena. But in an enterprise, queryable data is only half the story.

Without governance, teams duplicate datasets, misinterpret column meanings, and access data they shouldn't. Amazon DataZone solves this by providing a governed catalog where data assets are published, described with business terminology, and controlled through access policies.

In this step, get ready to:
  • Create a DataZone domain and project to establish your governance boundary.
  • Build a business glossary with standardized terms for your retail data.
  • Publish your Glue Data Catalog assets to the DataZone catalog for governed discovery.
Create a DataZone domain and project

A DataZone domain is the top-level governance container that groups your data assets, users, and projects together. A project within that domain lets a team collaborate on publishing and discovering data.

  • Navigate to the Amazon DataZone console at https://console.aws.amazon.com/datazone.
  • Use the region selector in the top navigation bar to confirm you are in the same region as your S3 bucket and Glue resources.
  • Click Create domain.
  • Enter retail-lakehouse-domain in the Name field.
  • Leave Data encryption on the default (AWS managed key).
  • Under Service access, choose to have DataZone create a new service role for you.
  • Check the Quick setup box to have DataZone automatically create IAM roles for Glue and Redshift access, an S3 bucket, an administrative project, and environment profiles.
  • Click Create domain.

What does Quick setup do?

Quick setup automates the creation of IAM roles (for Glue and Redshift access management), an S3 bucket, environment profiles, and an admin project. This saves you from manually wiring up permissions between DataZone and your other AWS services.

  • Wait for the domain status to show Available. This can take several minutes.
  • Once available, note the Data portal URL displayed on the domain summary page. You will use this URL for the remaining steps.
  • Click Open data portal to launch the DataZone portal in a new tab.

Now create a project within your domain.

  • In the DataZone data portal, click Create Project.
  • Enter retail-analytics as the Name.
  • Enter a short description like Retail lakehouse analytics project.
  • Click Create project.

You should see your new retail-analytics project appear in the portal with its own Data and Members tabs.

Domain stuck in Creating status?

Domain creation can take up to 5 minutes. If it stays in a creating state for longer than 10 minutes, check that your IAM user has permissions to create IAM roles (required by Quick setup).

Help me troubleshoot my DataZone domain creation

Why DataZone instead of Lake Formation alone?

Lake Formation handles fine-grained access control at the table and column level. DataZone builds on top of that by adding a business catalog, glossary, publishing workflows, and subscription-based data sharing.

For a Data Architect role, you need to demonstrate both the technical access layer AND the business governance layer. DataZone covers the governance side that Lake Formation does not.

Create a business glossary

A business glossary standardizes terminology across your organization. When different teams say "revenue," they might mean different things. A glossary eliminates that ambiguity by providing official definitions attached directly to data assets.

  • In the DataZone data portal, click Catalog in the top navigation bar.
  • Click Glossaries.
  • Click Create glossary.
  • Enter Retail Analytics Glossary as the name.
  • Enter a description like Standard business definitions for retail lakehouse data.
  • Set yourself as the owner.
  • Click Create glossary.
  • Enable the glossary by toggling the Enabled switch on the glossary details page.

Now add three business terms to your glossary.

  • Open your Retail Analytics Glossary and create the first term:
  • Enter Total Revenue as the term name.
  • Enter a description: The sum of all order amounts (quantity multiplied by unit price) within a given time period. Calculated as SUM(total_amount) from fact_orders.
  • Set yourself as the owner and click Create term.
  • Enable the term using the Enabled toggle.

Repeat the process for the second term.

  • Navigate back to the glossary and create a new term.
  • Enter Customer Lifetime Value as the term name.
  • Enter a description: The total revenue generated by a single customer across all their orders. Calculated by summing total_amount grouped by customer_id.
  • Set yourself as the owner, click Create term, and enable it.

Now add the third and final term.

  • Navigate back to the glossary and create a new term.
  • Enter Order as the term name.
  • Enter a description: A single purchase transaction containing one product line item. Each order has a unique order_id and is associated with one customer and one product.
  • Set yourself as the owner, click Create term, and enable it.

You should now see three enabled terms in your Retail Analytics Glossary: Total Revenue, Customer Lifetime Value, and Order.

Why create a business glossary?

In enterprise data platforms, different teams often interpret the same metric differently. A glossary provides a single source of truth for business definitions, tied directly to the data assets in your catalog.

When other teams browse your DataZone catalog, they can see exactly how "Total Revenue" is calculated rather than guessing from raw column names.

Don't see the Glossaries option?

Make sure you are in the DataZone data portal (the URL from your domain summary), not the main AWS console. The glossary feature lives under the Catalog menu in the portal's top navigation bar.

Help me find the glossary option in DataZone

Publish data assets to the catalog

Your Glue Data Catalog already holds schema metadata for both the raw and curated tables. By connecting DataZone to your Glue catalog, you make those assets discoverable through governed search with business context attached.

  • In the DataZone data portal, select your retail-analytics project from the top navigation.
  • Click the Data tab.
  • In the left navigation pane, click Data sources.
  • Click Create data source.
  • Enter glue-lakehouse-source as the Name.
  • Under Data source type, select AWS Glue.
  • Under Select an environment, choose the data lake environment profile that Quick setup created.
  • Under Data selection, type lakehouse_db as the database name. If it appears in the dropdown, select it. Otherwise, type it manually.
  • Leave table selection criteria as the default (include all tables).
  • Click Next.
  • Under Publishing settings, choose to make assets immediately discoverable in the business data catalog.
  • Under Run preference, select Run on demand.
  • Click Next, review your configuration, and then click Create.

Now run the data source to discover and publish your assets.

  • On the data source details page, click Run to start a scan.
  • Wait for the run status to show Succeeded. This typically takes 1-2 minutes.
  • Navigate to the Data tab, then click Inventory data in the left navigation.
  • Verify that both the orders table (CSV, raw zone) and orders_curated table (Parquet, curated zone) appear in your project inventory with their auto-discovered metadata.

What just happened?

DataZone scanned your Glue Data Catalog and imported the technical metadata (table names, schemas, partition keys, data formats) into your project inventory. Because you chose to make assets immediately discoverable, they are now published to the DataZone catalog.

Any user with access to your domain can now search for and find these assets, see their column definitions, and request access through DataZone's subscription workflow.

Confirm your assets are visible in the catalog by searching for them.

  • Click Catalog in the top navigation bar.
  • Type orders in the search bar and press Enter.
  • You should see both your orders and orders_curated assets listed with their schema details and metadata.

Assets not appearing in the catalog?

Check that your data source run completed successfully. Navigate to the data source and look at the run history. If it failed, verify that the environment has the correct IAM role with permissions to read from your Glue Data Catalog.

Also confirm you typed lakehouse_db exactly as your Glue database name. Database names are case-sensitive.

Help me troubleshoot my DataZone data source scan

Your lakehouse now has a complete governance layer. Data assets are discoverable through a central catalog, business terms standardize how teams interpret metrics, and the domain boundary controls who can access what.

Secret mission

Add Automated Pipeline Triggering with Lambda

Your ETL pipeline works, but you have to trigger it manually every time new data arrives. In a real enterprise, data lands continuously and pipelines need to react automatically. Can you wire up event-driven automation so your lakehouse processes new files the moment they arrive?

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. Most resources in this project are within free tier or have already completed their runs, but Redshift Serverless uses your $300 free trial credit.

Cost warning

Redshift Serverless scales to zero when idle, meaning you won't be charged while it's not processing queries. However, the workgroup remains active and counts against your 90-day free trial window.

If you want to conserve your $300 free trial credit for future use, choose the Pause option below to delete the Redshift workgroup and namespace.

Resources you used:

  • S3 bucket with raw/, curated/, and consumption/ zones (minimal storage cost, free tier covers 5GB).
  • AWS Glue database, crawlers, and ETL job (already ran and completed, no ongoing cost).
  • Redshift Serverless workgroup and namespace (uses $300 free trial credit, scales to zero when idle).
  • Amazon DataZone domain, project, and glossary (within free tier at 20MB metadata and 4,000 requests).
  • IAM roles (GlueLakehouseRole, RedshiftS3ReadRole).
  • Lambda function and S3 event notification (if you completed the Secret Mission).

Keep everything running

No action needed. Choose this if you're still actively preparing for interviews or want to revisit the project.

  • Redshift Serverless scales to zero when not queried, so you won't be charged while idle.
  • DataZone and the Glue Data Catalog stay within free tier limits.
  • S3 storage for your sample data is pennies per month.
  • Your free trial credit is valid for 90 days from first Redshift Serverless use.

Pause - I'll come back to this later

Delete Redshift Serverless to conserve your free trial credits for later. Keep everything else since it's within free tier.

  • In the Redshift console, go to Serverless dashboard.
  • Select your workgroup, then click Delete.
  • Select your namespace, then click Delete.
  • Leave your S3 bucket, Glue resources, and DataZone domain intact. They cost nothing or pennies while idle.

Delete - I don't want to use this again

Remove all project resources and return your AWS account to a clean state.

  • Delete the Redshift Serverless workgroup and namespace from the Redshift console.
  • In the DataZone console, delete your domain (this removes the project, glossary, and published assets).
  • In the Glue console, delete the ETL job, both crawlers, and the lakehouse_db database.
  • Empty and delete your S3 lake bucket using the S3 console or run:
aws s3 rb s3://[[BUCKET_NAME="your-lakehouse-bucket-name"]] --force
  • In the IAM console, delete the GlueLakehouseRole and RedshiftS3ReadRole roles.
  • If you completed the Secret Mission, delete the Lambda function and its execution role from the Lambda and IAM consoles.

That's a Wrap!

That's a Wrap!

Nice work! You just built a fully functional enterprise data lakehouse on AWS from scratch. Not a diagram or a whiteboard sketch. A working platform with data flowing from raw ingestion through transformation to a governed, queryable warehouse.

Here's what you accomplished:

  • Built a 3-zone S3 data lake (raw/curated/consumption), wrote a PySpark Glue ETL job to transform CSV into partitioned Parquet, and discovered schemas automatically with Glue Crawlers.
  • Designed a star schema in Redshift Serverless with dimension and fact tables, and queried the lake layer directly with Athena to demonstrate the lakehouse pattern.
  • Established enterprise data governance with Amazon DataZone, including a domain, business glossary, and published data assets.
  • Added event-driven pipeline automation with a Lambda function that triggers ETL jobs when new data lands in S3.

Ready to quiz yourself?