Build a Serverless Data Lake on AWS

Introduction

30 Second Summary

Every time you check airline statistics or search for flight delays, you're seeing the output of a data pipeline. Raw records get ingested, cleaned, transformed into an efficient format, and made available for fast querying.

In this project, you will build a serverless data lake on AWS that ingests raw flight data from a public dataset, catalogs it automatically, transforms it to an optimized columnar format, and lets you run analytical SQL queries against it.

What You'll Build

A complete serverless data lake pipeline using Amazon S3, AWS Glue, and Amazon Athena that transforms raw CSV flight data into optimized Parquet format for analytical querying.

By the end of this project, you'll have:

• An S3 bucket with separate raw and transformed data zones, forming the storage layer of your data lake.
• A Glue Data Catalog with automatically inferred schemas and partition keys discovered by crawlers.
• A Glue ETL job that converts CSV to Snappy-compressed Parquet, demonstrating 60-90% query cost reduction in Athena.
• Secret Mission: Add data quality validation to your pipeline using Glue's data quality rules.

Are there any prerequisites?

You need an AWS account (paid plan), the AWS CLI installed and configured, and basic familiarity with Python and SQL. This project costs approximately $1-$2 in AWS charges for Glue crawlers and ETL jobs.

Before We Start

Before We Start...

Set Up Your AWS Environment

Before you can build a data lake, you need the foundation in place. That means verifying your AWS CLI connection, creating the storage layer (an S3 bucket), and granting AWS Glue the permissions it needs to read and write data on your behalf.

Think of this step as building the plumbing. Every piece you set up here gets used directly in Steps 2 through 4.

In this step, get ready to:

• Verify your AWS CLI is configured and connected to your account.
• Create an S3 bucket with separate zones for raw and transformed data.
• Create an IAM role that AWS Glue will use to access your data.

Verify your AWS CLI configuration

The AWS CLI is your command-line interface to all AWS services. You need it configured with credentials that can create resources in your account.

• Open your terminal.
• Confirm your CLI is connected to your AWS account by running this command:

aws sts get-caller-identity

What does this command do?

`aws sts get-caller-identity` asks AWS to confirm who you are. It returns your Account ID, User ARN, and User ID. If you see those three values, your CLI is authenticated.

✔️ I see my account info

You should see output with your Account, UserId, and Arn fields. Your CLI is ready to go.

• Confirm your region is set to us-east-1 by running:

aws configure get region

If it returns us-east-1, you are all set. If it returns a different region, update it by running:

aws configure set region us-east-1

Why us-east-1?

The public flight dataset you will use in Step 2 lives in an S3 bucket in us-east-1. Keeping your resources in the same region avoids cross-region data transfer costs and latency.

ⓧ I see an error or nothing happens

Your CLI is not configured yet. You need to provide your AWS access credentials.

• Run the configuration wizard:

aws configure

• Enter your Access Key ID when prompted.
• Enter your Secret Access Key when prompted.
• Enter us-east-1 for the default region.
• Press Enter to accept json as the default output format.

Where to find your access keys

In the AWS Management Console, navigate to IAM, then Users, select your user, and open the Security credentials tab. Create a new access key if you do not have one.

• After configuring, verify it worked by running aws sts get-caller-identity again.

Create your S3 bucket

Your data lake needs a storage layer. Amazon S3 provides the object storage where raw data lands and transformed data gets written.

• Create your bucket by running this command (replace the placeholder with your own unique name):

aws s3 mb s3://[[BUCKET_NAME="data-lake-flights-abc-20260616"]] --region us-east-1

What does this command do?

`aws s3 mb` stands for "make bucket". The --region us-east-1 flag ensures the bucket is created in the same region as the public flight dataset.

S3 bucket names must be globally unique across all AWS accounts. Use the pattern data-lake-flights-<your-initials>-<date> to make yours unique. For example: data-lake-flights-jd-20260616.

• Create the raw/ prefix (folder) for incoming data:

aws s3api put-object --bucket [[BUCKET_NAME="data-lake-flights-abc-20260616"]] --key raw/

• Create the transformed/ prefix for processed data:

aws s3api put-object --bucket [[BUCKET_NAME="data-lake-flights-abc-20260616"]] --key transformed/

Why two prefixes?

A well-architected data lake separates raw data from transformed data. The raw/ zone holds data exactly as it arrived. The transformed/ zone holds data that has been cleaned, typed, and converted to an optimized format.

• Verify your bucket and prefixes exist by running:

aws s3 ls s3://[[BUCKET_NAME="data-lake-flights-abc-20260616"]]/

You should see raw/ and transformed/ listed in the output.

Bucket name already exists?

S3 bucket names are globally unique. If you see a "BucketAlreadyExists" error, try a different combination of initials and date. Make sure your name uses only lowercase letters, numbers, and hyphens.

help me fix my S3 bucket creation error

Create an IAM role for AWS Glue

AWS Glue needs an IAM role to access your S3 bucket and the public flights dataset. This role defines exactly what Glue is allowed to do in your account.

• Create a trust policy file that lets AWS Glue assume the role. Run this command:

cat > trust-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "glue.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

What is a trust policy?

A trust policy tells AWS which service is allowed to "wear" this role. Here, you are telling AWS that the Glue service (glue.amazonaws.com) can assume this role and use its permissions.

• Create the IAM role using the trust policy by running:

aws iam create-role \
  --role-name AWSGlueServiceRole-DataLake \
  --assume-role-policy-document file://trust-policy.json

• Attach the AWSGlueServiceRole managed policy to give Glue its baseline permissions:

aws iam attach-role-policy \
  --role-name AWSGlueServiceRole-DataLake \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole

What does AWSGlueServiceRole include?

This AWS-managed policy grants Glue baseline permissions: access to the Glue Data Catalog, CloudWatch logging, EC2 networking, and read access to public crawler buckets (like the flight dataset you will use). It does not grant access to your personal S3 bucket, which is why you need one more policy.

• Create an inline policy file that grants S3 read/write access to your bucket. Run this command (use the same bucket name you created earlier):

cat > s3-access-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::[[BUCKET_NAME="data-lake-flights-abc-20260616"]]",
        "arn:aws:s3:::[[BUCKET_NAME="data-lake-flights-abc-20260616"]]/*"
      ]
    }
  ]
}
EOF

• Attach this inline policy to the role by running:

aws iam put-role-policy \
  --role-name AWSGlueServiceRole-DataLake \
  --policy-name S3DataLakeAccess \
  --policy-document file://s3-access-policy.json

Why an inline policy on top of the managed policy?

The AWSGlueServiceRole managed policy only grants S3 access to buckets named aws-glue-* and the public crawler-public* buckets. Your personal data lake bucket has a custom name, so you need an additional policy that explicitly grants read/write access to it.

• Verify the role exists and has both policies attached by running:

aws iam list-attached-role-policies --role-name AWSGlueServiceRole-DataLake

You should see AWSGlueServiceRole listed in the output.

• Also verify the inline policy by running:

aws iam list-role-policies --role-name AWSGlueServiceRole-DataLake

You should see S3DataLakeAccess listed.

Getting an access denied error?

Your IAM user needs permission to create roles and attach policies. If you see "AccessDenied", check that your user has the IAMFullAccess managed policy or equivalent permissions attached.

help me troubleshoot my IAM permissions error

Your foundation is in place. You have a verified CLI connection, an S3 bucket with organized data zones, and an IAM role ready for Glue to use. Next up, you will point a Glue crawler at real flight data and watch it automatically discover the schema.

Crawl and Catalog Raw Data

Your AWS environment is ready with an S3 bucket and an IAM role. Now it's time to bring in the data and make it discoverable.

In this step, you'll use AWS Glue to automatically discover the schema of a public flights dataset stored in Amazon S3. A Glue crawler scans your data, infers its structure, and registers it in the Data Catalog so that other services can query it without manual schema definitions.

In this step, get ready to:

• Create a Glue Data Catalog database to organize your table metadata.
• Configure and run a crawler that infers the schema of a public CSV dataset.
• Verify the automatically cataloged table, its columns, and partition keys.

Create a Glue Data Catalog database

Before your crawler can write table metadata, it needs a database to write into. A database in the Glue Data Catalog is a logical container that groups related tables together.

• In the AWS Management Console, search for Glue and open the AWS Glue service.
• In the left navigation pane, under Data catalog, click Databases.
• Click Add database.
• In the Name field, enter flights_data_lake.
• Leave the Location field empty.
• Click Create database.

Why use only alphanumeric characters and underscores?

If you plan to query from Amazon Athena (which you will later in this project), database names must contain only alphanumeric characters and underscores. That's why the name is flights_data_lake instead of something with hyphens.

Your new database now appears in the Databases list. You can confirm by clicking on the database name to see its (currently empty) properties.

Create and run the CSV crawler

A crawler connects to a data store, uses classifiers to determine the format and schema of your data, and writes the resulting metadata to the Data Catalog. You'll point your crawler at a public S3 bucket that contains US flight data in CSV format.

• In the left navigation pane, click Crawlers.
• Click Create crawler.

This opens a multi-step wizard. Follow each step below.

• In Step 1: Set crawler properties, enter flights-csv-crawler in the Name field.
• Click Next.
• In Step 2: Choose data sources and classifiers, click Add a data source.
• For the data source, select S3.
• For Location of S3 data, choose In a different account.
• In the S3 path field, enter s3://crawler-public-us-east-1/flight/2016/csv.
• Click Add an S3 data source to confirm, then click Next.
• In Step 3: Configure security settings, select the existing IAM role AWSGlueServiceRole-DataLake from the dropdown.
• Click Next.
• In Step 4: Set output and scheduling, select flights_data_lake as the target database.
• For Crawler schedule, select On demand.
• Click Next.
• In Step 5: Review and create, review your settings and click Create crawler.

What is a classifier?

Glue uses classifiers to determine how to read your data. The crawler tries each classifier in order until one recognizes the format. For CSV files, Glue's built-in CSV classifier is used automatically. You don't need to configure one manually.

Now run the crawler to scan the public flights data.

• On the Crawlers page, select the checkbox next to flights-csv-crawler.
• Click Run.
• Wait for the crawler status to change to Completed. This takes approximately 1-2 minutes.

Refresh the status

The status doesn't auto-update. Click the refresh icon in the Crawlers table to see the latest status. When the crawler finishes, the Tables changes column shows the number of tables created.

Crawler showing Failed status?

The most common cause is insufficient IAM permissions. Confirm that your AWSGlueServiceRole-DataLake role has read access to s3://crawler-public-us-east-1/. The role also needs the AWSGlueServiceRole managed policy attached so it can write to the Data Catalog.

If you're in a region other than us-east-1, the crawler may be unable to reach the public bucket. Switch your console region to us-east-1 and try again.

Still stuck? Help me troubleshoot my failed Glue crawler run

Verify the cataloged table

The crawler has now read the CSV data, inferred the column names and data types, and registered a table in your Data Catalog. Time to inspect the results.

• In the left navigation pane, under Data catalog, click Tables.
• You should see a table named csv in the flights_data_lake database.
• Click the table name csv to open its detail page.
• Scroll down to the Schema section. You should see columns such as year, quarter, carrier, and dep_delay with their inferred data types.
• Note the Classification field at the top of the page. It should show csv.
• Click View partitions to see how the data is partitioned. The partition key is mon (representing the month of the flight data).

What just happened behind the scenes?

The crawler scanned the folder structure at s3://crawler-public-us-east-1/flight/2016/csv and found multiple subfolders (one per month). It recognized the CSV format using its built-in classifier, inferred column names from the CSV headers, and detected that the subfolder structure represents a partition key called mon.

All of this metadata is now stored in the Data Catalog. The catalog doesn't contain the actual data. It contains a blueprint of where the data lives and what it looks like. Any service (Athena, Glue ETL jobs, Redshift Spectrum) can use this blueprint to query the data directly from S3.

Your raw data is now fully cataloged and discoverable. Next up, you'll build an ETL job that transforms this CSV data into optimized Parquet format for faster, cheaper queries.

Build and Run the ETL Job

Your raw flight data is now cataloged in the Data Catalog, but it is still stored as CSV. CSV files are row-oriented, meaning any query that only needs a few columns still has to read every column in every row.

In this step, you will build an ETL job in AWS Glue Studio that transforms the CSV data into Parquet format with Snappy compression. Parquet is columnar, so analytical queries only read the columns they need. This single transformation can reduce query costs by 60-90%.

In this step, get ready to:

• Create a visual ETL job that reads from the Data Catalog and writes optimized Parquet to S3.
• Configure the job for cost safety and run it.
• Catalog the transformed Parquet output with a new crawler.

Create the visual ETL job in Glue Studio

AWS Glue Studio provides a drag-and-drop canvas where each node represents a step in your data pipeline: read, transform, write.

• In the AWS Glue console, expand Data Integration and ETL in the left navigation pane.
• Select ETL jobs.
• Under the Create job section, select Visual ETL.

The visual editor canvas opens with no nodes. You will now add a source, a transform, and a target.

• Click the + icon to add a node. Search for and select AWS Glue Data Catalog as your data source.
• In the source node's Data source properties panel on the right, set the Database to flights_data_lake.
• Set the Table to csv (the table created by your crawler in the previous step).

What is the Data Catalog source doing here?

Instead of pointing your job at a raw S3 path, you point it at the Data Catalog table. This means Glue already knows the schema, partitions, and file locations. No extra configuration needed.

• With the source node selected, click the + icon again and search for Change Schema under Transforms. Select it.
• Select the Change Schema node, then open the Transform tab in the node properties panel on the right.
• Find the year column and change its Data type to int.
• Find the quarter column and change its Data type to int.
• Find the dep_delay column and change its Data type to double.

Why cast these columns?

The crawler inferred all columns as strings because CSV has no embedded type information. Casting numeric columns to their proper types (int, double) means Parquet stores them efficiently and Athena can perform mathematical operations without runtime casting.

• With the Change Schema node selected, click the + icon and search for Amazon S3. Select Amazon S3 as the target.
• In the target node's Data target properties panel, set Format to Parquet.
• Set Compression Type to Snappy.
• For the S3 Target Location, click Browse S3 and navigate to your bucket's transformed/ prefix.

Why Snappy compression?

Snappy provides a good balance between compression ratio and decompression speed. It is the default compression for Parquet in AWS Glue. Your data will be smaller on disk (cheaper storage) and faster to read (cheaper queries in Athena).

Don't see your bucket in Browse S3?

Make sure your IAM role AWSGlueServiceRole-DataLake has S3 permissions for your bucket. If the role is missing bucket access, the browser will not display it.

help me troubleshoot why my S3 bucket is not showing up in the Glue Studio target browser

Configure job settings and run the job

Before running the job, you need to set resource limits to prevent runaway costs. Glue charges $0.44 per DPU-hour, so capping workers and adding a timeout keeps your bill predictable.

• Click the Job details tab at the top of the editor.
• In the Name field, enter flights-csv-to-parquet.
• For IAM Role, select AWSGlueServiceRole-DataLake (the role you created in Step 1).
• Set Worker type to G.1X.
• Set Requested number of workers to 2.
• Set Job timeout (minutes) to 10.
• Click Save in the top right corner.
• Click Run to start the job.

What is a G.1X worker?

Each G.1X worker maps to 1 DPU (4 vCPUs, 16 GB memory) with 94 GB disk. Two workers is the minimum for a PySpark job. At $0.44 per DPU-hour, this job (running a few minutes) will cost well under $1.

Seeing an access denied error on save?

Confirm that AWSGlueServiceRole-DataLake has the AWSGlueServiceRole managed policy attached. This policy grants Glue permission to write the auto-generated script to S3.

help me fix an access denied error when saving my AWS Glue ETL job

Monitor the job and catalog the Parquet output

The job typically takes 2-4 minutes to complete. While it runs, Glue provisions your workers, executes the Spark transformation, and writes Parquet files to your S3 bucket.

• Click the Runs tab at the top of the visual editor to monitor progress.
• Wait until the Run status shows Succeeded.

What if the job shows Running for more than 5 minutes?

The first minute or so is Glue provisioning workers (cold start). If the job exceeds your 10-minute timeout, it will automatically stop. Check the logs in the Runs tab for any data format issues.

• Open the Amazon S3 console and navigate to your bucket's transformed/ prefix.
• Confirm that Parquet files (.parquet extension) are now listed there.

The Parquet files exist in S3, but Athena does not know about them yet. You need to run a second crawler to catalog this new data layer.

• Return to the AWS Glue console. In the left navigation pane under Data Catalog, select Crawlers.
• Click Create crawler.
• Name the crawler flights-parquet-crawler.
• For the data source, choose S3 and set the path to your bucket's transformed/ prefix.
• Select the IAM role AWSGlueServiceRole-DataLake.
• Set the target database to flights_data_lake.
• Complete the wizard and click Create crawler.
• Select the new crawler and click Run.
• Wait for the crawler to complete (approximately 1-2 minutes).

• Navigate to Tables under the flights_data_lake database in the Data Catalog.
• Confirm a new table appears with Classification showing parquet.

Crawler succeeded but no new table appeared?

Make sure the crawler's S3 path points to the transformed/ prefix that contains the Parquet files. If the prefix is empty (the ETL job failed or wrote to a different location), the crawler will find nothing to catalog.

help me figure out why my Glue crawler did not create a table after running successfully

Your ETL pipeline is complete. You have raw CSV data flowing through a Glue job that produces optimized Parquet, and both layers are cataloged for querying. Next up, you will use Amazon Athena to run SQL analytics against your Parquet data and see the performance difference firsthand.

Query with Amazon Athena

Your data has traveled from raw CSV through a Glue ETL job and landed as optimized Parquet in S3. The final piece of any data lake is the ability to run analytical queries directly against that data without provisioning servers or loading it into a separate database.

In this step, you will use Amazon Athena to run SQL queries against both your Parquet and CSV tables. You will then compare how much data each query scans to see the real-world cost and performance benefits of columnar storage.

In this step, get ready to:

• Configure Athena and run a test query against your Parquet table.
• Run an analytical query calculating average departure delays by carrier and month.
• Compare data scanned between Parquet and CSV to prove the pipeline's value.

Open Athena and configure your query result location

Before you can run any queries, Athena needs a place to store query results. You will point it to the athena-results/ prefix in your S3 bucket.

• In the AWS Management Console, search for Athena in the top search bar and open the Athena service.
• If this is your first time visiting Athena, click Explore the query editor to open the query editor.
• Click Edit Settings (or Settings then Manage if you have used Athena before).
• In the Location of query result box, enter your S3 path:

s3://[[DATALAKE_BUCKET="data-lake-flights-your-initials-date"]]/athena-results/

• Click Save.
• Click Editor to return to the query editor.
• In the left panel, set Data source to AwsDataCatalog.
• From the Database dropdown below it, select flights_data_lake.

What is Amazon Athena?

Athena is a serverless query engine that lets you run standard SQL against data stored in S3. You pay only for the data scanned per query ($5 per TB). There are no clusters to manage and no infrastructure to provision.

Because Athena reads directly from S3 using the schema defined in the Glue Data Catalog, the tables you cataloged in the previous steps are immediately queryable here.

Now run a test query to confirm everything is connected. Enter the following SQL in the query editor:

SELECT * FROM transformed LIMIT 10;

• Click Run or press Ctrl+Enter.

You should see 10 rows of flight data appear in the results pane below the editor. Columns include year, quarter, carrier, dep_delay, and others from your transformed Parquet data.

Not seeing any results?

• Make sure you selected flights_data_lake from the Database dropdown. If you query against the wrong database, the table will not be found.
• Confirm the table name matches what your parquet crawler created. Check the Tables list in the left panel to see the exact name.
• If you see a permissions error, verify that your IAM user has s3:GetObject access to both your data lake bucket and the athena-results prefix.

Help me troubleshoot my Athena query that is returning no results or an error.

Run an analytical query on your Parquet data

A simple SELECT proves connectivity. Now you will run a real analytical query that demonstrates the business value your pipeline provides. This query calculates the average departure delay grouped by airline carrier and month.

• Clear the query editor and enter the following SQL:

SELECT carrier, mon, AVG(dep_delay) as avg_delay
FROM transformed
WHERE dep_delay IS NOT NULL
GROUP BY carrier, mon
ORDER BY avg_delay DESC
LIMIT 20;

• Click Run.

What does this query do?

• It filters out rows where dep_delay is NULL so the average calculation is accurate.
• It groups every flight record by carrier (airline code) and mon (month number).
• It calculates the average departure delay for each carrier-month combination.
• It orders the results by worst delays first, showing the top 20.

You should see 20 rows showing carrier codes, month numbers, and average delay values. Note the Data scanned value shown at the bottom of the query results. Write this number down. You will compare it against the CSV table next.

Reading the data scanned value

Athena displays the amount of data scanned beneath each query result. This number directly determines your cost at $5 per TB. Smaller scans mean faster queries and lower bills.

Compare query performance between Parquet and CSV

The whole point of your ETL pipeline was to optimize data for analytical queries. Now you will prove that optimization by running the identical query against the original CSV table and comparing how much data Athena needs to scan.

• Clear the query editor and enter this query targeting the CSV table:

SELECT carrier, mon, AVG(dep_delay) as avg_delay
FROM csv
WHERE dep_delay IS NOT NULL
GROUP BY carrier, mon
ORDER BY avg_delay DESC
LIMIT 20;

• Click Run.

Once the results appear, compare the Data scanned value for this CSV query against the value you noted for the Parquet query.

Why does Parquet scan less data?

CSV is a row-based format. To read just three columns (carrier, mon, dep_delay), Athena must scan every row in the entire file. Every column of every row passes through the engine.

Parquet is a columnar format. Athena reads only the columns your query references. Combined with Snappy compression, the Parquet table typically requires 60-90% less data to be scanned for the same analytical result.

At $5 per TB scanned, this difference translates directly into cost savings at production scale.

Both queries scanning similar amounts?

• Confirm you are querying the correct tables. The Parquet query uses FROM transformed and the CSV query uses FROM csv.
• Check the table classification in the Glue Data Catalog. Navigate to AWS Glue and verify that the transformed table shows classification parquet and the csv table shows classification csv.
• If your ETL job did not apply compression or output to the wrong prefix, re-run the job from Step 3.

Help me figure out why my Parquet and CSV queries are scanning similar amounts of data in Athena.

You have just proven the value of your entire pipeline. Raw CSV data entered your data lake, Glue discovered its schema, an ETL job transformed it to Parquet, and Athena now queries it at a fraction of the cost. Next up, you can take on the Secret Mission to add data quality validation to your pipeline.

Secret mission

Add Data Quality Validation

Your pipeline works, but how do you know the data flowing through it is trustworthy? In production, bad data slips in silently. Add automated quality gates that validate every row before it reaches your analytics layer.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project uses serverless AWS services that only charge when you actively run crawlers, jobs, or queries. S3 storage costs are negligible for this data volume (pennies per month).

Cost warning

AWS Glue crawlers and ETL jobs charge $0.44 per DPU-hour each time they run. Athena charges $5 per TB scanned per query. Your total project cost is approximately $1-$2. If you leave these resources in place but do not re-run anything, no additional charges accrue beyond minimal S3 storage.

If you created your AWS account after July 15, 2025, your $200 in free credits likely covers this project. If you are on an older account, the $1-$2 cost has already been incurred. Either way, deleting the S3 bucket prevents future storage charges.

Resources you used:

• S3 bucket data-lake-flights-<initials>-<date> with raw/, transformed/, athena-results/, and quarantine/ prefixes.
• AWS Glue Data Catalog database flights_data_lake with two tables (csv and parquet).
• AWS Glue crawlers: flights-csv-crawler and flights-parquet-crawler.
• AWS Glue ETL job: flights-csv-to-parquet.
• IAM role: AWSGlueServiceRole-DataLake.

Keep everything running

No action needed. Choose this if you plan to continue exploring your data lake, adding new data sources, or experimenting with additional Glue transforms.

• All services are serverless. You only pay when you actively run a crawler, ETL job, or Athena query.
• S3 storage for the Parquet files and Athena results is negligible (a few cents per month at most).
• The Glue Data Catalog is free for up to 1 million stored objects and 1 million requests per month.

Pause - I'll come back to this later

No special action is required to pause. AWS Glue crawlers, ETL jobs, and Athena are fully serverless and only incur charges when you trigger them. Your data stays in S3 at minimal storage cost until you return.

• Your S3 bucket, Glue catalog, and IAM role remain intact.
• No crawlers or jobs will run unless you manually trigger them.
• When you come back, everything is ready to pick up where you left off.

Delete - I don't want to use this again

Remove all project resources to avoid any future charges and clean up your AWS account.

• Delete the S3 bucket and all its contents by running this command:

aws s3 rb s3://[[DATALAKE_BUCKET="data-lake-flights-<initials>-<date>"]] --force

• Delete the Glue ETL job by running:

aws glue delete-job --job-name flights-csv-to-parquet

• Delete both Glue crawlers by running:

aws glue delete-crawler --name flights-csv-crawler
aws glue delete-crawler --name flights-parquet-crawler

• Delete the Glue Data Catalog database (this also removes all tables within it) by running:

aws glue delete-database --name flights_data_lake

• Detach the managed policy from the IAM role, then delete the role. Run these commands:

aws iam detach-role-policy --role-name AWSGlueServiceRole-DataLake --policy-arn arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole
aws iam delete-role --role-name AWSGlueServiceRole-DataLake

What about inline policies?

If you added an inline policy to the role for S3 access (rather than attaching a second managed policy), you need to delete it before the role can be removed. List inline policies with:

aws iam list-role-policies --role-name AWSGlueServiceRole-DataLake

For each policy name returned, delete it with:

aws iam delete-role-policy --role-name AWSGlueServiceRole-DataLake --policy-name [[INLINE_POLICY_NAME="the policy name from the list above"]]

Nice Work!

Nice Work!

You just built a production-style serverless data lake on AWS. From raw CSV ingestion to optimized columnar querying, you now have hands-on proof that you can architect cloud-native data pipelines end to end.

You've learned how to:

• Built a serverless data lake architecture using S3, AWS Glue crawlers, the Data Catalog, and Amazon Athena.
• Implemented an ETL pipeline that transforms raw CSV into optimized Parquet with Snappy compression.
• Ran analytical SQL queries in Athena and demonstrated 60-90% data scan reduction with columnar formats over raw CSV.
• Secret Mission: Added data quality validation to your pipeline, routing bad data to a quarantine zone for production-grade observability.

Ready to quiz yourself?