Serverless Data Pipeline on AWS

Build an ETL pipeline that transforms JSON data into Parquet using S3, Glue, and Athena.

Introduction

30 Second Summary

Every time you order something online, that transaction becomes a piece of raw data sitting in a file somewhere, waiting to be turned into something useful like sales reports, inventory forecasts, or personalized recommendations.

In this project, you will build a serverless data pipeline on AWS that takes raw e-commerce order data, automatically discovers its structure, transforms it into an optimized format, and makes it queryable with SQL.

What You'll Build

A complete serverless data pipeline using Amazon S3, AWS Glue, and Amazon Athena that processes e-commerce order data from raw JSON files into analytics-ready Parquet format.

By the end of this project, you'll have:

  • An S3 data lake with organized raw and processed zones, following real-world data lake design patterns.
  • A Glue Data Catalog with automatically discovered table schemas, built by running a Glue Crawler against your data.
  • A Visual ETL job that converts JSON to Parquet with no code required.
  • SQL queries in Athena running against your processed data, plus an advanced analytics query in the Secret Mission.

Are there any prerequisites?

None. This guide walks you through AWS account creation and all setup from scratch. You just need a web browser and some comfort with SQL. New AWS accounts get up to $200 in free credits, which will cover this project entirely.

Create Your AWS Account and Set Up S3

Every data pipeline needs a place to store data. Before you can build your pipeline, you need an AWS account and an S3 bucket to act as your data lake. In this step, you'll create your AWS account and set up the storage layer that the rest of your pipeline depends on.

In This Step, Get Ready To:
  • Create a free AWS account and sign in to the AWS Management Console.
  • Create an S3 bucket to serve as your data lake.
  • Organize your bucket with raw, processed, and results folders.
Sign Up for an AWS Account

AWS offers a Free Plan for new accounts that lasts up to 6 months. You won't be charged unless you switch to a Paid Plan. New customers also get up to $200 in free credits, which will cover this entire project.

  • Go to https://aws.amazon.com/free and click Create a Free Account.
  • Choose the Free Plan.
  • Enter your email, create a password, and fill in your contact information.
  • Choose Personal for the account type.
  • Add a payment method when prompted. A credit card is required for verification, but you will not be charged for this project.
  • Verify your phone number when prompted.
  • Choose the Basic (Free) support plan.
  • Wait for your account to activate. This usually takes a few minutes.
  • Sign in to the AWS Management Console at https://console.aws.amazon.com.

What is the AWS Management Console?

The AWS Management Console is a browser-based dashboard for all AWS services. No software to install. You'll use it for every step of this project.

Account Not Activating?

AWS account activation can occasionally take longer than a few minutes. Check your email for a confirmation message from AWS. If your account still isn't active after 15 minutes, help me troubleshoot my AWS account activation.

Create Your S3 Data Lake Bucket

Amazon S3 (Simple Storage Service) is where your data lives. It works like a cloud-based folder system that can hold any type of file at any scale.

  • In the AWS Console, use the search bar at the top and type S3.
  • Click on S3 in the search results.
  • Click Create bucket.
  • For Bucket name, enter something unique like my-data-pipeline-your-name.
  • For AWS Region, choose US East (N. Virginia) us-east-1.
  • Leave all other settings as default.
  • Click Create bucket.

Why do I have to make my bucket name unique?

S3 bucket names must be globally unique across all AWS accounts. That means no two people in the world can have the same bucket name. Adding your name or some random numbers to the end ensures yours won't collide with anyone else's.

Why us-east-1?

The us-east-1 region (N. Virginia) has the widest service availability across all AWS services. Using it ensures every service you need in this project works without compatibility issues.

Create the Data Lake Folder Structure

A data lake organizes data into zones. You'll create three: raw for untouched incoming data, processed for cleaned and transformed data, and athena-results for query output. This raw/processed pattern is how real data engineers organize data at companies.

  • Click on your new bucket name to open it.
  • Click Create folder, name it raw, and click Create folder.
  • Click Create folder again, name it processed, and click Create folder.
  • Click Create folder one more time, name it athena-results, and click Create folder.

Why three folders?

The raw folder holds your original, untouched data. The processed folder holds data after your pipeline transforms it. Keeping them separate means you can always go back to the original data if something goes wrong.

The athena-results folder is where Athena stores query output. Athena requires a dedicated location for this.

Your storage layer is ready. You have a data lake with organized zones, following the same pattern used in production data platforms. Next up, you'll upload sample data and use a Glue Crawler to automatically discover its structure.

Upload Sample Data and Run a Glue Crawler

Your S3 data lake is set up with organized zones. Now it needs some data to work with.

In this step, you'll upload sample e-commerce data into your raw zone and then use an AWS Glue Crawler to automatically discover the data's structure. The Crawler figures out column names, data types, and how the data is organized. It saves all of this to the Glue Data Catalog, which acts like a table of contents for your entire data lake.

In This Step, Get Ready To:
  • Create and upload a sample JSON data file to your S3 raw zone.
  • Create an IAM role that gives AWS Glue permission to access your S3 data.
  • Run a Glue Crawler to automatically discover your data's schema.
Create and Upload Sample E-Commerce Data

Every data pipeline starts with source data. You'll create a small JSON file containing e-commerce orders and upload it to your S3 raw zone.

  • On your computer, open any text editor (Notepad on Windows, TextEdit on Mac, or VS Code).
  • Paste the following JSON data and save the file as orders.json:
{"order_id": "ORD001", "customer_name": "Alice Johnson", "product": "Wireless Headphones", "category": "Electronics", "quantity": 2, "price": 49.99, "order_date": "2025-03-15", "city": "New York"}
{"order_id": "ORD002", "customer_name": "Bob Smith", "product": "Running Shoes", "category": "Sports", "quantity": 1, "price": 89.99, "order_date": "2025-03-15", "city": "Los Angeles"}
{"order_id": "ORD003", "customer_name": "Carol Davis", "product": "Python Cookbook", "category": "Books", "quantity": 3, "price": 34.99, "order_date": "2025-03-16", "city": "Chicago"}
{"order_id": "ORD004", "customer_name": "David Lee", "product": "Yoga Mat", "category": "Sports", "quantity": 1, "price": 29.99, "order_date": "2025-03-16", "city": "Houston"}
{"order_id": "ORD005", "customer_name": "Eve Martinez", "product": "Bluetooth Speaker", "category": "Electronics", "quantity": 1, "price": 79.99, "order_date": "2025-03-17", "city": "Phoenix"}
{"order_id": "ORD006", "customer_name": "Frank Wilson", "product": "Desk Lamp", "category": "Home", "quantity": 2, "price": 24.99, "order_date": "2025-03-17", "city": "New York"}
{"order_id": "ORD007", "customer_name": "Grace Chen", "product": "Trail Runners", "category": "Sports", "quantity": 1, "price": 119.99, "order_date": "2025-03-18", "city": "San Francisco"}
{"order_id": "ORD008", "customer_name": "Henry Brown", "product": "USB-C Hub", "category": "Electronics", "quantity": 3, "price": 39.99, "order_date": "2025-03-18", "city": "Seattle"}

What is JSON Lines?

This file uses the JSON Lines format, where each line is a self-contained JSON object representing one record. It's a common format for data pipelines because tools can process each line independently without loading the entire file into memory.

  • Go back to the S3 console and click into your bucket.
  • Click into the raw folder.
  • Click Upload, then Add files.
  • Select your orders.json file and click Upload.

What should I see?

You should see a green success banner at the top of the page confirming the upload. Your orders.json file now appears inside the raw folder.

Create an IAM Role for AWS Glue

AWS Glue needs permission to access your S3 data. You grant this permission by creating an IAM role. Think of an IAM role as a permission badge that tells AWS "this service is allowed to do these things."

  • In the AWS Console search bar, type IAM and click on it.
  • In the left sidebar, click Roles, then click Create role.
  • For Trusted entity type, select AWS service.
  • For Use case, search for and select Glue, then click Next.
  • On the permissions page, search for AmazonS3FullAccess and check the box next to it.
  • Also search for AWSGlueServiceRole and check the box next to it.
  • Click Next.
  • For Role name, enter GlueDataPipelineRole.
  • Click Create role.

Why two permission policies?

The AmazonS3FullAccess policy lets Glue read and write data in your S3 bucket. The AWSGlueServiceRole policy lets Glue manage its own internal resources like the Data Catalog. In production, you would scope these permissions more tightly to specific S3 paths. For learning, full access keeps things simple.

Run a Glue Crawler to Discover Your Data Schema

A Glue Crawler scans your data and automatically figures out the column names, data types, and structure. It saves this information in the Glue Data Catalog, which is like a table of contents for all your data. Without this catalog, services like Athena wouldn't know how to read your files.

  • In the AWS Console search bar, type Glue and click on AWS Glue.
  • In the left sidebar, under Data Catalog, click Crawlers.
  • Click Create crawler.
  • For Name, enter raw-orders-crawler.
  • Click Next.
  • Click Add a data source.
  • For S3 path, click Browse S3 and navigate to your bucket. Select the raw folder.

The path should look like:

s3://[[BUCKET_NAME="my-data-pipeline-yourname"]]/raw/
  • Click Add an S3 data source.
  • Click Next.
  • For IAM role, select Choose an existing IAM role and pick GlueDataPipelineRole.
  • Click Next.
  • For Target database, click Add database.
  • In the popup, enter ecommerce_pipeline as the database name and click Create database.
  • Back on the crawler page, make sure ecommerce_pipeline is selected as the target database.
  • Click Next, review your settings, and click Create crawler.
  • On the crawler page, select your crawler and click Run.
  • Wait for the status to change to Completed. This takes about 1-2 minutes.

What is schema discovery?

A schema describes the structure of your data: what columns exist, what type each column is (string, integer, double), and how records are organized. Instead of defining this manually, the Crawler reads your files and figures it out automatically. This is called schema discovery.

After the crawler finishes, verify that it discovered your data correctly.

  • In the left sidebar, navigate to Data Catalog, then Databases.
  • Click on ecommerce_pipeline, then click Tables.
  • Click on the table called raw to see the schema the crawler discovered.

You should see all eight columns from your JSON data listed with their automatically detected data types. The crawler figured out that price is a double, quantity is an integer, and text fields like customer_name are strings. You didn't have to define any of this manually.

Don't see a table listed?

Make sure the crawler status shows Completed (not Running or Failed). If the crawler failed, check that the S3 path points to your raw folder and that the IAM role was created correctly. Help me troubleshoot my Glue Crawler.

Your data lake now has raw data and a catalog that describes it. Next up, you'll build a Visual ETL job that transforms this raw JSON into optimized Parquet format, making it faster and cheaper to query.

Transform Your Data with Glue Visual ETL

Your Glue Crawler just cataloged the raw data, so AWS now knows exactly what your JSON looks like. The next challenge is transforming that raw data into a format optimized for analytics. In this step, you'll use Glue Visual ETL to convert your JSON data into Parquet format. Parquet is a columnar format that is much faster and cheaper to query than JSON.

In This Step, Get Ready To:
  • Build a Visual ETL job that reads from the Glue Data Catalog and writes to S3.
  • Configure the job to output data in Parquet format with Snappy compression.
  • Run the ETL job and verify the transformed files in your S3 bucket.
Create a Visual ETL Job

Glue's Visual ETL editor is a drag-and-drop canvas where you connect three types of nodes: a Source (where data comes from), a Transform (what changes to make), and a Target (where data goes). Behind the scenes, the canvas generates PySpark code for you.

  • In the AWS Glue console, click ETL jobs in the left sidebar.
  • Under Create job, select Visual ETL.

This opens the visual canvas. You'll see a blank diagram where you can drag and drop nodes.

Why Visual ETL Instead of Writing Code?

Glue also supports scripted jobs where you write PySpark code directly. Visual ETL lets you focus on understanding the pipeline architecture and data flow instead of debugging code syntax. The visual canvas generates clean PySpark code behind the scenes, so you can inspect it later if you're curious.

Now you'll add three nodes to the canvas: a source, a transform, and a target.

Add a Source node:

  • Click the + (plus) icon or the Source dropdown at the top of the canvas.
  • Choose AWS Glue Data Catalog.
  • In the right-side panel, set Database to ecommerce_pipeline.
  • Set Table to raw.

Add a Transform node:

  • Click the + icon or the Transform dropdown.
  • Choose Change Schema (also called ApplyMapping).
  • This node connects to your source automatically. In the right-side panel, you can see all the columns from your data. You don't need to change anything here for now.

What Does Change Schema Do?

The Change Schema node lets you rename columns, change data types, or drop columns you don't need. For this project, the schema is already clean, so this node simply passes the data through. In production pipelines, this is where you'd clean up messy column names or cast strings to numbers.

Add a Target node:

  • Click the + icon or the Target dropdown.
  • Choose Amazon S3.
  • In the right-side panel, set Format to Parquet.
  • Set Compression Type to Snappy.
  • For S3 Target Location, click Browse S3 and select your bucket's processed folder.

The path should look like s3://my-data-pipeline-yourname/processed/.

Why Parquet with Snappy Compression?

Parquet is a columnar format, meaning queries only read the columns they need instead of scanning every field in every row. Athena charges $5.00 per TB of data scanned, so reading fewer columns directly saves money. Snappy compression makes the files smaller and faster to read without adding significant processing overhead.

Configure and Run the ETL Job

Before running the job, you need to configure its resource settings. Glue ETL jobs run on managed compute workers that you pay for by the second.

  • Click the Job details tab at the top of the canvas.
  • For Name, replace "Untitled job" with orders-json-to-parquet.
  • For IAM Role, select GlueDataPipelineRole.
  • For Glue version, keep the default (Glue 4.0).
  • For Worker type, select G 1X (the smallest option).
  • For Requested number of workers, set to 2 (the minimum).
  • Click Save in the top right corner.
  • Click Run to start the job.

What Do These Worker Settings Mean?

Glue ETL jobs run on managed compute units called DPUs (Data Processing Units). Each G 1X worker provides 1 DPU. Glue charges $0.44 per DPU-hour, billed per second. With 2 workers running for about 5 minutes, this job costs roughly $0.07.

The job takes about 3-5 minutes to complete. You can monitor progress in the Runs tab. Wait for the status to show Succeeded.

Job Status Shows Failed?

The most common cause is an incorrect IAM role. Make sure you selected GlueDataPipelineRole and that the role has both the AmazonS3FullAccess and AWSGlueServiceRole policies attached. Also double-check that the S3 target path points to your processed folder. Help me debug my failed Glue job.

Verify the Transformed Data

Once the job succeeds, your raw JSON data has been transformed into optimized Parquet files. Time to confirm the output landed in the right place.

  • Go back to S3 in the AWS Console.
  • Navigate to your bucket and click into the processed folder.

You should see one or more .parquet files. These are your transformed data files, stored in columnar format and compressed with Snappy.

What Should I See?

You'll see files with names like part-00000-xxxxxxxx.snappy.parquet. Glue splits output into parts automatically. For your small dataset, you'll likely see just one or two files. The file size will be much smaller than the original JSON because Parquet is a highly efficient format.

Your ETL pipeline is complete. Raw JSON went in, and optimized Parquet came out. Next up, you'll catalog this processed data and run SQL queries against it using Athena.

Catalog the Processed Data and Query with Athena

Your ETL job successfully transformed raw JSON into optimized Parquet files sitting in your processed folder. But right now, no service knows what those files contain. You need to catalog the processed data so Athena can read it and let you run SQL queries against it. In this step, you will run a second Glue Crawler to catalog the Parquet data, then query it with Amazon Athena to run real analytics.

In This Step, Get Ready To:
  • Run a second Glue Crawler to catalog your processed Parquet data.
  • Set up Amazon Athena and run your first SQL query against the processed data.
  • Run analytics queries to calculate revenue and order counts from your pipeline data.
Run a Second Crawler for the Processed Data

Your Glue Data Catalog currently has a table for the raw JSON data, but it has no idea about the Parquet files your ETL job just created. A second crawler will scan the processed folder and register a new table so Athena can query it.

  • Navigate to AWS Glue in the AWS Console.
  • In the left sidebar, under Data Catalog, click Crawlers.
  • Click Create crawler.
  • For Name, enter processed-orders-crawler.
  • Click Next.
  • Click Add a data source.
  • For S3 path, click Browse S3 and select the processed folder in your bucket.
  • Click Add an S3 data source, then click Next.
  • For IAM role, select Choose an existing IAM role and pick GlueDataPipelineRole.
  • Click Next.
  • For Target database, select the existing ecommerce_pipeline database.
  • Click Next, review your settings, and click Create crawler.
  • Select your crawler and click Run.
  • Wait for the status to change to Completed (about 1-2 minutes).

After the crawler finishes, navigate to Data Catalog > Databases > ecommerce_pipeline > Tables in the left sidebar.

What should I see?

You should now see two tables: raw (your original JSON data) and processed (your Parquet data). Click on the processed table to confirm the crawler discovered the same columns as the raw table, but now backed by Parquet files.

Why do I need a second crawler?

The first crawler cataloged your raw JSON files. The ETL job created brand new Parquet files in a different folder. The Glue Data Catalog does not automatically detect new files in new locations. Running a second crawler tells the Data Catalog about the processed data so that query engines like Athena know where to find it and how to read it.

Set Up Athena and Run Your First Query

Amazon Athena is a serverless query engine that lets you run SQL directly on data stored in S3. There is no database server to set up or manage. You just point Athena at your Data Catalog table and start writing queries.

  • In the AWS Console search bar, type Athena and click on it.
  • If this is your first time using Athena, you will see a prompt to set a query result location. Click Settings (gear icon) or Edit settings.
  • For Query result location, click Browse S3 and select the athena-results folder in your bucket.
  • Click Save.
  • In the Editor tab, set the Data source to AwsDataCatalog and the Database to ecommerce_pipeline.
  • In the query editor, type the following SQL and click Run:
SELECT * FROM processed LIMIT 10;

What does this query do?

This query selects all columns from the processed table and limits the results to 10 rows. It is a quick way to verify that Athena can read your Parquet files and that the data looks correct.

You should see your e-commerce order data displayed in a results table below the query editor. This data is being read directly from the Parquet files in S3.

Seeing a "Table not found" error?

Make sure the Database dropdown is set to ecommerce_pipeline and not default. Also confirm that your second crawler finished with a status of Completed and that the processed table appears in the Data Catalog. Help me troubleshoot my Athena query.

Why is Athena so fast to set up?

Athena is serverless. There are no servers to provision, no clusters to manage, and no databases to configure. Athena reads the schema from the Glue Data Catalog and reads the data directly from S3. You only pay $5.00 per TB of data scanned, and your tiny dataset costs fractions of a cent to query.

Run Analytics Queries

Now that you have confirmed Athena can read your data, it is time to run some real analytics. These queries demonstrate the kind of business intelligence work that data engineers enable with pipelines like yours.

Paste the following query into the Athena editor and click Run to see total revenue per product category:

SELECT category, 
       SUM(price * quantity) AS total_revenue,
       COUNT(*) AS number_of_orders
FROM processed
GROUP BY category
ORDER BY total_revenue DESC;

What does this query do?

This query groups all orders by their category column, then calculates the total revenue and order count for each category. The ORDER BY total_revenue DESC sorts the results so the highest-earning category appears first.

Now paste this query to see orders broken down by city:

SELECT city, 
       COUNT(*) AS order_count,
       ROUND(SUM(price * quantity), 2) AS city_revenue
FROM processed
GROUP BY city
ORDER BY city_revenue DESC;

What does this query do?

This query groups orders by city and calculates how many orders each city placed along with the total revenue. The ROUND() function formats the revenue to two decimal places for cleaner output.

You just ran SQL analytics on data stored in S3, without setting up any database servers. This is the power of a serverless data pipeline. Raw JSON goes in, and queryable business insights come out.

Pro tip: why Parquet matters here

Athena charges $5.00 per TB of data scanned. Because Parquet is a columnar format, Athena only reads the columns your query actually uses. If your query only references category and price, it skips every other column entirely. With JSON, Athena would have to read every single byte of every row. On large datasets, this difference can cut costs and query times dramatically.

Your end-to-end pipeline is complete. You took raw JSON data, cataloged it, transformed it into Parquet, cataloged the output, and queried it with SQL. Next up, the secret mission challenges you to push your analytics skills even further.

Secret mission

Write Advanced Analytics Queries

Your pipeline is working and your data is queryable. But a real data engineer does not stop at SELECT *. Push your SQL skills further by writing queries that produce actual business intelligence insights from your e-commerce data.

Before you go

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. Since this project uses serverless AWS services, there are no servers running and ongoing costs are minimal.

Cost warning

Your S3 bucket stores only a few small files, costing fractions of a cent per month. The Glue Data Catalog is free for up to 1 million objects. Athena charges $5 per TB scanned, but your queries scanned tiny amounts. The Glue ETL job and crawlers have already completed, so there are no ongoing compute costs. Leaving these resources in place is safe and essentially free.

Resources you used:

  • S3 bucket with raw JSON data, processed Parquet data, and Athena query results.
  • Glue Data Catalog database (ecommerce_pipeline) with two tables (raw and processed).
  • Glue ETL job (orders-json-to-parquet).
  • Two Glue Crawlers (raw-orders-crawler and processed-orders-crawler).
  • IAM role (GlueDataPipelineRole).

Keep everything running

No action needed. Choose this if you are still actively building or want to experiment further with your pipeline.

  • Your S3 bucket costs fractions of a cent per month for this amount of data.
  • The Glue Data Catalog is free for up to 1 million objects and 1 million requests per month.
  • There are no servers running, so there are no ongoing compute costs.

Pause - I'll come back to this later

Everything in this project is serverless. Resources only cost money when actively used, so there is nothing to pause.

  • Your S3 data, Glue catalog, crawlers, and ETL job will sit idle at near-zero cost.
  • You can return anytime and run queries in Athena or re-run your ETL job without any setup.

Delete - I don't want to use this again

Remove all project resources and start fresh. Follow these steps in order.

  • Go to S3, select your bucket, click Empty to remove all objects, then click Delete to remove the bucket.
  • Go to AWS Glue, click ETL jobs in the left sidebar, select your orders-json-to-parquet job, and click Delete.
  • Go to AWS Glue, click Crawlers in the left sidebar, select both raw-orders-crawler and processed-orders-crawler, and click Delete.
  • Go to AWS Glue, click Databases under Data Catalog, select ecommerce_pipeline, and click Delete.
  • Go to IAM, click Roles in the left sidebar, search for GlueDataPipelineRole, select it, and click Delete.

Mission accomplished

That's a Wrap!

Nice work! You just built a fully functional serverless data pipeline on AWS from scratch. Raw JSON data went in, and clean, queryable Parquet data came out. This is the exact pattern data engineers build at companies every day.

Here's what you learned:

  • Built an S3 data lake with raw and processed zones, used a Glue Crawler to automatically discover your data schema, and transformed JSON into optimized Parquet format using a Visual ETL job.
  • Queried your processed data with Athena SQL, running real analytics without setting up any database servers.
  • Learned the S3 + Glue + Athena stack, the most common serverless data pipeline pattern on AWS.
  • Wrote advanced SQL analytics queries to calculate revenue by category, daily trends, and average order values from your pipeline data.

Ready to quiz yourself?