Build a Serverless Data Lake on AWS

Build an ETL pipeline with Lambda, S3, Glue, and Athena on AWS.

Introduction

30 Second Summary

Every time you browse trending repositories on GitHub, there is a data pipeline behind the scenes collecting, transforming, and organizing that information so it can be searched and analyzed. Companies rely on this same pattern to turn raw data into business insights.

In this project, you will build a serverless data lake pipeline on AWS that pulls live data from the GitHub API, transforms it into a query-optimized format, and makes it searchable using standard SQL.

What You'll Build

A fully serverless ETL pipeline using AWS Lambda, Amazon S3, AWS Glue, and Amazon Athena that extracts trending GitHub repository data, transforms it to Parquet format, and enables SQL queries on the processed data.

By the end of this project, you'll have:

  • An S3 bucket with raw and processed data lake zones following the production pattern used by data engineering teams.
  • Two Lambda functions that extract live GitHub data and transform it into an optimized columnar format.
  • A Glue Data Catalog with auto-discovered schema and Athena queries proving end-to-end pipeline functionality.
  • Secret Mission: An event-driven S3 trigger that automates the entire pipeline so it runs without manual intervention.

Are there any prerequisites?

You need an AWS account (a free tier trial works perfectly) and basic familiarity with Python and SQL. No prior AWS experience is required.

Before We Start

Before We Start...

Install and Configure AWS CLI

The AWS CLI is the command-line tool that lets you manage AWS services directly from your terminal. Every resource you create in this project (S3 buckets, Lambda functions, Glue crawlers) gets deployed through this tool.

In this step, you'll install the AWS CLI, create an IAM user with the right permissions, and configure the CLI to authenticate with your AWS account.

In this step, get ready to:
  • Install AWS CLI v2 and verify it works.
  • Create an IAM user with programmatic access credentials.
  • Configure the CLI with your credentials and default region.
Download and install AWS CLI v2

The AWS CLI v2 is the latest major version and supports all current AWS features. You need it to create and manage the AWS resources in this project from your terminal.

  • Open the integrated terminal in VS Code by pressing Cmd+` (macOS) or Ctrl+` (Windows).
  • Check if you already have the AWS CLI installed by running this command:
aws --version

Find your result below.

✔️ I see version 2.x or higher

If you see output starting with aws-cli/2. followed by a version number, your AWS CLI v2 is already installed and ready.

Skip ahead to the next substep.

ⓧ I see version 1.x

You have AWS CLI v1 installed. This project requires v2 for full feature support. You need to install v2, which will replace v1.

macOS

  • Download the macOS installer by opening this URL in your browser:
  • https://awscli.amazonaws.com/AWSCLIV2.pkg
  • Run the downloaded .pkg file and follow the on-screen instructions.
  • When the installer finishes, close and reopen your terminal in VS Code.
  • Verify the new version by running:
aws --version

You should now see output starting with aws-cli/2. confirming v2 is active.

Windows

  • Download the Windows installer by opening this URL in your browser:
  • https://awscli.amazonaws.com/AWSCLIV2.msi
  • Run the downloaded .msi file and follow the on-screen instructions.
  • When the installer finishes, close and reopen your terminal in VS Code.
  • Verify the new version by running:
aws --version

You should now see output starting with aws-cli/2. confirming v2 is active.

ⓧ Command not found

The AWS CLI is not installed on your machine yet. Follow the instructions below for your operating system.

macOS

  • Download the macOS installer by opening this URL in your browser:
  • https://awscli.amazonaws.com/AWSCLIV2.pkg
  • Run the downloaded .pkg file and follow the on-screen instructions.
  • Choose the default installation option (install for all users).
  • When the installer finishes, close and reopen your terminal in VS Code.
  • Verify the installation by running:
aws --version

You should see output starting with aws-cli/2. confirming a successful install.

Windows

  • Download the Windows installer by opening this URL in your browser:
  • https://awscli.amazonaws.com/AWSCLIV2.msi
  • Run the downloaded .msi file and follow the on-screen instructions.
  • Accept the license agreement and use the default installation path.
  • When the installer finishes, close and reopen your terminal in VS Code.
  • Verify the installation by running:
aws --version

You should see output starting with aws-cli/2. confirming a successful install.

Still seeing command not found after installing?

Close your entire VS Code window and reopen it. The terminal needs a fresh session to pick up the new PATH entries created by the installer.

On macOS, if you installed for the current user only, you may need to add /usr/local/bin to your PATH manually.

Why isn't aws --version working after I installed it?

Create an IAM user with programmatic access

AWS uses IAM (Identity and Access Management) to control who can access your account and what they can do. You need an IAM user with an access key so the CLI can authenticate on your behalf.

  • Open the AWS Management Console in your browser and sign in to your AWS account.
  • In the search bar at the top, type IAM and select IAM from the results.
  • In the left navigation pane, click Users.
  • Click Create user.
  • For User name, enter datalake-cli-user.
  • Click Next.
  • On the Set permissions page, select Attach policies directly.
  • In the search box, type AdministratorAccess and select the checkbox next to the AdministratorAccess policy.
  • Click Next, then click Create user.

Why AdministratorAccess for a learning project?

In production, you would follow the principle of least privilege and grant only the specific permissions each service needs. For this learning project, AdministratorAccess lets you focus on building the pipeline without hitting permissions errors at every step.

You will delete this user and its credentials in the Before You Go cleanup section.

Now create an access key for the new user.

  • Click on the user name datalake-cli-user to open the user details page.
  • Select the Security credentials tab.
  • Scroll down to the Access keys section and click Create access key.
  • On the best practices page, select Command Line Interface (CLI) as your use case.
  • Check the confirmation checkbox at the bottom, then click Next.
  • Skip the description tag and click Create access key.
  • Click Download .csv file to save your credentials securely.
  • Click Done.

Save your credentials now

This is the only time AWS shows you the Secret Access Key. If you lose it, you will need to delete the access key and create a new one. Store the downloaded .csv file somewhere safe on your computer.

Configure the AWS CLI with your credentials

Now you'll connect the CLI to your AWS account using the credentials you just downloaded. The CLI stores these in a configuration file so you do not need to enter them for every command.

  • Return to your terminal in VS Code and configure the CLI by running this command:
aws configure

What does this command do?

The aws configure command starts an interactive prompt that asks for four pieces of information. It saves them to configuration files in your home directory so the CLI can authenticate all future requests.

  • When prompted, enter the following values one at a time (open your downloaded .csv file to find your keys):
  • AWS Access Key ID: paste your Access Key ID from the CSV file.
  • AWS Secret Access Key: paste your Secret Access Key from the CSV file.
  • Default region name: enter us-east-1.
  • Default output format: enter json.

Why us-east-1?

The us-east-1 region has the broadest service availability and is where most AWS tutorials and documentation default to. Using a consistent region avoids confusion when resources appear in different locations.

  • Verify that your CLI is properly authenticated by running this command:
aws sts get-caller-identity

You should see a JSON response containing your Account ID, User ID, and an ARN that includes datalake-cli-user.

Seeing an error about credentials?

Double-check that you pasted the Access Key ID and Secret Access Key correctly. A common mistake is accidentally including a trailing space when copying from the CSV file.

If you see a message about an invalid security token, run aws configure again and carefully re-enter your credentials.

Help me fix my AWS CLI credentials error.

Your AWS CLI is authenticated and ready to go. Next up, you'll create the S3 bucket that forms the foundation of your data lake.

Create the S3 Data Lake Structure

Every data lake needs a home. In production, data engineering teams store all their data in Amazon S3 because it offers virtually unlimited storage at a fraction of the cost of a database.

In this step, you will create an S3 bucket and organize it into zones. This zone-based structure keeps raw ingested data separate from cleaned, processed data.

In this step, get ready to:
  • Create an S3 bucket with a globally unique name.
  • Set up raw and processed data lake zones.
  • Verify the bucket structure in the AWS Console.
Create the S3 bucket using the CLI

S3 bucket names must be globally unique across all AWS accounts. You will create a bucket using a naming pattern that includes your initials and the current year to avoid conflicts.

  • Create your S3 bucket by running this command:
aws s3 mb s3://[[BUCKET_NAME="datalake-pipeline-yourname-2026"]]

What does this command do?

The aws s3 mb command stands for "make bucket." It creates a new S3 bucket in the region your CLI is configured to use (us-east-1).

Your bucket name must contain only lowercase letters, numbers, and hyphens. It must start and end with a letter or number.

  • You should see output similar to:
make_bucket: s3://datalake-pipeline-yourname-2026

Seeing a BucketAlreadyExists error?

This means someone else in the world already has a bucket with that name. Try a more unique name by adding extra numbers or your full name.

help me fix my S3 bucket name conflict

Create the data lake zone prefixes

Production data lakes organize data into zones. The "raw" zone holds data exactly as it arrived from the source. The "processed" zone holds cleaned, transformed data ready for analysis.

Why use zones in a data lake?

Keeping raw and processed data separate protects your original data. If a transformation introduces a bug, you can always re-process from the raw zone without re-fetching from the source API.

This pattern is called the "medallion architecture" or "zone-based organization." It is the standard structure you will find in every production data lake.

S3 does not have traditional folders. Instead, it uses key prefixes to simulate a folder structure. You will create the zone prefixes by uploading small placeholder files to each path.

  • Create the raw zone by uploading a placeholder file:
echo placeholder | aws s3 cp - s3://[[BUCKET_NAME="datalake-pipeline-yourname-2026"]]/raw/.keep
  • Create the processed zone by uploading a placeholder file:
echo placeholder | aws s3 cp - s3://[[BUCKET_NAME="datalake-pipeline-yourname-2026"]]/processed/.keep

What do these commands do?

The echo placeholder part creates a tiny piece of text. The pipe (|) sends that text to aws s3 cp -, which uploads it from standard input to S3.

The .keep filename is a convention for placeholder files. Their only job is to establish the raw/ and processed/ prefixes so that the zones appear in the S3 console.

  • Verify both zones exist by listing the bucket contents:
aws s3 ls s3://[[BUCKET_NAME="datalake-pipeline-yourname-2026"]]/
  • You should see two prefixes listed:
                           PRE processed/
                           PRE raw/

Not seeing both prefixes?

Make sure you included the trailing slash (/) in the aws s3 ls command after your bucket name. Without it, the command lists buckets rather than objects inside the bucket.

Also confirm your bucket name matches exactly in all commands. A typo in the name will return a "NoSuchBucket" error.

help me troubleshoot my S3 bucket listing

Verify the bucket structure in the AWS Console

The CLI confirmed the structure, but let's also verify it visually in the AWS Management Console.

  • Switch back to the AWS Management Console in your browser.
  • In the search bar at the top, type S3 and select S3 from the results.
  • Click on your bucket name in the list (e.g., datalake-pipeline-yourname-2026).
  • You should see two folders listed: processed/ and raw/.
  • Click into raw/ to confirm the .keep file is there.

What should I see?

Inside your bucket, the S3 console shows processed/ and raw/ as folder icons. Clicking into either one reveals the .keep placeholder file. This confirms your data lake zones are set up correctly.

Your data lake storage is ready. Next up, you will create the IAM role that gives your Lambda functions permission to read and write to this bucket.

Create the IAM Role for Lambda

Your data lake bucket is ready with its zone structure. Before deploying any Lambda functions, you need to set up the right permissions.

AWS uses IAM roles to control what each service can do. Your Lambda functions will need permission to read and write to S3 and log their activity to CloudWatch. In this step, you will create a custom IAM role that follows the principle of least privilege, granting only the specific permissions your pipeline needs.

In this step, get ready to:
  • Create IAM policy files that define least-privilege permissions for your Lambda functions.
  • Create an IAM role with a Lambda trust policy and attach your custom permissions.
  • Verify the role in the AWS Console and copy the Role ARN for later use.
Create the IAM policy JSON files

Your Lambda role needs two JSON documents. The first is a trust policy that tells AWS which service is allowed to assume the role. The second is a permissions policy that defines what the role can actually do.

  • Switch back to VS Code.
  • Press Cmd+Shift+P (macOS) or Ctrl+Shift+P (Windows) to open the Command Palette.
  • Type Terminal: Create New Terminal and press Enter.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir datalake-pipeline
cd datalake-pipeline
  • Confirm the folder was created by running:
ls

You should see the datalake-pipeline folder listed in the output if you ran ls from Desktop. You are now inside that folder.

Now create the trust policy file. This tells AWS that the Lambda service is allowed to assume this role.

  • Create a new file called trust-policy.json in your datalake-pipeline folder. Copy the following code into it:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

What does this code do?

This trust policy is like an access badge. It declares that only the Lambda service (lambda.amazonaws.com) is allowed to "wear" this role.

Without a trust policy, no service could assume the role, and your Lambda functions would have no permissions at all.

  • Save trust-policy.json.

You will not see a visible change yet. This file is used in the next substep when you create the role via the CLI.

Next, create the permissions policy. This defines exactly what your Lambda functions are allowed to do with S3 and CloudWatch.

  • Create a new file called lambda-s3-policy.json in your datalake-pipeline folder. Copy the following code into it:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject"
      ],
      "Resource": "arn:aws:s3:::[[BUCKET_NAME="your-datalake-bucket-name"]]/*"
    },
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::[[BUCKET_NAME="your-datalake-bucket-name"]]"
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:*"
    }
  ]
}

What does this code do?

  • The first statement grants s3:PutObject and s3:GetObject on all objects inside your bucket. This lets Lambda upload raw JSON and read files for transformation.
  • The second statement grants s3:ListBucket on the bucket itself. This lets the transform function find the latest raw file.
  • The third statement grants CloudWatch Logs permissions so your Lambda functions can write execution logs for debugging.

Why is this least privilege?

This policy only grants access to your specific bucket, not all S3 buckets in your account. It also only allows the specific actions your functions need. In a production environment, this prevents accidental data exposure if a function is compromised.

  • Save lambda-s3-policy.json.
  • Confirm both files exist by running:
ls

You should see trust-policy.json and lambda-s3-policy.json in the output.

Don't see both files?

  • Make sure you saved both files inside the datalake-pipeline folder on your Desktop.
  • Run pwd to confirm you are in ~/Desktop/datalake-pipeline.
  • If VS Code saved a file elsewhere, move it into this folder using mv or drag it in your file explorer.

Help me find my policy files.

✔️ Awesome, I've got everything!

Great. Make sure both files are saved before moving on.

ⓧ I'd like to double check the full code

Here is trust-policy.json:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Here is lambda-s3-policy.json:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject"
      ],
      "Resource": "arn:aws:s3:::[[BUCKET_NAME="your-datalake-bucket-name"]]/*"
    },
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::[[BUCKET_NAME="your-datalake-bucket-name"]]"
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:*"
    }
  ]
}
Create the IAM role and attach the policy

Now you will use the AWS CLI to create the role, create a managed policy from your JSON file, and attach the policy to the role.

  • Create the IAM role by running this command:
aws iam create-role --role-name lambda-datalake-role --assume-role-policy-document file://trust-policy.json

What does this command do?

  • The --role-name flag sets the name of your new role to lambda-datalake-role.
  • The --assume-role-policy-document flag points to the trust policy file you just created. The file:// prefix tells the CLI to read from a local file.

You should see a JSON response containing the role details, including an Arn field. Copy and save this ARN somewhere. You will need it in the next step.

Seeing an error about the file not found?

  • Make sure your terminal is in the same folder as your JSON files. Run pwd and confirm it shows ~/Desktop/datalake-pipeline.
  • Check that the filename matches exactly, including the .json extension.

Help me fix this create-role error.

  • Create the managed policy from your permissions file by running:
aws iam create-policy --policy-name lambda-datalake-s3-policy --policy-document file://lambda-s3-policy.json

You should see a JSON response with the new policy's details. Look for the Arn field in the output. It will look something like arn:aws:iam::123456789012:policy/lambda-datalake-s3-policy. Copy this policy ARN.

  • Attach the policy to the role by running this command, replacing the ARN with the one you just copied:
aws iam attach-role-policy --role-name lambda-datalake-role --policy-arn [[POLICY_ARN="arn:aws:iam::123456789012:policy/lambda-datalake-s3-policy"]]

What does this command do?

This links your custom permissions policy to the lambda-datalake-role role. Any Lambda function that assumes this role will inherit the S3 and CloudWatch permissions you defined.

This command produces no output on success. If you see no error message, the policy was attached correctly.

Seeing an access denied error?

  • Make sure your IAM user (the one configured with aws configure) has the AdministratorAccess policy attached.
  • Double-check that the policy ARN you pasted matches the one from the create-policy output exactly.

Help me fix this attach-role-policy error.

Verify the role in the AWS Console

The CLI commands created everything, but it is good practice to verify resources in the Console. This also gives you an easy way to find the Role ARN.

  • Return to the AWS Management Console in your browser.
  • In the search bar at the top, type IAM and select IAM from the results.
  • In the left sidebar, click Roles.
  • Search for lambda-datalake-role in the search box.
  • Click on the role name to open its details.

You should see lambda-datalake-s3-policy listed under the Permissions policies section. At the top of the page, you will see the Role ARN.

  • Copy the ARN value shown at the top of the role summary. Save it in a text file or note. You will use this in the next step when deploying your Lambda function.

Keep the ARN handy

The Role ARN looks like arn:aws:iam::123456789012:role/lambda-datalake-role. Paste it somewhere accessible. You will reference it multiple times in the next steps.

Don't see the role?

  • Make sure you are in the correct region (us-east-1). IAM is global, but the console sometimes filters by region.
  • Try refreshing the page and searching again. It can take a few seconds for new roles to appear.
  • If the role is missing entirely, re-run the aws iam create-role command from the previous substep and check for errors in the output.

Help me find my IAM role in the console.

Your IAM role is locked and loaded. Next up, you will write the Python function that extracts live data from the GitHub API and lands it in your S3 raw zone.

Build and Deploy the Extract Lambda

Your IAM role is ready and waiting to grant your Lambda function the permissions it needs. Now it is time to write the actual code that extracts data from the real world and lands it in your data lake.

In this step, you will write a Python function that calls the GitHub API to fetch trending repositories, then deploy it to AWS Lambda and run it to populate your S3 raw zone with live data.

In this step, get ready to:
  • Write the Python extraction function in VS Code.
  • Package and deploy the function to AWS Lambda.
  • Test the function and verify data lands in S3.
Write the Python extraction function

The extract function is the first stage of your ETL pipeline. It reaches out to the GitHub REST API, pulls back trending Python repositories, and saves the results as JSON in your S3 raw zone.

  • In VS Code, create a new folder called extract_lambda on your Desktop.
  • Inside that folder, create a new file called lambda_function.py.
  • Copy the following code into lambda_function.py:
import json
import boto3
import urllib.request
from datetime import datetime, timedelta

def lambda_handler(event, context):
    # Calculate date 30 days ago
    thirty_days_ago = (datetime.now() - timedelta(days=30)).strftime("%Y-%m-%d")

    # Call GitHub API - search for trending Python repos
    url = f"https://api.github.com/search/repositories?q=language:python+created:>{thirty_days_ago}&sort=stars&order=desc&per_page=30"

    req = urllib.request.Request(url)
    req.add_header("Accept", "application/vnd.github+json")
    req.add_header("User-Agent", "AWS-Lambda-DataLake-Pipeline")

    with urllib.request.urlopen(req) as response:
        data = json.loads(response.read().decode())

    # Extract relevant fields from each repository
    repos = []
    for item in data["items"]:
        repos.append({
            "name": item["name"],
            "full_name": item["full_name"],
            "description": item.get("description", ""),
            "stars": item["stargazers_count"],
            "forks": item["forks_count"],
            "language": item.get("language", "Unknown"),
            "created_at": item["created_at"],
            "updated_at": item["updated_at"],
            "url": item["html_url"],
            "owner": item["owner"]["login"],
            "open_issues": item["open_issues_count"],
            "extracted_at": datetime.now().isoformat()
        })

    # Upload to S3 raw zone
    s3 = boto3.client("s3")
    bucket_name = "[[BUCKET_NAME=\"your-datalake-bucket-name\"]]"
    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
    key = f"raw/github_repos_{timestamp}.json"

    s3.put_object(
        Bucket=bucket_name,
        Key=key,
        Body=json.dumps(repos, indent=2),
        ContentType="application/json"
    )

    return {
        "statusCode": 200,
        "body": json.dumps({
            "message": f"Extracted {len(repos)} repos to s3://{bucket_name}/{key}"
        })
    }

What does this code do?

This function has three main sections that mirror the Extract stage of an ETL pipeline:

  • API call: It queries the GitHub search endpoint for Python repositories created in the last 30 days, sorted by stars. The urllib.request module is built into Python so no extra dependencies are needed.
  • Data extraction: It loops through the API response and pulls out only the fields you care about (name, stars, forks, language, etc.), discarding the rest. This keeps your raw zone clean and focused.
  • S3 upload: It writes the extracted data as a timestamped JSON file to the raw/ prefix in your S3 bucket. Each invocation creates a new file, preserving historical extracts.
  • Replace the bucket_name placeholder with your actual S3 bucket name (e.g., datalake-pipeline-jb-2026).
  • Save lambda_function.py.

Why use urllib instead of the requests library?

Lambda comes with a minimal Python environment. The urllib.request module is part of the Python standard library, so it is available without installing any extra packages. This keeps your deployment package small and avoids needing a Lambda Layer for a simple HTTP call.

Not sure where to create the folder?

Create the extract_lambda folder anywhere convenient on your Desktop. You will zip it in the next substep. Make sure lambda_function.py is directly inside the folder, not nested in a subfolder.

help me set up my extract_lambda folder structure for AWS Lambda

Package and deploy the function to Lambda

AWS Lambda expects your code as a .zip file. You will package your function and then use the AWS CLI to create the Lambda function in the cloud.

  • Navigate into the extract_lambda folder in your terminal by running:
cd ~/Desktop/extract_lambda
  • Package the function into a .zip file by running:
zip extract_lambda.zip lambda_function.py
  • Confirm the zip file was created by running:
ls

You should see both lambda_function.py and extract_lambda.zip in the output.

  • Deploy the function to AWS Lambda by running this command:
aws lambda create-function \
    --function-name extract-github-repos \
    --runtime python3.12 \
    --handler lambda_function.lambda_handler \
    --role [[ROLE_ARN="your Lambda IAM role ARN"]] \
    --zip-file fileb://extract_lambda.zip \
    --timeout 30

What do these flags mean?

  • --function-name gives your Lambda a human-readable name.
  • --runtime python3.12 tells Lambda which language environment to use.
  • --handler lambda_function.lambda_handler points Lambda to the file (lambda_function) and function (lambda_handler) to execute.
  • --role attaches the IAM role you created in the previous step, granting S3 and CloudWatch permissions.
  • --zip-file fileb://extract_lambda.zip uploads your deployment package. The fileb:// prefix tells the CLI to read the file as binary.
  • --timeout 30 gives the function 30 seconds to complete. The default of 3 seconds is too short for an external API call.

You should see a JSON response in your terminal containing your function's ARN, runtime, and state. The State field should show Active or Pending.

Seeing an InvalidParameterValueException?

This usually means the --role ARN is incorrect. Double check you copied the full ARN from the previous step (it starts with arn:aws:iam::). Also make sure the role has a Lambda trust policy attached.

help me fix an InvalidParameterValueException when creating my Lambda function

Test the function and verify data in S3

Your function is deployed, but it will not run until something invokes it. You will trigger it manually using the CLI and then confirm the raw data landed in your S3 bucket.

  • Invoke the function by running:
aws lambda invoke \
    --function-name extract-github-repos \
    --cli-binary-format raw-in-base64-out \
    --payload '{}' \
    response.json

What does this command do?

The aws lambda invoke command triggers your function synchronously and saves the response to response.json. The --cli-binary-format raw-in-base64-out flag is required for AWS CLI v2 when passing a payload. You are sending an empty JSON object {} because this function does not need any input event data.

Your terminal should display output similar to:

{
    "ExecutedVersion": "$LATEST",
    "StatusCode": 200
}
  • Check the function's response by running:
cat response.json

You should see a JSON message confirming how many repos were extracted and the S3 path where the file was saved.

  • Verify the file landed in S3 by running:
aws s3 ls s3://[[BUCKET_NAME="your-datalake-bucket-name"]]/raw/

You should see a file named something like github_repos_20260605_143022.json in the output. The timestamp in the filename will match when you invoked the function.

Seeing a StatusCode 200 but FunctionError in the output?

A 200 status code means Lambda ran the function, but the function itself may have thrown an error. Run cat response.json to see the error details. Common causes:

  • The bucket_name in your code does not match your actual S3 bucket name. Open lambda_function.py, fix the name, re-zip, and redeploy.
  • The IAM role does not have s3:PutObject permission on your bucket. Check the policy from the previous step.

help me debug my Lambda function error when writing to S3

✔️ Awesome, I've got everything!

Great. Double check you have saved lambda_function.py and that your S3 raw zone contains a timestamped JSON file.

ⓧ I'd like to double check the full code

Here is the complete extract_lambda/lambda_function.py file:

import json
import boto3
import urllib.request
from datetime import datetime, timedelta

def lambda_handler(event, context):
    # Calculate date 30 days ago
    thirty_days_ago = (datetime.now() - timedelta(days=30)).strftime("%Y-%m-%d")

    # Call GitHub API - search for trending Python repos
    url = f"https://api.github.com/search/repositories?q=language:python+created:>{thirty_days_ago}&sort=stars&order=desc&per_page=30"

    req = urllib.request.Request(url)
    req.add_header("Accept", "application/vnd.github+json")
    req.add_header("User-Agent", "AWS-Lambda-DataLake-Pipeline")

    with urllib.request.urlopen(req) as response:
        data = json.loads(response.read().decode())

    # Extract relevant fields from each repository
    repos = []
    for item in data["items"]:
        repos.append({
            "name": item["name"],
            "full_name": item["full_name"],
            "description": item.get("description", ""),
            "stars": item["stargazers_count"],
            "forks": item["forks_count"],
            "language": item.get("language", "Unknown"),
            "created_at": item["created_at"],
            "updated_at": item["updated_at"],
            "url": item["html_url"],
            "owner": item["owner"]["login"],
            "open_issues": item["open_issues_count"],
            "extracted_at": datetime.now().isoformat()
        })

    # Upload to S3 raw zone
    s3 = boto3.client("s3")
    bucket_name = "your-datalake-bucket-name"
    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
    key = f"raw/github_repos_{timestamp}.json"

    s3.put_object(
        Bucket=bucket_name,
        Key=key,
        Body=json.dumps(repos, indent=2),
        ContentType="application/json"
    )

    return {
        "statusCode": 200,
        "body": json.dumps({
            "message": f"Extracted {len(repos)} repos to s3://{bucket_name}/{key}"
        })
    }

Your extract function is live and producing real data. Next up, you will build the transform Lambda that converts this raw JSON into optimized Parquet format for fast querying.

Build and Deploy the Transform Lambda

Your extract Lambda is pulling raw GitHub data into S3. Now you need to transform that raw JSON into a query-optimized format that analytical tools can read efficiently.

In this step, you'll write a second Lambda function that reads the raw JSON, cleans and enriches the data using pandas, and writes it as Parquet to the processed/ zone. Parquet is a columnar file format that reduces query costs by 30-90% compared to scanning raw JSON.

In this step, get ready to:
  • Write a Python transform function that converts raw JSON to Parquet.
  • Attach an AWS-managed Lambda Layer with pandas and pyarrow.
  • Deploy and test the transform Lambda, verifying Parquet output in S3.
Write the transform function

The transform function reads raw JSON from S3, structures it into a DataFrame, adds enrichment columns, and outputs optimized Parquet. This is the "T" in your ETL pipeline.

  • In VS Code, create a new folder called transform_lambda next to your existing extract_lambda folder.
  • Inside transform_lambda, create a new file called lambda_function.py.
  • Paste the following code into transform_lambda/lambda_function.py:
import json
import boto3
import pandas as pd
from datetime import datetime

def lambda_handler(event, context):
    s3 = boto3.client("s3")
    bucket_name = "your-datalake-bucket-name"

    # Find the most recent raw file
    response = s3.list_objects_v2(
        Bucket=bucket_name,
        Prefix="raw/github_repos_"
    )

    if "Contents" not in response:
        return {
            "statusCode": 404,
            "body": json.dumps({"message": "No raw files found"})
        }

What does this code do?

  • import pandas as pd brings in the pandas library for data manipulation. This will be provided by the Lambda Layer you attach later.
  • s3.list_objects_v2() lists all files in the raw/ zone that match the naming pattern from your extract function.
  • The if "Contents" not in response check handles the case where no raw files exist yet.
  • Now add the transformation logic below the code you just pasted:
    # Get the most recent file
    latest_file = sorted(response["Contents"], key=lambda x: x["LastModified"])[-1]
    raw_key = latest_file["Key"]

    # Read the raw JSON
    raw_object = s3.get_object(Bucket=bucket_name, Key=raw_key)
    raw_data = json.loads(raw_object["Body"].read().decode())

    # Transform to DataFrame
    df = pd.DataFrame(raw_data)

    # Clean and enrich
    df["description"] = df["description"].fillna("No description")
    df["stars_category"] = df["stars"].apply(
        lambda x: "viral" if x > 1000 else "popular" if x > 100 else "emerging"
    )
    df["created_at"] = pd.to_datetime(df["created_at"])
    df["updated_at"] = pd.to_datetime(df["updated_at"])

What does the transformation do?

  • sorted(..., key=lambda x: x["LastModified"])[-1] finds the most recently uploaded raw file so you always transform fresh data.
  • pd.DataFrame(raw_data) converts the JSON list into a tabular structure.
  • The enrichment step adds a stars_category column that classifies repos as "viral", "popular", or "emerging" based on star count. This is a common data engineering pattern for adding business logic during transformation.
  • Finally, add the Parquet output section below the enrichment code:
    # Write Parquet to processed zone
    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
    parquet_key = f"processed/github_repos_{timestamp}.parquet"

    parquet_buffer = df.to_parquet(index=False)

    s3.put_object(
        Bucket=bucket_name,
        Key=parquet_key,
        Body=parquet_buffer,
        ContentType="application/octet-stream"
    )

    return {
        "statusCode": 200,
        "body": json.dumps({
            "message": f"Transformed {len(df)} records to s3://{bucket_name}/{parquet_key}"
        })
    }

Why Parquet instead of CSV?

Parquet stores data in columns rather than rows. When Athena queries only specific columns (like stars and language), it skips all other columns entirely. This dramatically reduces the amount of data scanned per query.

Since Amazon Athena charges $5 per TB of data scanned, Parquet directly saves money on every query you run.

  • Replace "your-datalake-bucket-name" with your actual S3 bucket name (the same one you used in the extract function).
  • Save transform_lambda/lambda_function.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved transform_lambda/lambda_function.py before moving on.

ⓧ I'd like to double check the full code

import json
import boto3
import pandas as pd
from datetime import datetime

def lambda_handler(event, context):
    s3 = boto3.client("s3")
    bucket_name = "your-datalake-bucket-name"

    # Find the most recent raw file
    response = s3.list_objects_v2(
        Bucket=bucket_name,
        Prefix="raw/github_repos_"
    )

    if "Contents" not in response:
        return {
            "statusCode": 404,
            "body": json.dumps({"message": "No raw files found"})
        }

    # Get the most recent file
    latest_file = sorted(response["Contents"], key=lambda x: x["LastModified"])[-1]
    raw_key = latest_file["Key"]

    # Read the raw JSON
    raw_object = s3.get_object(Bucket=bucket_name, Key=raw_key)
    raw_data = json.loads(raw_object["Body"].read().decode())

    # Transform to DataFrame
    df = pd.DataFrame(raw_data)

    # Clean and enrich
    df["description"] = df["description"].fillna("No description")
    df["stars_category"] = df["stars"].apply(
        lambda x: "viral" if x > 1000 else "popular" if x > 100 else "emerging"
    )
    df["created_at"] = pd.to_datetime(df["created_at"])
    df["updated_at"] = pd.to_datetime(df["updated_at"])

    # Write Parquet to processed zone
    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
    parquet_key = f"processed/github_repos_{timestamp}.parquet"

    parquet_buffer = df.to_parquet(index=False)

    s3.put_object(
        Bucket=bucket_name,
        Key=parquet_key,
        Body=parquet_buffer,
        ContentType="application/octet-stream"
    )

    return {
        "statusCode": 200,
        "body": json.dumps({
            "message": f"Transformed {len(df)} records to s3://{bucket_name}/{parquet_key}"
        })
    }
Create a Lambda Layer and deploy the function

Your transform function imports pandas and uses pyarrow (under the hood for Parquet). These libraries are not included in the default Lambda runtime. You need to attach a Lambda Layer that provides them.

What is a Lambda Layer?

A Lambda Layer is a .zip archive containing libraries, custom runtimes, or other dependencies. Instead of packaging large libraries inside every function's deployment .zip, you attach a shared Layer. Multiple functions can reference the same Layer.

AWS maintains an official Layer called AWSSDKPandas (formerly AWS Data Wrangler) that bundles pandas, pyarrow, and boto3 extensions for Lambda.

  • First, package your transform function code into a .zip file by running this command from the folder containing transform_lambda/:
cd transform_lambda && zip ../transform_function.zip lambda_function.py && cd ..

What does this command do?

This enters the transform_lambda directory, zips only lambda_function.py into a file called transform_function.zip one level up, then navigates back. The .zip is your deployment package.

  • Deploy the transform Lambda with the AWS-managed pandas layer attached by running:
aws lambda create-function \
    --function-name transform-github-repos \
    --runtime python3.12 \
    --role [[LAMBDA_ROLE_ARN="your Lambda role ARN from Step 3"]] \
    --handler lambda_function.lambda_handler \
    --zip-file fileb://transform_function.zip \
    --timeout 60 \
    --memory-size 512 \
    --layers arn:aws:lambda:us-east-1:336392948345:layer:AWSSDKPandas-Python312:14

What does this command do?

  • --function-name transform-github-repos names your new function.
  • --runtime python3.12 sets the Python version.
  • --timeout 60 gives the function 60 seconds to complete. The pandas transformation needs more time than the default 3 seconds.
  • --memory-size 512 allocates 512 MB of memory. Pandas needs more memory than the default 128 MB to process data.
  • --layers attaches the AWS-managed AWSSDKPandas layer, which provides pandas and pyarrow in the Lambda execution environment.

Find the latest layer version

The layer version number at the end of the ARN (e.g., :14) changes as AWS publishes updates. If you get an error about the layer not being found, search for AWSSDKPandas-Python312 in the Lambda console under **Layers** to find the current version number for your region.

  • Verify the function was created successfully. You should see a JSON response containing "FunctionName": "transform-github-repos" and a "Layers" array with the AWSSDKPandas ARN.

Seeing a layer version error?

  • If you see an error mentioning the layer version does not exist, the version number may have changed. Open the AWS Lambda console, click **Layers** in the left sidebar, search for AWSSDKPandas-Python312, and use the latest version number in your command.
  • If you see an "InvalidParameterValueException" about the role, double check the Role ARN from Step 3. It should start with arn:aws:iam::.

Help me troubleshoot my Lambda deployment error.

Test the transform function

Time to run your transform Lambda and confirm it reads from raw/ and writes Parquet to processed/.

  • Invoke the transform function by running:
aws lambda invoke \
    --function-name transform-github-repos \
    --cli-binary-format raw-in-base64-out \
    --payload '{}' \
    transform_output.json
  • Check the response by running:
cat transform_output.json
  • You should see a response like: {"statusCode": 200, "body": "{\"message\": \"Transformed 30 records to s3://your-bucket/processed/github_repos_20260605_143022.parquet\"}"} .
  • Verify the Parquet file exists in S3 by running:
aws s3 ls s3://[[BUCKET_NAME="your-datalake-bucket-name"]]/processed/
  • You should see a .parquet file listed with a timestamp in its name.

Seeing a timeout or memory error?

  • If the function times out, increase the timeout with: aws lambda update-function-configuration --function-name transform-github-repos --timeout 120.
  • If you see a memory error, increase memory with: aws lambda update-function-configuration --function-name transform-github-repos --memory-size 1024.
  • If the output shows "No raw files found", re-run your extract Lambda first to make sure raw data exists in S3.

Help me fix my transform Lambda invocation error.

Your ETL pipeline now has both the Extract and Transform stages working. Next up, you'll set up AWS Glue to automatically discover the schema of your Parquet data so you can query it with SQL.

Set Up Glue Crawler and Data Catalog

Your transform Lambda successfully converted raw JSON into optimized Parquet format. The processed data is sitting in your S3 bucket, ready to be queried.

The missing piece is a way to query that data with standard SQL. AWS Glue solves this by automatically discovering the structure of your Parquet files and registering it in a metadata catalog. Once registered, Amazon Athena can query the data as if it were a traditional database table.

In this step, get ready to:
  • Create a Glue Database to organize your table metadata.
  • Create an IAM role for the Glue Crawler with least-privilege permissions.
  • Run a Glue Crawler to auto-discover the schema of your Parquet data.
Create the Glue Database

A Glue Database is a logical container in the AWS Glue Data Catalog that holds table definitions. The Crawler will register tables into this database after discovering your data's schema.

  • Create the Glue Database by running this command in your terminal:
aws glue create-database --database-input "{\"Name\":\"github_datalake\"}"

What does this command do?

The --database-input flag accepts a JSON structure with the database name. This creates a new entry called github_datalake in your Glue Data Catalog.

The Data Catalog stores only metadata (column names, types, file locations). It does not move or copy your actual data.

  • The command produces no output on success. Verify the database exists by running:
aws glue get-database --name github_datalake
  • You should see a JSON response containing "Name": "github_datalake" confirming the database was created.

Seeing an error?

If you see a permissions error, confirm your IAM user has the AdministratorAccess policy attached (set up in Step 1).

If the database already exists, you will see an AlreadyExistsException. This is safe to ignore.

Help me troubleshoot my Glue database creation error.

Create an IAM role for the Glue Crawler

The Glue Crawler needs its own IAM role to access your S3 bucket and write to the Data Catalog. This follows the same least-privilege pattern you used for the Lambda role earlier.

  • Create a trust policy file for Glue by running this command:
echo '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"glue.amazonaws.com"},"Action":"sts:AssumeRole"}]}' > glue-trust-policy.json
  • Create the IAM role using the trust policy file by running:
aws iam create-role --role-name GlueCrawlerRole --assume-role-policy-document file://glue-trust-policy.json

What does the trust policy do?

The trust policy tells AWS which service is allowed to assume this role. By specifying glue.amazonaws.com as the principal, you allow the Glue service to use this role's permissions when the crawler runs.

  • Attach the AWS-managed AWSGlueServiceRole policy by running:
aws iam attach-role-policy --role-name GlueCrawlerRole --policy-arn arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole

What is AWSGlueServiceRole?

This is an AWS-managed policy that gives the Glue Crawler permissions to write to the Data Catalog and access CloudWatch Logs. It does not include S3 access, which is why you need to add a separate S3 policy next.

  • Create an S3 policy file for the crawler. In VS Code, create a new file called glue-s3-policy.json and paste this content:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::[[DATALAKE_BUCKET="your-datalake-bucket-name"]]",
        "arn:aws:s3:::[[DATALAKE_BUCKET="your-datalake-bucket-name"]]/*"
      ]
    }
  ]
}

What does this policy grant?

The s3:GetObject action lets the crawler read individual files from your bucket. The s3:ListBucket action lets it list what files exist in the processed/ prefix.

The first Resource line targets the bucket itself (needed for listing). The second targets all objects inside the bucket (needed for reading).

  • Save glue-s3-policy.json, then attach it to the role by running:
aws iam put-role-policy --role-name GlueCrawlerRole --policy-name GlueCrawlerS3Policy --policy-document file://glue-s3-policy.json
  • Verify the role was created correctly by running:
aws iam get-role --role-name GlueCrawlerRole --query "Role.Arn"
  • You should see the Role ARN in the output. Note this value for the next substep.

Role creation failed?

If you see EntityAlreadyExists, a role with that name already exists. You can proceed to the next substep using the existing role.

If the put-role-policy command fails, double-check that you replaced the bucket name placeholder in glue-s3-policy.json with your actual bucket name and saved the file.

Help me troubleshoot my Glue IAM role creation.

Create and run the Glue Crawler

Now you have a database to store table metadata and a role with the right permissions. The crawler will scan your processed/ zone, detect the Parquet file's columns and types, and register a table in the github_datalake database.

  • Create the Glue Crawler by running:
aws glue create-crawler --name github-repos-crawler --role GlueCrawlerRole --database-name github_datalake --targets '{"S3Targets":[{"Path":"s3://[[DATALAKE_BUCKET="your-datalake-bucket-name"]]/processed/"}]}'

What does this command do?

The --name gives the crawler a human-readable identifier. The --role tells the crawler which IAM role to assume when it runs.

The --targets JSON structure points the crawler at the processed/ prefix in your S3 bucket. The --database-name tells it where to register the discovered table.

Why only the processed/ zone?

You point the crawler at processed/ rather than raw/ because the Parquet files in the processed zone have a clean, flat schema optimized for SQL queries. The raw JSON files would create a messier table with nested fields.

  • Start the crawler by running:
aws glue start-crawler --name github-repos-crawler

This takes 1-2 minutes

The crawler needs time to spin up, scan your S3 files, infer the schema, and register the table. Wait about 1-2 minutes before checking the status.

  • Check the crawler's status by running:
aws glue get-crawler --name github-repos-crawler --query "Crawler.State"
  • Wait until the output shows "READY". If it still shows "RUNNING", wait another 30 seconds and run the command again.
  • Once the crawler is ready, verify it created a table in the AWS Console.
  • In the AWS Console, search for Glue in the top search bar and select AWS Glue.
  • In the left sidebar, click Tables under Data Catalog.
  • Select the github_datalake database from the database dropdown.
  • You should see a table named processed (the crawler names it based on the S3 folder structure).
  • Click the table name to inspect its schema. You should see columns matching your Parquet data: name, full_name, stars, language, stars_category, and others.

What is schema-on-read?

Traditional databases require you to define a schema before loading data. A data lake flips this pattern. You store the data first, then let the Glue Crawler discover the schema automatically.

This is called schema-on-read because the schema is applied when you read (query) the data, not when you write it. It makes data lakes flexible enough to handle evolving data formats.

How much does the crawler cost?

Glue Crawlers cost $0.44 per DPU-hour, billed per second with a 10-minute minimum. Your tiny dataset runs for about 1-2 minutes but gets billed for the 10-minute minimum at 2 DPUs. That is roughly $0.15 per run.

The Data Catalog itself is free for the first 1 million objects and 1 million requests per month.

Crawler finished but no table appeared?

Confirm the crawler's S3 target path matches where your Parquet file is stored. The path must end with a trailing slash (e.g., processed/).

Also check that your GlueCrawlerRole has s3:GetObject and s3:ListBucket permissions on the correct bucket. If the role cannot read the files, the crawler finishes without creating any tables.

Help me figure out why my Glue Crawler did not create a table.

Your data lake now has a fully registered schema. Next up, you will query this data with standard SQL using Amazon Athena.

Query with Athena

Your data lake now has structured Parquet data sitting in the processed zone, with its schema registered in the Glue Data Catalog.

The final piece is querying that data using standard SQL. Amazon Athena lets you run SQL queries directly against data in S3 without provisioning any servers or loading data into a separate database.

In this step, get ready to:
  • Configure Athena's query result location and select the data lake database.
  • Run SQL queries to explore your processed GitHub repository data.
  • Observe Athena's cost model by checking the data scanned metric per query.
Set up Athena's query result location

Before Athena can run any query, it needs an S3 location to store query output files. You will point it to a path inside your existing data lake bucket.

  • In the AWS Console search bar at the top, type Athena and select Amazon Athena from the results.
  • If this is your first time opening Athena, you may see a banner prompting you to set a query result location. Click Settings in the navigation bar at the top.
  • Click Manage.
  • In the Location of query result box, enter the S3 path for your Athena output:
s3://[[BUCKET_NAME="your-datalake-bucket-name"]]/athena-results/
  • Click Save.

Why does Athena need a result location?

Every query Athena runs produces output files (CSV results and metadata). These files are stored in the S3 path you just configured. Without this setting, Athena refuses to execute any query.

  • Return to the query editor by clicking Editor in the left navigation pane.
  • In the Data source dropdown on the left panel, confirm it says AwsDataCatalog.
  • In the Database dropdown below it, select github_datalake.

You should see the table processed_github_repos listed under your database. This is the table the Glue Crawler created in the previous step.

Don't see the github_datalake database?

Make sure the Data source dropdown says AwsDataCatalog. If you still don't see it, confirm your Glue Crawler ran successfully in the previous step.

Help me troubleshoot why my database isn't showing up

Run SQL queries against the processed data

With the database selected, you can now write SQL to explore your GitHub repository data. You will run five queries that progressively demonstrate Athena's analytical power.

  • Preview the data by pasting this query into the query editor and clicking Run:
-- Preview the data
SELECT * FROM github_datalake.processed_github_repos LIMIT 10;

What does this query do?

This selects all columns from your processed table but limits the output to 10 rows. It is a quick sanity check that confirms Athena can read your Parquet data and that the schema looks correct.

You should see 10 rows of repository data in the results panel below the editor, with columns like name, stars, language, and stars_category.

  • Clear the editor, then count the total records by pasting this query and clicking Run:
-- Count total records
SELECT COUNT(*) as total_repos FROM github_datalake.processed_github_repos;

You should see a single row with the value 30 (since your extract function fetches 30 repositories per run).

  • Find the top 10 most starred repositories by running this query:
-- Top 10 most starred repositories
SELECT name, owner, stars, language, stars_category
FROM github_datalake.processed_github_repos
ORDER BY stars DESC
LIMIT 10;

What does this query do?

This selects five specific columns and orders the results by the stars column in descending order. You get the most popular repositories first.

Notice the query only reads the columns you specified. This is where Parquet's columnar format shines. Athena skips the columns you did not request, scanning less data.

  • Calculate average stars by programming language by running this query:
-- Average stars by programming language
SELECT language, 
       COUNT(*) as repo_count, 
       AVG(stars) as avg_stars,
       MAX(stars) as max_stars
FROM github_datalake.processed_github_repos
GROUP BY language
ORDER BY avg_stars DESC;
  • Check the distribution of star categories by running this final query:
-- Distribution of star categories
SELECT stars_category, COUNT(*) as count
FROM github_datalake.processed_github_repos
GROUP BY stars_category;

What do these aggregation queries show?

The language query groups repositories by their primary language and calculates the count, average stars, and maximum stars for each group.

The star category query shows how your transform Lambda's enrichment logic classified repos as viral, popular, or emerging. This proves the entire pipeline works end to end.

Seeing a query error?

If you see a "table not found" error, confirm you selected github_datalake in the Database dropdown. The fully qualified table name is github_datalake.processed_github_repos.

If you see a permissions error, ensure your IAM user has the AdministratorAccess policy attached (configured in Step 1).

Help me troubleshoot my Athena query error

✔️ Awesome, I've got everything!

Great. Double check you've saved all five queries to a local file called athena_queries.sql in your project folder for future reference.

ⓧ I'd like to double check the full code

-- Preview the data
SELECT * FROM github_datalake.processed_github_repos LIMIT 10;

-- Count total records
SELECT COUNT(*) as total_repos FROM github_datalake.processed_github_repos;

-- Top 10 most starred repositories
SELECT name, owner, stars, language, stars_category
FROM github_datalake.processed_github_repos
ORDER BY stars DESC
LIMIT 10;

-- Average stars by programming language
SELECT language, 
       COUNT(*) as repo_count, 
       AVG(stars) as avg_stars,
       MAX(stars) as max_stars
FROM github_datalake.processed_github_repos
GROUP BY language
ORDER BY avg_stars DESC;

-- Distribution of star categories
SELECT stars_category, COUNT(*) as count
FROM github_datalake.processed_github_repos
GROUP BY stars_category;
Explore Athena's cost model

Athena charges based on the amount of data scanned per query. Understanding this metric helps you design cost-efficient queries in production.

  • After running your most recent query, click the Query stats tab below the results panel.
  • Look at the Data scanned field. This shows how many bytes Athena read from S3 to answer your query.
  • Run the preview query again (SELECT *) and note the data scanned value.
  • Now run the top-10 query (which only selects 5 columns) and compare the data scanned value.

Why is the data scanned different?

Parquet stores data in columns rather than rows. When your query only requests specific columns, Athena physically skips the columns you did not ask for. This means a query selecting 5 columns out of 12 reads roughly 40% of the data compared to SELECT *.

At Athena's pricing of $5 per TB scanned, this columnar efficiency translates directly into cost savings at scale. Your tiny dataset costs fractions of a cent, but imagine billions of rows in a production data lake.

Save money with column selection

In production, always select only the columns you need. Avoid SELECT * on large Parquet tables. Combined with partitioning, you can reduce Athena costs by 90% or more.

Your serverless data lake pipeline is now fully operational. You have extracted live data, transformed it into an optimized format, cataloged its schema, and queried it with standard SQL. Next up, you will take it further with a secret mission that makes this pipeline fully event-driven.

Secret mission

Event-Driven Pipeline with S3 Triggers

Right now your pipeline requires you to manually invoke the transform Lambda after extraction. What if the entire pipeline ran itself? In this secret mission, you'll wire up an S3 event trigger so the transform Lambda fires automatically whenever new raw data lands in the bucket.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project uses serverless AWS resources that incur near-zero ongoing cost when idle. The main ongoing charge is S3 storage (fractions of a cent per month for your small dataset).

Cost warning

Your Lambda functions only cost money when invoked, and your Glue Crawler only charges when it runs ($0.44 per DPU-hour with a 10-minute minimum). If you leave the S3 event trigger active and invoke the extract Lambda again, the pipeline will execute automatically.

The IAM user you created has AdministratorAccess. Even if cost is minimal, leaving an admin-level access key active is a security risk. Delete it if you are done with this project.

Resources you used:

  • S3 bucket (contains raw JSON, processed Parquet, and Athena results).
  • 2 Lambda functions (extract-github-repos and transform-github-repos).
  • 1 Lambda Layer (pandas and pyarrow for Python 3.12).
  • Glue Crawler and Glue Database (github_datalake).
  • IAM roles (Lambda execution role, Glue Crawler role) and custom IAM policy.
  • IAM user with programmatic access and AdministratorAccess policy.
  • S3 event notification (triggering the transform Lambda on new raw data).

Keep everything running

No action needed. Choose this if you plan to continue experimenting with your data lake pipeline.

  • Your Lambda functions will not run unless you invoke them. No compute charges accrue while idle.
  • S3 storage for your small dataset costs fractions of a cent per month.
  • The Glue Crawler will not run unless you trigger it manually. No charges while idle.
  • The S3 event trigger remains active. If you invoke the extract Lambda, the transform Lambda fires automatically.
  • AWS CLI remains installed on your machine for future projects.

Pause - I'll come back to this later

Remove the event trigger so the pipeline does not fire unexpectedly. Keep all resources intact for later use.

  • Remove the S3 event notification by running this command:
  • Replace the bucket name with your own bucket name.
aws s3api put-bucket-notification-configuration --bucket [[BUCKET_NAME="your-datalake-bucket-name"]] --notification-configuration "{}"
  • This resets the bucket to have no event notifications, so the transform Lambda will not fire automatically.
  • All your data, Lambda functions, Glue resources, and IAM roles remain intact for when you return.

Delete - I don't want to use this again

Remove all project resources completely. Run the following commands in your terminal in order.

Delete both Lambda functions:

aws lambda delete-function --function-name extract-github-repos
aws lambda delete-function --function-name transform-github-repos

Delete the Lambda Layer:

aws lambda delete-layer-version --layer-name [[LAYER_NAME="your-lambda-layer-name"]] --version-number 1

Delete the Glue Crawler and Database:

aws glue delete-crawler --name [[CRAWLER_NAME="your-glue-crawler-name"]]
aws glue delete-database --name github_datalake

What about the tables inside the database?

Deleting a Glue Database also removes all the tables within it. You do not need to delete the processed_github_repos table separately.

Empty and delete the S3 bucket:

aws s3 rb s3://[[BUCKET_NAME="your-datalake-bucket-name"]] --force

What does --force do?

The --force flag empties all objects in the bucket before deleting it. This removes your raw JSON files, processed Parquet files, and Athena query results in one command.

Delete IAM roles and policies:

  • First, detach all policies from the Lambda execution role and delete it:
aws iam detach-role-policy --role-name [[LAMBDA_ROLE="your-lambda-role-name"]] --policy-arn [[POLICY_ARN="arn:aws:iam::ACCOUNT_ID:policy/your-policy-name"]]
aws iam delete-role --role-name [[LAMBDA_ROLE="your-lambda-role-name"]]
  • Then detach the managed policy from the Glue Crawler role and delete it:
aws iam detach-role-policy --role-name [[GLUE_ROLE="your-glue-crawler-role-name"]] --policy-arn arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole
aws iam delete-role --role-name [[GLUE_ROLE="your-glue-crawler-role-name"]]
  • Delete the custom IAM policy you created for Lambda:
aws iam delete-policy --policy-arn [[POLICY_ARN="arn:aws:iam::ACCOUNT_ID:policy/your-policy-name"]]

Delete the IAM user:

  • First, detach the AdministratorAccess policy from the user:
aws iam detach-user-policy --user-name [[IAM_USER="your-iam-username"]] --policy-arn arn:aws:iam::aws:policy/AdministratorAccess
  • List the access keys for the user, then delete them:
aws iam list-access-keys --user-name [[IAM_USER="your-iam-username"]]
  • Copy the AccessKeyId from the output above and use it in this command:
aws iam delete-access-key --user-name [[IAM_USER="your-iam-username"]] --access-key-id your-access-key-id-here
  • Finally, delete the user:
aws iam delete-user --user-name [[IAM_USER="your-iam-username"]]

Reconfigure your AWS CLI

After deleting the IAM user, the credentials stored in your AWS CLI configuration are no longer valid. If you use the AWS CLI for other projects, run aws configure again with different credentials.

Nice Work!

Nice Work!

You just built a production-pattern serverless data lake pipeline on AWS. This is the exact Extract-Transform-Load architecture that data engineering teams use at scale to power analytics and reporting.

You've learned how to:

  • Built a serverless ETL pipeline that extracts live data from the GitHub API using AWS Lambda and lands it in an S3 data lake with raw and processed zones.
  • Transformed raw JSON into optimized Parquet format and made it queryable with standard SQL using AWS Glue and Amazon Athena.
  • Designed IAM roles following least-privilege principles to secure every service in the pipeline end-to-end.
  • Made the pipeline fully event-driven with S3 triggers so data flows automatically from extraction through transformation to queryable results.

Ready to quiz yourself?