Build a FIFA World Cup Analytics Pipeline

Transform raw FIFA JSON data into an automated analytics pipeline.

Introduction

30 Second Summary

Raw data is like a messy filing cabinet. You know the answers are buried in there somewhere, but every time you search, you end up sifting through everything just to find one fact.

In this project, you will build a fully automated analytics pipeline that transforms raw FIFA World Cup JSON data into a query-optimized format, schedules the transformation to run on its own, and delivers the results in an interactive dashboard.

What You'll Build

When you show this off, you'll run the same query against raw data and then optimized data, watch the amount scanned plummet, and then open a live dashboard of World Cup trends powered entirely by your pipeline.

By the end of this project, you'll have:

  • A running AWS Glue ETL pipeline that reads raw JSON, enriches it with derived columns, and writes compressed Parquet files partitioned by tournament year into a multi-zone data lake.
  • A side-by-side Athena query comparison proving your optimized format scans a fraction of the data, plus an EventBridge Scheduler that triggers the pipeline daily without manual intervention.
  • A published QuickSight dashboard with interactive charts showing goals-per-tournament trends and match distribution across teams.
  • Secret Mission: Build a curated aggregation layer with a second Glue job that pre-computes tournament-level statistics.

Are there any prerequisites?

This project picks up where the FIFA World Cup Data Lake project left off. You need an existing S3 bucket with raw JSON match data, a Glue database, and Athena access already configured.

You also need the AWS CLI configured on your Mac and an active AWS account with IAM admin permissions.

Before We Start

Before we dive in, let's get clear on what you're about to build and why it matters for your data engineering skills. This is your moment to commit to the end goal before any hands-on work begins.

Set Up the Multi-Zone Data Lake and IAM Roles

Your existing FIFA World Cup project stores all match data in a single S3 prefix. That works for a first pass, but production data lakes separate data into distinct zones based on its processing stage.

In this step, you'll carve your bucket into three clear zones and create the IAM roles your pipeline needs to read, transform, and schedule data across those zones.

In this step, get ready to:
  • Create S3 zone prefixes for your multi-zone data lake.
  • Create the GlueETLRole for your ETL job.
  • Create the EventBridgeSchedulerGlueRole for automated scheduling.
Create the S3 zone structure

A multi-zone data lake organizes data by processing stage. The raw zone holds original ingested data. The processed zone holds cleaned, optimized data. The curated zone holds pre-aggregated summaries for dashboards.

Your bucket already has the raw/matches/ prefix from the previous project. You need to create the other two zones.

  • Create the processed/ and curated/ prefixes in your existing bucket by running these commands:
aws s3api put-object --bucket [[BUCKET_NAME="your-bucket-name"]] --key processed/
aws s3api put-object --bucket [[BUCKET_NAME="your-bucket-name"]] --key curated/

What do these commands do?

S3 doesn't have real folders. The put-object command creates a zero-byte object with a trailing slash, which Amazon S3 and the console display as a folder.

Each prefix acts as a zone boundary. Data landing in processed/ has been transformed and optimized. Data in curated/ has been aggregated for fast dashboard queries.

  • Verify the new prefixes exist by running:
aws s3 ls s3://[[BUCKET_NAME="your-bucket-name"]]/

You should see three prefixes listed: curated/, processed/, and raw/.

Not seeing all three prefixes?

  • Make sure you included the trailing slash in both the --key values. Without it, S3 creates a file rather than a folder-like prefix.
  • Double-check that you replaced the bucket name with your actual bucket from the previous project.
  • If you see an access denied error, confirm your AWS CLI is configured with admin credentials.

Ask for help: Why aren't my S3 prefixes showing up after running put-object?

Create the GlueETLRole IAM role

AWS Glue needs an IAM role to access your S3 data. This role tells AWS which services Glue can talk to and which buckets it can read from and write to.

  • Create the GlueETLRole with a trust policy that allows the Glue service to assume it by running:
aws iam create-role --role-name GlueETLRole --assume-role-policy-document '{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "glue.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}'

What is a trust policy?

A trust policy defines which AWS service is allowed to "wear" this role. By specifying glue.amazonaws.com as the principal, you're saying only the Glue service can assume this role and use its permissions.

You should see JSON output confirming the role was created, including the role's ARN and the trust policy you specified.

  • Attach the AWSGlueServiceRole managed policy, which grants Glue the baseline permissions it needs to run jobs:
aws iam attach-role-policy --role-name GlueETLRole --policy-arn arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole
  • Add an inline policy granting Glue read and write access to your specific S3 bucket by running:
aws iam put-role-policy --role-name GlueETLRole --policy-name S3AccessPolicy --policy-document '{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::[[BUCKET_NAME="your-bucket-name"]]/*"
    }
  ]
}'

Why two separate policies?

The managed policy (AWSGlueServiceRole) provides Glue's internal service permissions like logging and catalog access. The inline policy adds your bucket-specific S3 permissions on top.

Separating them means you can swap buckets later by editing only the inline policy, without touching the managed baseline.

  • Verify both policies are attached by running:
aws iam list-attached-role-policies --role-name GlueETLRole

You should see AWSGlueServiceRole in the output list.

Seeing 'NoSuchEntity' or 'MalformedPolicy'?

  • If you see a NoSuchEntity error on attach-role-policy, the create-role command may have failed. Re-run the create-role command above first.
  • If you see a MalformedPolicy error on put-role-policy, check that your JSON is valid. Single quotes wrap the entire document, and internal strings use escaped double quotes.
  • Make sure you replaced the bucket name placeholder with your actual bucket name from the previous project.

Ask for help: I'm getting an error when creating or configuring my GlueETLRole. Can you help me troubleshoot?

Create the EventBridge Scheduler execution role

EventBridge Scheduler needs its own IAM role to trigger your Glue job on a schedule. This role gives the scheduler permission to call glue:StartJobRun and nothing else.

  • Create the EventBridgeSchedulerGlueRole with a trust policy for the scheduler service by running:
aws iam create-role --role-name EventBridgeSchedulerGlueRole --assume-role-policy-document '{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "scheduler.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}'

You should see JSON output confirming the role was created with scheduler.amazonaws.com in the trust policy.

  • Add an inline policy granting permission to start your Glue job by running:
aws iam put-role-policy --role-name EventBridgeSchedulerGlueRole --policy-name GlueStartJobPolicy --policy-document '{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "glue:StartJobRun",
      "Resource": "arn:aws:glue:*:*:job/worldcup-json-to-parquet"
    }
  ]
}'

Why scope to a specific job?

The resource ARN targets only the worldcup-json-to-parquet job. This follows the principle of least privilege. The scheduler can start only this one job, not every Glue job in your account.

  • Verify the role and its policy by running:
aws iam get-role --role-name EventBridgeSchedulerGlueRole --query 'Role.AssumeRolePolicyDocument'

You should see the trust policy with scheduler.amazonaws.com as the principal service.

Role already exists?

  • If you see an EntityAlreadyExists error, the role was created on a previous attempt. You can skip the create-role command and proceed to the put-role-policy command.
  • If the trust policy shows the wrong service, delete the role with aws iam delete-role --role-name EventBridgeSchedulerGlueRole and recreate it.

Ask for help: I'm having trouble creating the EventBridgeSchedulerGlueRole. What should I check?

Before you run the final verification, what do you expect to see when you list the contents of your processed/ prefix?

  • Confirm the full setup by running these two verification commands:
aws s3 ls s3://[[BUCKET_NAME="your-bucket-name"]]/processed/
aws iam get-role --role-name GlueETLRole --query 'Role.AssumeRolePolicyDocument'

The first command returns empty output. The processed/ prefix exists but contains no data yet. The second command shows your trust policy with glue.amazonaws.com as the trusted service.

Your data lake structure is in place and both IAM roles are ready. Next up, you'll write the PySpark ETL script that transforms your raw JSON into query-optimized Parquet.

Build and Run the Glue ETL Job

In the previous step, you laid the groundwork with S3 zones and IAM roles. Now it is time to put those zones to work.

Right now, every Athena query against your raw JSON must parse the entire file, even if you only need one column. Converting to Parquet with Snappy compression and partitioning by year solves both problems at once: columnar storage lets Athena skip irrelevant columns, and partitions let it skip irrelevant rows.

In this step, get ready to:
  • Create a Glue ETL job in the AWS console with minimal-cost configuration.
  • Paste a PySpark script that transforms JSON to partitioned Parquet.
  • Run the job and verify Parquet files appear in your processed zone.
Create the Glue ETL job

You will create a new Spark script job in the AWS Glue console, configure it for minimum cost, and open the script editor where you will paste the ETL code.

  • In the AWS Management Console, search for and open AWS Glue.
  • In the left navigation pane, choose ETL jobs.
  • In the Create job section, select Script editor.
  • For Engine, choose Spark.
  • Under Options, select Start fresh.
  • Click Create.

The script editor opens with boilerplate code. You will replace it shortly.

  • Click the Job details tab at the top.
  • For Name, enter worldcup-json-to-parquet.
  • For IAM Role, select GlueETLRole from the dropdown.
  • For Glue version, choose Glue 4.0.
  • For Worker type, select G.1X.
  • For Requested number of workers, enter 2.

Why G.1X with 2 workers?

G.1X is 1 DPU per worker (4 vCPUs, 16 GB memory). Two workers is the minimum for a Spark ETL job and costs approximately $0.03 for a 1-2 minute run at $0.44 per DPU-hour.

The default is 10 workers, which would cost 5x more with zero benefit on a small dataset. Always right-size for your data volume.

Paste the PySpark ETL script

Now you will write the actual transformation logic. The script reads JSON from the raw zone, derives analytics columns, and writes compressed Parquet partitioned by tournament year.

  • Click the Script tab to return to the code editor.
  • Select all the boilerplate code and delete it.
  • Paste the following imports and job initialization code:
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.sql.functions import col, year, to_date, upper, coalesce, lit

# Resolve job arguments passed by Glue at runtime
args = getResolvedOptions(sys.argv, ['JOB_NAME'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

What does this code do?

  • The imports bring in Glue's DynamicFrame API, PySpark's SparkContext, and the SQL functions you will use for transformations.
  • getResolvedOptions reads job parameters that Glue passes at runtime (like the job name).
  • GlueContext wraps SparkContext with Glue-specific helpers for reading from and writing to S3, JDBC, and the Data Catalog.
  • job.init() registers the job lifecycle so features like job bookmarks work correctly.
  • Below the initialization code, paste the ETL logic that reads, transforms, and writes the data:
# Read JSON from raw zone
raw_dyf = glueContext.create_dynamic_frame.from_options(
    connection_type="s3",
    connection_options={"paths": ["s3://[[BUCKET_NAME=\"your-bucket-name\"]]/raw/matches/"], "recurse": True},
    format="json",
    transformation_ctx="raw_dyf"
)

# Convert to Spark DataFrame for transformations
df = raw_dyf.toDF()

# Add derived columns for analytics
df_transformed = df \
    .withColumn("tournament_year", year(to_date(col("date")))) \
    .withColumn("total_goals", coalesce(col("home_score"), lit(0)) + coalesce(col("away_score"), lit(0)))

# Write as Parquet with Snappy compression, partitioned by tournament_year
df_transformed.write \
    .mode("overwrite") \
    .partitionBy("tournament_year") \
    .option("compression", "snappy") \
    .parquet("s3://[[BUCKET_NAME=\"your-bucket-name\"]]/processed/matches/")

job.commit()

What does the ETL logic do?

  • create_dynamic_frame.from_options reads all JSON files from your raw zone into a Glue DynamicFrame.
  • toDF() converts to a standard Spark DataFrame so you can use PySpark functions.
  • tournament_year is derived by extracting the year from the date column. This becomes the partition key so Athena can skip entire years of data.
  • total_goals sums home and away scores, using coalesce to handle nulls gracefully.
  • The final write uses partitionBy("tournament_year") to create Hive-style tournament_year=YYYY/ directories, and Snappy compression to shrink file sizes.
  • job.commit() signals that the job completed successfully.
  • Click Save in the top right corner of the console.

Replace the bucket name

Make sure both S3 paths in the script use your actual bucket name from the previous project. The paths should match the bucket where your raw JSON data lives and where you created the processed/ prefix.

✔️ Awesome, I've got everything!

Great. Confirm you have saved the job with both chunks of code in the script editor.

ⓧ I'd like to double check the full code

Scroll up in the script editor and confirm the following sections appear in order:

  • The imports and job initialization block (starting with import sys and ending with job.init(args['JOB_NAME'], args)).
  • The ETL logic block (starting with the # Read JSON from raw zone comment and ending with job.commit()).

Both chunks together form the complete worldcup_etl.py script.

Run the job and verify the output

The script is saved. Time to execute it and confirm Parquet files land in your processed zone.

  • Click the Run button in the top right corner of the Glue console.
  • Click the Runs tab to monitor progress.

How long does this take?

For a small dataset, expect 1-2 minutes. The job spins up Spark workers, processes the JSON, writes Parquet, and shuts down. Wait until the Run status column shows Succeeded.

Before you verify the output, predict what you expect to see in the processed zone. Will the Parquet files be in one flat directory, or organized into subdirectories?

  • Verify the output by running this command in your terminal:
aws s3 ls s3://[[BUCKET_NAME="your-bucket-name"]]/processed/matches/ --recursive

You should see Parquet files organized in tournament_year=YYYY/ directories. Each directory contains the match data for that specific year, compressed with Snappy.

Job failed or no files appeared?

  • Check the Error logs link on the Runs tab. The most common issue is an incorrect S3 path in your script. Confirm the bucket name matches your actual bucket.
  • If you see a permissions error, verify that GlueETLRole has both s3:GetObject and s3:PutObject on your bucket. Return to IAM and check the inline policy.
  • If the job shows Running for more than 5 minutes, it may be stuck provisioning. Cancel it and re-run. Small datasets finish in under 2 minutes.

Why did my Glue ETL job fail or produce no output?

Your raw JSON is now optimized Parquet, partitioned by year and compressed with Snappy. Next up, you will catalog this new data so Athena can query it and measure just how much faster Parquet is.

Catalog the Processed Data and Measure the Difference

Your Glue ETL job has successfully written Parquet files into the processed zone. But those files are invisible to Athena right now.

Athena relies on the Glue Data Catalog to know where data lives, what format it uses, and how it is partitioned. Until a Crawler scans your Parquet output and registers a table, Athena has no way to query it. In this step, you will catalog the processed data and then run the same query against both formats to see how much less data Parquet needs to scan.

In this step, get ready to:
  • Create and run a Glue Crawler that registers your processed Parquet data in the Data Catalog.
  • Run a COUNT query on the raw JSON table and note how much data Athena scans.
  • Run the same query on the Parquet table, then add a partition filter and compare all three Data scanned values.
Create and run the Glue Crawler

A Glue Crawler automatically scans your S3 data, infers its schema, detects partitions, and writes table metadata into the Data Catalog. This is what makes your Parquet data queryable.

  • In the AWS Management Console, search for Glue and open the AWS Glue console.
  • In the left navigation pane, click Crawlers.
  • Click Create crawler.
  • For the crawler name, enter worldcup-processed-crawler.
  • Click Next.
  • Under Data source configuration, click Add a data source.
  • For Data source, select S3.
  • For S3 path, enter the path to your processed matches folder:
s3://[[BUCKETNAME="your-bucket-name"]]/processed/matches/
  • Click Add an S3 data source.
  • Click Next.
  • For Existing IAM role, select GlueETLRole (the role you created in Step 1).
  • Click Next.
  • Under Set output and scheduling, click Add database.
  • In the pop-up, enter worldcup_processed for the database name.
  • Click Create database.
  • For Target database, select worldcup_processed.
  • For Frequency, choose On demand.
  • Click Next.
  • Review the configuration and click Create crawler.

Now run the crawler to populate the Data Catalog.

  • Select your worldcup-processed-crawler in the list and click Run crawler.
  • Wait for the status to change to Completed. This typically takes under a minute for a small dataset.

Confirm the table was created by checking the Data Catalog.

  • In the left navigation pane, click Databases.
  • Click worldcup_processed.
  • You should see a table named matches listed with classification parquet.

What did the Crawler do?

The Crawler scanned your s3://your-bucket-name/processed/matches/ path, detected Parquet files, inferred the column schema, and identified the tournament_year=YYYY folder structure as Hive-style partitions. It registered all of this as table metadata in the Glue Data Catalog.

Athena queries the Data Catalog to understand where data lives and how to read it. Without this catalog entry, Athena would have no idea your Parquet files exist.

Crawler shows 0 tables added?

Make sure the S3 path ends with a trailing slash and points to processed/matches/, not processed/. The Crawler needs to target the folder that directly contains the partition directories.

Also verify that GlueETLRole has s3:GetObject permission on your bucket. Without it, the Crawler cannot read the files to infer their schema.

Help me debug this issue

Compare query costs between raw JSON and processed Parquet

Now for the payoff. You will run the same COUNT query on both your raw JSON table and the new Parquet table, then compare how much data each one scans.

  • In the AWS Management Console, search for Athena and open the Athena console.
  • In the Query Editor, make sure your workgroup has a query result location configured. If prompted, set one up in an S3 bucket.

First, query the raw JSON table. Select your original database from the database dropdown.

  • Paste this query into the editor and click Run:
SELECT COUNT(*) FROM your_raw_table_name;
  • After the query completes, look at the Data scanned value shown below the results. Write this number down.

Where is Data scanned?

After a query finishes, Athena displays the Data scanned metric in the query results area. This tells you how many bytes Athena read from S3 to answer your query. Athena charges $5 per TB scanned, so smaller scans mean lower cost.

Now query the Parquet table. Switch the database dropdown to worldcup_processed.

  • Paste this query and click Run:
SELECT COUNT(*) FROM matches;
  • Note the Data scanned value for this query. It should be significantly smaller than the raw JSON scan.

Why is Parquet so much smaller?

JSON stores every field name alongside every value in plain text. Athena must parse the entire file even if you only need one column.

Parquet is a columnar format. It stores data by column, applies compression (Snappy in your case), and embeds schema metadata in the file footer. For a COUNT(*), Athena only reads a tiny column-level metadata block instead of the entire dataset.

Demonstrate partition pruning

Partitioning divides your data into folders by a key value. When you filter on that key, Athena skips every folder that does not match. This is called partition pruning.

Before you run this next query, predict what you expect to see. Will the Data scanned go up, stay the same, or drop even further compared to the full Parquet COUNT?

  • In the Athena Query Editor (still using the worldcup_processed database), paste this query and click Run:
SELECT COUNT(*) FROM matches WHERE tournament_year = 2014;
  • Note the Data scanned value. It drops even further because Athena only read the tournament_year=2014 partition folder.

What is partition pruning?

Your ETL job wrote Parquet files into folders like tournament_year=2014/, tournament_year=2018/, and so on. When you add WHERE tournament_year = 2014 to your query, Athena reads only the matching folder and skips everything else.

This is why Hive-style partitioning is so powerful for analytics. The more partitions you skip, the less data you scan, and the less you pay.

Record all three Data scanned values for your portfolio documentation:

  • Raw JSON COUNT: the largest scan.
  • Parquet COUNT: significantly smaller.
  • Parquet COUNT with WHERE tournament_year = 2014: the smallest scan.

Now run the final verification query to confirm everything works end-to-end.

  • Paste this query and click Run:
SELECT * FROM worldcup_processed.matches WHERE tournament_year = 2014 LIMIT 5;
  • You should see 5 rows of match data including columns like date, home_team, away_team, home_score, away_score, tournament_year, and total_goals.

Query returns zero results?

If the Parquet table exists but returns no rows, the Crawler may not have detected the partitions. Go back to the Glue console, open the matches table, and check the Partitions tab. If it shows zero partitions, re-run the Crawler.

If the table does not appear at all, double-check that the database dropdown in Athena is set to worldcup_processed, not your original database.

Help me troubleshoot this

Your processed data is now fully cataloged and queryable. You have concrete proof that columnar Parquet with partitioning scans a fraction of the data compared to raw JSON. Next up, you will automate this pipeline so it runs on a schedule without manual intervention.

Automate with EventBridge Scheduler

In the previous step, you proved that querying processed Parquet data is dramatically cheaper than querying raw JSON. But right now, your pipeline only runs when you manually click Run in the Glue console.

A production pipeline needs to run on its own. EventBridge Scheduler triggers your Glue job on a recurring schedule so new data is automatically processed without human intervention. In this step, you will create a daily schedule, test it once, and then disable it to avoid any cost.

In this step, get ready to:
  • Create an EventBridge schedule targeting your Glue ETL job.
  • Test-fire the schedule and confirm the Glue job runs successfully.
  • Disable the schedule to prevent recurring charges.
Create the EventBridge schedule

EventBridge Scheduler supports calling thousands of API operations across AWS services. You will point it at the Glue StartJobRun operation to kick off your ETL automatically.

  • In the AWS Console search bar, type EventBridge Scheduler and select Amazon EventBridge Scheduler.
  • On the Schedules page, click Create schedule.
  • For Schedule name, enter worldcup-etl-daily.
  • Leave Schedule group set to default.
  • In the Schedule pattern section, for Occurrence, select Recurring schedule.
  • Select Rate-based schedule.
  • Enter 1 for the value and select day for the unit. This gives you a rate(1 day) expression.
  • For Flexible time window, choose Off.
  • Click Next.

What is a rate expression?

A rate expression like rate(1 day) tells EventBridge Scheduler to invoke the target once every 24 hours. You could also use a cron expression for more precise scheduling (e.g., every day at 8 AM UTC).

Setting the flexible time window to Off means the schedule fires at the exact interval. A flexible window would let AWS shift the invocation slightly to reduce burst load across the service.

Configure the target and permissions

Now you need to tell EventBridge which API operation to call and which IAM role to use when it fires.

  • On the Select target page, choose All APIs.
  • In the search box, type AWS Glue and select AWS Glue.
  • In the second search box, type StartJobRun and select StartJobRun.
  • In the StartJobRun section, for the JobName field, enter worldcup-json-to-parquet.
  • Click Next.
  • On the Settings page, under Schedule state, make sure the Enable schedule toggle is turned OFF. You will create it disabled first.
  • Scroll down to the Permissions section.
  • Select Use existing role.
  • From the dropdown, choose EventBridgeSchedulerGlueRole.
  • Click Next.
  • On the Review and create schedule page, verify your settings (name: worldcup-etl-daily, expression: rate(1 day), target: StartJobRun, role: EventBridgeSchedulerGlueRole).
  • Click Create schedule.

You should now see worldcup-etl-daily in your Schedules list with status Disabled.

Why create it disabled?

If you create a recurring schedule in the enabled state, it starts invoking immediately based on the rate. Each invocation triggers a Glue job run that costs ~$0.03. By creating it disabled, you control exactly when it fires for testing without racking up unexpected charges.

Don't see EventBridgeSchedulerGlueRole in the dropdown?

Make sure you created this role back in Step 1 with the trust policy for scheduler.amazonaws.com. The role must have that trust entity for it to appear in the EventBridge Scheduler permissions dropdown.

Help me troubleshoot my missing role

Enable the schedule, test it, and disable it again

Your schedule exists but is dormant. You will temporarily enable it to trigger one run, confirm success in the Glue console, and then disable it again.

  • On the Schedules page, click the name worldcup-etl-daily to open its detail page.
  • Click Edit.
  • Navigate to the Settings page and toggle Enable schedule to ON.
  • Click Save (or Update schedule) to apply the change.

The schedule is now enabled. It will fire within the rate interval. Wait 1-2 minutes for EventBridge to trigger the Glue job.

  • Switch back to the AWS Glue console.
  • In the left navigation, select ETL jobs.
  • Click on worldcup-json-to-parquet to open the job.
  • Select the Runs tab.

Before you check the result, predict: do you think you will see a new run here, or will it still show only the run from your manual test in Step 2?

You should see a new run entry with status Running or Succeeded. This run was triggered by EventBridge Scheduler, not by you clicking Run manually. If the status is still Running, wait a minute or two and refresh until it shows Succeeded.

How do I know EventBridge triggered this?

Check the Trigger column (or the run details). Runs triggered by EventBridge show a different trigger source than manual runs. You will also see the timestamp aligns with when you enabled the schedule, not when you clicked anything in the Glue console.

  • Once the run shows Succeeded, switch back to the EventBridge Scheduler console.
  • Click worldcup-etl-daily, then click Edit.
  • Navigate to Settings and toggle Enable schedule back to OFF.
  • Click Save to confirm.

Your schedule is now back in Disabled state. It will not fire again until you re-enable it, so no further Glue job costs will accrue.

Run not appearing in the Glue Runs tab?

If you do not see a new run after 2-3 minutes, check the following:

  • Confirm the schedule state is Enabled (not still Disabled).
  • Verify that EventBridgeSchedulerGlueRole has an inline policy allowing glue:StartJobRun on your job.
  • Check that the JobName in the target matches exactly: worldcup-json-to-parquet.

Help me troubleshoot my EventBridge schedule

Your pipeline now has automation built in. EventBridge Scheduler can trigger your Glue job daily, weekly, or at whatever cadence your data ingestion demands. Next up, you will build an interactive dashboard to visualize the processed data and prove end-to-end value.

Build a QuickSight Dashboard

Your pipeline transforms raw JSON into optimized Parquet and runs on a daily schedule. But data sitting in S3 is invisible to stakeholders who want to explore trends and ask questions about FIFA World Cup history.

A dashboard makes your pipeline's output tangible and proves you can deliver end-to-end from ingestion to insight. Amazon QuickSight connects directly to Athena, so your visuals always reflect the latest processed data without any additional data movement.

In this step, get ready to:
  • Sign up for Amazon QuickSight and authorize access to Athena and your S3 bucket.
  • Create a dataset from your Parquet-backed Athena table.
  • Build visualizations and publish an interactive dashboard.
Sign up for Amazon QuickSight

QuickSight is a serverless BI service that renders interactive dashboards from your Athena queries. You need an account before you can connect it to your data.

  • In the AWS Management Console, type QuickSight in the top search bar and select the service.
  • You will see a signup page. Click Sign up for QuickSight.
  • Select Enterprise edition.
  • Enter a unique account name (e.g. worldcup-analytics).
  • Enter your email address for notifications.
  • Choose your AWS region (use the same region as your S3 bucket and Athena setup).
  • Click Finish to create your account.

After a moment, your QuickSight console loads. You now have a 30-day free trial with all Enterprise features.

Cost warning

After 30 days, QuickSight charges $24 per Author per month. Set a calendar reminder to unsubscribe before your trial ends if you do not plan to continue using the service.

Now you need to authorize QuickSight to access Athena and your S3 bucket.

  • In the QuickSight console, click your account icon in the top-right corner.
  • Select Manage QuickSight.
  • Click Security & permissions in the left sidebar.
  • Under QuickSight access to AWS services, click Add or remove.
  • Check the box next to Amazon Athena.
  • Under Amazon S3, click Select S3 buckets and check the box next to your bucket (your-bucket-name).
  • Click Finish to save.

Return to the QuickSight home page by clicking the QuickSight logo in the top-left corner. Your QuickSight console should load with sample visuals on the landing page.

QuickSight console not loading?

If you see a permissions error, ensure your IAM user has the quicksight:CreateAdmin permission. If your browser shows a blank screen, try refreshing or using a different browser.

help me troubleshoot QuickSight access issues

Create a dataset from Athena

QuickSight needs a dataset that points to your Athena table. Once connected, it imports the data into SPICE (Super-fast, Parallel, In-memory Calculation Engine) for fast dashboard rendering.

  • In the QuickSight console, click Datasets in the left navigation.
  • Click New dataset in the top-right corner.
  • Select the Athena card from the list of data sources.
  • In the Data source name field, enter WorldCupPipeline.
  • Select your Athena workgroup (use primary if you have not changed it).
  • Click Create data source.

A table selection dialog appears. Now point QuickSight at your processed Parquet table.

  • For Catalog, select AwsDataCatalog.
  • For Database, select worldcup_processed (your Glue database containing the Parquet-backed table).
  • Select the matches table.
  • Click Select.

You are prompted to choose how to query the data.

  • Select Import to SPICE for quicker analytics.
  • Click Visualize.

QuickSight imports your data into SPICE. After a few seconds, a blank analysis sheet opens with a field list on the left showing your table columns.

What is SPICE?

SPICE is QuickSight's in-memory engine that caches your data for fast dashboard rendering. Instead of running a live Athena query every time someone views the dashboard, SPICE serves results from its cache. You can refresh the SPICE dataset manually or on a schedule to pick up new data.

Not seeing your database or table?

Return to Manage QuickSight and confirm Athena is checked under Security & permissions. Also verify your S3 bucket is selected. Without these, QuickSight cannot read your Glue Data Catalog.

help me fix QuickSight not showing my Athena database

Build visuals and publish the dashboard

Your analysis sheet is ready with a field list on the left. You will build three visuals that showcase different angles of your World Cup data, then publish them as a shareable dashboard.

  • In the Visual types pane (bottom-left of the analysis), click the vertical bar chart icon.
  • From the Fields list on the left, drag tournament_year to the X axis field well.
  • Drag total_goals to the Value field well. QuickSight automatically aggregates it as a sum.

You should see a bar chart with tournament years on the x-axis and total goals on the y-axis. Each bar represents the total goals scored in that World Cup year.

Now add a second visual showing which teams played the most matches.

  • Click Add at the top of the sheet, then select Add visual.
  • In the Visual types pane, click the horizontal bar chart icon.
  • Drag home_team to the Y axis field well.
  • Drag home_team again to the Value field well (it automatically counts occurrences).

A horizontal bar chart appears showing the count of home matches by team. You can see which countries hosted or played the most matches.

Finally, add a KPI visual showing the total number of matches in your dataset.

  • Click Add, then Add visual.
  • In the Visual types pane, select the KPI icon (a number with an arrow).
  • Drag any field (e.g. date) to the Value field well. It defaults to Count, which gives you total matches.

A large number appears showing the total count of matches in your dataset. Your analysis now has three visuals on one sheet.

What do these three visuals show?

The bar chart reveals how scoring trends changed across World Cup tournaments. The horizontal bar shows which nations appeared most often as the home team. The KPI gives a quick headline number for total matches in the dataset.

Together, they demonstrate three distinct chart types drawing from one Athena-backed dataset.

Now publish your analysis as a shareable dashboard.

  • Click Publish in the top-right corner of the analysis.
  • In the dialog, enter FIFA World Cup Analytics as the dashboard name.
  • Click Publish dashboard.

QuickSight publishes your dashboard. A sharing dialog may appear. You can close it for now.

Before you verify, what do you think will happen when you filter the dashboard by a specific tournament year? Will all three visuals update, or just the bar chart?

  • In the published dashboard, click the filter icon in the top toolbar.
  • Add a filter on tournament_year.
  • Select a specific year (e.g. 2014) and apply the filter.

All three visuals update to reflect only that tournament year. The bar chart shows a single bar, the horizontal bar shows only teams from that year, and the KPI count drops to that year's matches. Your dashboard is live and interactive.

Dashboard showing no data?

If your visuals are empty, return to Datasets and confirm the SPICE import succeeded. Click on your dataset and check for a green Import complete status. If the import failed, verify that QuickSight has permission to access your S3 bucket under Security & permissions.

help me troubleshoot empty QuickSight visuals

Secret mission

Build a Curated Aggregation Layer

Your processed zone has detailed row-level match data. Analysts want instant answers to questions like 'how many goals were scored in 2014?' without scanning every row. Build a curated aggregation layer that pre-computes tournament-level statistics and surfaces them in your QuickSight dashboard.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project uses several AWS services that charge per use or on a monthly basis.

Cost warning

Amazon QuickSight charges $24/month per Author after the 30-day free trial ends. If you do not cancel before your trial expires, you will be billed automatically. Glue jobs and crawlers only charge when they run, so they cost nothing while idle.

Resources you used:

  • AWS Glue ETL job: worldcup-json-to-parquet (charged per run, approximately $0.03 per run at 2 DPUs).
  • AWS Glue ETL job: worldcup-curated-aggregation (charged per run, approximately $0.03 per run at 2 DPUs).
  • AWS Glue Crawler: worldcup-processed-crawler (charged per run, approximately $0.01 per run).
  • EventBridge Schedule: worldcup-etl-daily (free while disabled).
  • Amazon QuickSight account (free for 30 days, then $24/month per Author).
  • S3 prefixes: processed/ and curated/ (negligible storage cost for small datasets).
  • IAM roles: GlueETLRole and EventBridgeSchedulerGlueRole (no cost, but good practice to remove unused roles).

Keep everything running

No action needed. Choose this if you are still actively building on this pipeline or exploring your QuickSight dashboard.

  • The EventBridge schedule is already disabled, so no Glue jobs will run automatically.
  • Your QuickSight free trial continues for 30 days from when you signed up.
  • S3 storage costs for your small dataset are less than $0.10/month.
  • Set a calendar reminder to cancel QuickSight before your 30-day trial ends to avoid the $24/month charge.

Pause - I'll come back to this later

Shut down running processes to free up resources while keeping your files and configurations intact.

  • The EventBridge schedule is already disabled from Step 4. No further action needed there.
  • Your Glue jobs, crawler, and IAM roles cost nothing while idle.
  • Your QuickSight free trial continues counting down. Set a reminder to cancel before it expires if you do not plan to use it.

Delete - I don't want to use this again

Remove all project resources. This is recommended after you have taken portfolio screenshots.

  • Delete the EventBridge schedule by running:
  • Delete the Glue ETL jobs by running:
  • Delete the Glue Crawler by running:
  • Delete the processed and curated S3 data by running:
  • Delete the IAM roles (you must detach policies first) by running the commands below.
  • Unsubscribe from QuickSight before your 30-day trial ends.
aws scheduler delete-schedule --name worldcup-etl-daily
aws glue delete-job --job-name worldcup-json-to-parquet
aws glue delete-job --job-name worldcup-curated-aggregation
aws glue delete-crawler --name worldcup-processed-crawler
aws s3 rm s3://your-bucket-name/processed/ --recursive
aws s3 rm s3://your-bucket-name/curated/ --recursive

To delete the GlueETLRole role, first detach the managed policy and delete the inline policy:

aws iam detach-role-policy --role-name GlueETLRole --policy-arn arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole
aws iam delete-role-policy --role-name GlueETLRole --policy-name S3AccessPolicy
aws iam delete-role --role-name GlueETLRole

To delete the EventBridgeSchedulerGlueRole role, delete its inline policy first:

aws iam delete-role-policy --role-name EventBridgeSchedulerGlueRole --policy-name GlueStartJobPolicy
aws iam delete-role --role-name EventBridgeSchedulerGlueRole

To unsubscribe from QuickSight, open this URL in your browser while signed in to AWS:

https://us-east-1.quicksight.aws.amazon.com/sn/console/unsubscribe
  • Follow the on-screen prompts to confirm account termination.
  • This deletes all QuickSight data, analyses, and dashboards permanently.

That's a Wrap!

That's a Wrap!

Nice work! You've just built a fully automated analytics pipeline that transforms raw FIFA World Cup data into optimized, queryable insights delivered through an interactive dashboard.

You've learned how to:

  • Design a multi-zone data lake and write a PySpark ETL job that converts JSON to Parquet with Snappy compression and Hive-style partitioning.
  • Measure query cost optimization by comparing data scanned across raw JSON, Parquet, and partition-pruned Parquet in Athena.
  • Automate a pipeline with EventBridge Scheduler and deliver end-to-end insights through a published Amazon QuickSight dashboard connected to your optimized data.
  • Secret Mission: Created a curated aggregation layer with pre-computed tournament statistics and surfaced them in a summary dashboard sheet.

Ready to quiz yourself?