Crypto Market Data Pipeline on AWS

Build a containerized ETL pipeline that ingests live crypto data into S3.

Introduction

30 Second Summary

Cryptocurrency prices change every second. Behind every trading app and market dashboard is a data pipeline that captures, cleans, and stores that data before anyone can analyze it.

In this project, you will build a production-grade data pipeline that pulls live cryptocurrency market data, validates its quality, and stores it in AWS S3 as analytics-ready files. You'll containerize the pipeline with Docker and automate deployment with GitHub Actions CI/CD.

What You'll Build

A containerized Python ETL pipeline that ingests live crypto market data from the CoinGecko API, validates data quality, transforms it into Parquet format, and deploys to AWS via CI/CD.

By the end of this project, you'll have:

  • A Python ETL pipeline that extracts live cryptocurrency prices, validates data quality, and stores clean Parquet files in S3 with date partitioning.
  • A Docker container packaging your pipeline for consistent, reproducible execution anywhere.
  • A GitHub Actions CI/CD workflow that automatically builds and pushes your Docker image to AWS ECR on every commit to main.
  • Secret Mission: Add automated daily scheduling with AWS EventBridge so your pipeline runs every 24 hours without manual intervention.

Are there any prerequisites?

You'll need a GitHub account, Docker Desktop installed, and Python 3.11+ installed on your machine. AWS account creation and AWS CLI setup are covered in Step 1.

Set Up AWS Account and CLI

Before you can build a data pipeline that stores files in the cloud, you need an AWS account and the tools to talk to it from your terminal.

In this step, you will create an AWS Free Tier account, set up an IAM user with the right permissions, install the AWS CLI, and provision the two cloud resources your pipeline needs.

In this step, get ready to:
  • Create an AWS account and IAM user with programmatic access.
  • Install and configure the AWS CLI v2.
  • Create an S3 bucket and ECR repository for your pipeline.
Create an AWS account and IAM user

AWS Free Tier gives new accounts up to $200 in credits over 6 months. That is more than enough to cover everything in this project at zero cost.

  • Open https://aws.amazon.com/free/ in your browser.
  • Click Create a Free Account.
  • Follow the signup wizard. When prompted to choose a support plan, select the Free plan.
  • Once you are signed in to the AWS Management Console, search for IAM in the top search bar and open it.
  • In the left sidebar, click Users, then click Create user.
  • Enter a username (e.g. crypto-pipeline-user) and click Next.
  • Select Attach policies directly.
  • Search for and check the box next to AmazonS3FullAccess.
  • Search for and check the box next to AmazonEC2ContainerRegistryFullAccess.
  • Click Next, then click Create user.

Now create an access key so the CLI can authenticate as this user.

  • Click the username you just created to open the user details page.
  • Click the Security credentials tab.
  • Scroll down to Access keys and click Create access key.
  • Select Command Line Interface (CLI) as the use case.
  • Check the confirmation box and click Next.
  • Click Create access key.
  • Copy the Access key ID and Secret access key and save them somewhere safe (you will need both in the next substep).

Save your credentials now

AWS only shows the secret access key once. If you close this page without copying it, you will need to delete the key and create a new one.

What is an IAM user?

IAM (Identity and Access Management) lets you create users with specific permissions instead of using your root account for everything. The two policies you attached grant this user permission to manage S3 buckets and ECR repositories.

Install and configure the AWS CLI v2

The AWS CLI lets you create and manage AWS resources directly from your terminal. You need version 2 for this project.

  • Check if the AWS CLI is already installed by running this command in your terminal:
aws --version

✔️ I see aws-cli/2.x.x

Your AWS CLI v2 is ready. Skip ahead to the configuration step below.

ⓧ I see aws-cli/1.x.x

You have version 1 installed, which is missing features this project needs. Install version 2 alongside it using the platform instructions below.

macOS

  • Download and install the AWS CLI v2 package by running these commands:
curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
sudo installer -pkg AWSCLIV2.pkg -target /
  • Close and reopen your terminal, then run aws --version again to confirm you see aws-cli/2.x.x.

Windows

  • Download and run the MSI installer by running this command in an elevated Command Prompt (right-click Command Prompt and select Run as administrator):
msiexec.exe /i https://awscli.amazonaws.com/AWSCLIV2.msi
  • Close and reopen your terminal, then run aws --version again to confirm you see aws-cli/2.x.x.

Linux

  • Download and install the AWS CLI v2 by running these commands:
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
  • Close and reopen your terminal, then run aws --version again to confirm you see aws-cli/2.x.x.

ⓧ Command not found

The AWS CLI is not installed yet. Install version 2 using the instructions for your platform below.

macOS

  • Download and install the AWS CLI v2 package by running these commands:
curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
sudo installer -pkg AWSCLIV2.pkg -target /
  • Close and reopen your terminal, then run aws --version to confirm you see aws-cli/2.x.x.

Windows

  • Download and run the MSI installer by running this command in an elevated Command Prompt (right-click Command Prompt and select Run as administrator):
msiexec.exe /i https://awscli.amazonaws.com/AWSCLIV2.msi
  • Close and reopen your terminal, then run aws --version to confirm you see aws-cli/2.x.x.

Linux

  • Download and install the AWS CLI v2 by running these commands:
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
  • Close and reopen your terminal, then run aws --version to confirm you see aws-cli/2.x.x.

Still seeing version 1 or command not found?

Make sure you closed and reopened your terminal after installation. On macOS, if aws still points to v1, try running /usr/local/bin/aws --version to confirm v2 is installed at the expected path.

Help me troubleshoot my AWS CLI installation.

Now configure the CLI with your IAM credentials.

  • Configure your credentials by running this command:
aws configure
  • When prompted for AWS Access Key ID, paste the access key ID you saved earlier.
  • When prompted for AWS Secret Access Key, paste the secret access key you saved earlier.
  • When prompted for Default region name, enter us-east-1.
  • When prompted for Default output format, press Enter to accept the default.

What does aws configure do?

This command stores your credentials in a file at ~/.aws/credentials and your region preference in ~/.aws/config. Every AWS CLI command you run after this will automatically use these credentials.

  • Verify the configuration works by running:
aws sts get-caller-identity

You should see a JSON response showing your account ID, user ARN, and user ID. This confirms the CLI can authenticate with AWS.

Seeing 'InvalidClientTokenId' or 'SignatureDoesNotMatch'?

Double-check that you pasted the access key ID and secret key correctly with no extra spaces. Run aws configure again to re-enter them.

Help me fix my AWS CLI authentication error.

Create the S3 bucket and ECR repository

Your pipeline needs two cloud resources: an S3 bucket to store the output Parquet files, and an ECR repository to store your Docker images.

  • Create the S3 bucket by running this command (replace your initials with your actual initials to make the bucket name unique):
aws s3 mb s3://crypto-pipeline-data-[[INITIALS="your-initials"]]

Why do I need my initials in the bucket name?

S3 bucket names must be globally unique across all AWS accounts worldwide. Adding your initials ensures your bucket name does not conflict with anyone else's.

  • Create the ECR repository by running this command:
aws ecr create-repository --repository-name crypto-pipeline --region us-east-1

What is ECR?

Amazon Elastic Container Registry (ECR) is a Docker image registry hosted by AWS. When your CI/CD pipeline builds a Docker image, it pushes the image here so it can be pulled and run anywhere in your AWS environment.

ECR gives you 500 MB of private storage free for the first year. This project will use far less than that.

  • Verify both resources were created by running these commands:
aws s3 ls
aws ecr describe-repositories --region us-east-1

You should see your bucket name (crypto-pipeline-data- followed by your initials) in the S3 output, and crypto-pipeline listed in the ECR output.

Seeing 'BucketAlreadyExists'?

Someone else already has a bucket with that name. Try adding more characters to make it unique (e.g. your full name or a number).

Help me fix my S3 bucket creation error.

Your AWS environment is fully set up. Next up, you will build the Python ETL pipeline that ingests live cryptocurrency data and stores it in the S3 bucket you just created.

Build the ETL Pipeline

Your AWS environment is ready with an S3 bucket and ECR repository waiting for data. Now it's time to build the pipeline that fills them.

In this step, you'll write a Python ETL pipeline that pulls live cryptocurrency market data from the CoinGecko API, validates its quality, transforms it into analytics-ready Parquet format, and uploads it to your S3 bucket with date-based partitioning.

In this step, get ready to:
  • Create the project directory structure and install dependencies.
  • Write extraction, validation, and transformation functions.
  • Write the load function and run the full pipeline against your S3 bucket.
Create the project structure and extraction function

Every data pipeline starts with a clean project structure. You'll set up a folder for your pipeline code, declare your dependencies, and write the first function that pulls live market data.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and the pipeline subdirectory by running these commands:
mkdir crypto-pipeline
cd crypto-pipeline
mkdir pipeline
  • Confirm the folder exists by running:
ls

You should see pipeline/ listed in the output.

  • Create the requirements.txt file by running:
touch requirements.txt
  • Open the crypto-pipeline folder in your code editor.
  • Open requirements.txt and paste the following dependencies:
requests==2.34.2
pandas==3.0.3
pyarrow==24.0.0
boto3

What are these dependencies?

  • requests handles HTTP calls to the CoinGecko API.
  • pandas provides DataFrames for data transformation.
  • pyarrow enables writing Parquet columnar files.
  • boto3 is the AWS SDK for Python, used to upload files to S3.
  • Save requirements.txt.
  • Install the dependencies by running:
pip install -r requirements.txt

You should see download progress bars followed by a line starting with Successfully installed listing all four packages and their sub-dependencies.

Install failed or command not found?

  • If pip is not recognized, try pip3 instead.
  • If you see a permissions error, add --user to the end of the command.
  • Make sure you are inside the crypto-pipeline folder where requirements.txt lives.

Help me troubleshoot my pip install failure.

  • Create the main pipeline file by running:
touch pipeline/etl.py
  • Open pipeline/etl.py in your editor and paste the following module header:
import requests
import pandas as pd
import boto3
import os
import time
import logging
from datetime import datetime, timezone

logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s")
logger = logging.getLogger(__name__)

S3_BUCKET = os.environ.get("S3_BUCKET", "crypto-pipeline-data")
COINGECKO_URL = "https://api.coingecko.com/api/v3/coins/markets"

What does this code do?

  • The imports bring in everything the pipeline needs: HTTP requests, data manipulation, AWS access, and logging.
  • logging.basicConfig configures timestamped log messages so you can trace pipeline progress.
  • S3_BUCKET reads your bucket name from an environment variable, falling back to a default.
  • COINGECKO_URL stores the CoinGecko API endpoint for the markets list.

You won't see a change yet. The next chunk adds the extraction function that calls this API.

  • Below the COINGECKO_URL line, add the extraction function:


def extract():
    """Extract top 10 cryptocurrency market data from CoinGecko."""
    params = {
        "vs_currency": "usd",
        "per_page": 10,
        "page": 1,
    }
    headers = {"accept": "application/json"}

    for attempt in range(3):
        response = requests.get(COINGECKO_URL, params=params, headers=headers, timeout=30)
        if response.status_code == 200:
            logger.info("Successfully extracted %d records from CoinGecko", len(response.json()))
            return response.json()
        elif response.status_code == 429:
            wait_time = 2 ** attempt * 10
            logger.warning("Rate limited (429). Retrying in %d seconds...", wait_time)
            time.sleep(wait_time)
        else:
            response.raise_for_status()

    raise Exception("Failed to extract data after 3 attempts")

What does the extract function do?

  • It calls the CoinGecko Keyless Public API to fetch live market data for the top 10 cryptocurrencies by market cap.
  • The for attempt in range(3) loop implements retry logic. If the API returns a 429 (rate limit) response, it waits exponentially longer before trying again.
  • Any other non-200 status raises an exception immediately.

Why no API key?

The CoinGecko Keyless Public API requires no signup and no API key. It has a rate limit of roughly 10-30 calls per minute, which is more than enough for a pipeline that runs once daily.

  • Save pipeline/etl.py.

You won't see a change yet. The next functions validate and transform the data this function returns.

Write the validation and transformation functions

Raw API data can contain nulls, missing fields, or stale records. The validation function catches these quality issues before the data reaches your storage layer.

  • In pipeline/etl.py, below the extract() function, add the validation and transformation functions:


def validate(data):
    """Validate data quality: null checks, schema validation, staleness detection."""
    required_fields = ["id", "current_price", "market_cap", "total_volume", "last_updated"]
    issues = []

    for i, record in enumerate(data):
        for field in required_fields:
            if record.get(field) is None:
                issues.append(f"Record {i} ({record.get('id', 'unknown')}): null {field}")

        if record.get("last_updated"):
            last_updated = datetime.fromisoformat(record["last_updated"].replace("Z", "+00:00"))
            age_minutes = (datetime.now(timezone.utc) - last_updated).total_seconds() / 60
            if age_minutes > 60:
                issues.append(f"Record {i} ({record.get('id', 'unknown')}): stale data ({age_minutes:.0f} min old)")

    if issues:
        logger.warning("Data quality issues found: %s", issues)
    else:
        logger.info("Data quality validation passed for %d records", len(data))

    return len(issues) == 0, issues


def transform(data):
    """Transform raw JSON into a clean pandas DataFrame."""
    columns = [
        "id", "symbol", "name", "current_price", "market_cap",
        "total_volume", "price_change_percentage_24h", "last_updated",
    ]
    df = pd.DataFrame(data)[columns]
    df["ingestion_timestamp"] = datetime.now(timezone.utc).isoformat()
    df["last_updated"] = pd.to_datetime(df["last_updated"])
    logger.info("Transformed data into DataFrame with shape %s", df.shape)
    return df

What do these functions do?

validate() checks every record for three types of quality problems:

  • Null values in critical fields like current_price and market_cap.
  • Missing required columns (schema validation).
  • Stale data where last_updated is older than 60 minutes.

transform() selects only the columns you need, adds an ingestion_timestamp to track when the pipeline ran, and converts the result into a pandas DataFrame.

  • Save pipeline/etl.py.

You won't see a change yet. The next function loads this DataFrame into S3 as a Parquet file.

Write the load function and test the pipeline

The load function is the final stage. It converts your DataFrame to a Parquet file, organizes it with date-based partitioning, and uploads it to S3.

  • In pipeline/etl.py, below the transform() function, add the load function and main entrypoint:


def load(df):
    """Save DataFrame as Parquet and upload to S3 with date partitioning."""
    now = datetime.now(timezone.utc)
    partition_path = f"market_data/year={now.year}/month={now.month:02d}/day={now.day:02d}"
    filename = f"crypto_market_{now.strftime('%Y%m%d_%H%M%S')}.parquet"
    local_path = f"/tmp/{filename}"

    df.to_parquet(local_path, engine="pyarrow", index=False)
    logger.info("Saved Parquet file locally: %s", local_path)

    s3_key = f"{partition_path}/{filename}"
    s3_client = boto3.client("s3")
    s3_client.upload_file(local_path, S3_BUCKET, s3_key)
    logger.info("Uploaded to s3://%s/%s", S3_BUCKET, s3_key)

    os.remove(local_path)
    return s3_key


def main():
    """Orchestrate the ETL pipeline."""
    logger.info("Starting crypto market data pipeline")

    data = extract()

    is_valid, issues = validate(data)
    if not is_valid:
        logger.warning("Proceeding with %d quality issues", len(issues))

    df = transform(data)

    s3_key = load(df)

    logger.info("Pipeline complete. Output: s3://%s/%s", S3_BUCKET, s3_key)


if __name__ == "__main__":
    main()

What do load and main do?

load() saves the DataFrame as a Parquet file with date partitioning (year=YYYY/month=MM/day=DD). It then uploads the file to your S3 bucket using boto3 and cleans up the temporary local copy.

main() orchestrates the full pipeline in order: extract, validate, transform, load. It logs progress at each stage so you can trace exactly what happened during a run.

Why Parquet instead of CSV?

Parquet is a columnar storage format that compresses data much more efficiently than CSV. Analytics tools like AWS Athena and Spark can read specific columns from Parquet without scanning the entire file, making queries faster and cheaper.

  • Save pipeline/etl.py.

✔️ Awesome, I've got everything!

Great. Make sure you've saved pipeline/etl.py with all functions in place.

ⓧ I'd like to double check the full code

Here is the complete pipeline/etl.py file. Compare it against yours:

import requests
import pandas as pd
import boto3
import os
import time
import logging
from datetime import datetime, timezone

logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s")
logger = logging.getLogger(__name__)

S3_BUCKET = os.environ.get("S3_BUCKET", "crypto-pipeline-data")
COINGECKO_URL = "https://api.coingecko.com/api/v3/coins/markets"


def extract():
    """Extract top 10 cryptocurrency market data from CoinGecko."""
    params = {
        "vs_currency": "usd",
        "per_page": 10,
        "page": 1,
    }
    headers = {"accept": "application/json"}

    for attempt in range(3):
        response = requests.get(COINGECKO_URL, params=params, headers=headers, timeout=30)
        if response.status_code == 200:
            logger.info("Successfully extracted %d records from CoinGecko", len(response.json()))
            return response.json()
        elif response.status_code == 429:
            wait_time = 2 ** attempt * 10
            logger.warning("Rate limited (429). Retrying in %d seconds...", wait_time)
            time.sleep(wait_time)
        else:
            response.raise_for_status()

    raise Exception("Failed to extract data after 3 attempts")


def validate(data):
    """Validate data quality: null checks, schema validation, staleness detection."""
    required_fields = ["id", "current_price", "market_cap", "total_volume", "last_updated"]
    issues = []

    for i, record in enumerate(data):
        for field in required_fields:
            if record.get(field) is None:
                issues.append(f"Record {i} ({record.get('id', 'unknown')}): null {field}")

        if record.get("last_updated"):
            last_updated = datetime.fromisoformat(record["last_updated"].replace("Z", "+00:00"))
            age_minutes = (datetime.now(timezone.utc) - last_updated).total_seconds() / 60
            if age_minutes > 60:
                issues.append(f"Record {i} ({record.get('id', 'unknown')}): stale data ({age_minutes:.0f} min old)")

    if issues:
        logger.warning("Data quality issues found: %s", issues)
    else:
        logger.info("Data quality validation passed for %d records", len(data))

    return len(issues) == 0, issues


def transform(data):
    """Transform raw JSON into a clean pandas DataFrame."""
    columns = [
        "id", "symbol", "name", "current_price", "market_cap",
        "total_volume", "price_change_percentage_24h", "last_updated",
    ]
    df = pd.DataFrame(data)[columns]
    df["ingestion_timestamp"] = datetime.now(timezone.utc).isoformat()
    df["last_updated"] = pd.to_datetime(df["last_updated"])
    logger.info("Transformed data into DataFrame with shape %s", df.shape)
    return df


def load(df):
    """Save DataFrame as Parquet and upload to S3 with date partitioning."""
    now = datetime.now(timezone.utc)
    partition_path = f"market_data/year={now.year}/month={now.month:02d}/day={now.day:02d}"
    filename = f"crypto_market_{now.strftime('%Y%m%d_%H%M%S')}.parquet"
    local_path = f"/tmp/{filename}"

    df.to_parquet(local_path, engine="pyarrow", index=False)
    logger.info("Saved Parquet file locally: %s", local_path)

    s3_key = f"{partition_path}/{filename}"
    s3_client = boto3.client("s3")
    s3_client.upload_file(local_path, S3_BUCKET, s3_key)
    logger.info("Uploaded to s3://%s/%s", S3_BUCKET, s3_key)

    os.remove(local_path)
    return s3_key


def main():
    """Orchestrate the ETL pipeline."""
    logger.info("Starting crypto market data pipeline")

    data = extract()

    is_valid, issues = validate(data)
    if not is_valid:
        logger.warning("Proceeding with %d quality issues", len(issues))

    df = transform(data)

    s3_key = load(df)

    logger.info("Pipeline complete. Output: s3://%s/%s", S3_BUCKET, s3_key)


if __name__ == "__main__":
    main()

Now run the pipeline to confirm it can pull live data and write to S3.

  • Set the S3_BUCKET environment variable to your bucket name and run the pipeline:
export S3_BUCKET=[[BUCKET_NAME="crypto-pipeline-data-your-initials"]]
python pipeline/etl.py

You should see log lines tracking each pipeline stage, ending with a line like Pipeline complete. Output: s3://... confirming the upload succeeded.

Seeing an error when running the pipeline?

  • If you see a NoCredentialsError, your AWS CLI credentials are not configured. Run aws configure and re-enter your IAM user's access key and secret key.
  • If you see a NoSuchBucket error, double-check that the S3_BUCKET value matches the exact bucket name you created in Step 1.
  • If you see ModuleNotFoundError, make sure you ran pip install -r requirements.txt successfully in the crypto-pipeline folder.

Help me debug my ETL pipeline error.

  • Verify the Parquet file landed in your S3 bucket by running:
aws s3 ls s3://[[BUCKET_NAME="crypto-pipeline-data-your-initials"]]/ --recursive

You should see output with a date, file size, and a path like market_data/year=2026/month=07/day=08/crypto_market_20260708_HHMMSS.parquet (with today's date and time).

Your ETL pipeline is pulling live crypto data and storing clean Parquet files in S3. Next up, you'll containerize this pipeline with Docker so it runs identically anywhere.

Containerize with Docker

Your ETL pipeline runs perfectly on your local machine. But what happens when you need to run it on a different machine, a cloud server, or inside a CI/CD workflow? Different Python versions, missing packages, and OS quirks can break things silently.

Docker solves this by packaging your code, dependencies, and runtime into a single container image that runs identically everywhere. In this step, you'll create a Dockerfile for your pipeline and verify the container writes to S3 just like the local version does.

In this step, get ready to:
  • Create a Dockerfile that packages the pipeline and its dependencies.
  • Build the Docker image locally.
  • Run the container and confirm it writes a new Parquet file to S3.
Create the Dockerfile

A Dockerfile is a recipe that tells Docker exactly how to build your container image. It specifies the base operating system, installs your dependencies, copies your code, and defines what command runs when the container starts.

  • In your code editor, create a new file called Dockerfile in the root of your project directory (the same level as requirements.txt).
  • Paste the following code into Dockerfile:
FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY pipeline/ pipeline/

ENTRYPOINT ["python", "pipeline/etl.py"]

What does this Dockerfile do?

  • FROM python:3.11-slim starts from a lightweight Python 3.11 image. The slim variant keeps the image small by excluding development tools you don't need at runtime.
  • WORKDIR /app sets the working directory inside the container. All subsequent commands run from this path.
  • COPY requirements.txt . and RUN pip install install dependencies first. Docker caches this layer, so rebuilds are fast unless your dependencies change.
  • COPY pipeline/ pipeline/ copies your ETL code into the image.
  • ENTRYPOINT defines what runs when the container starts. Your pipeline executes automatically with no manual intervention.
  • Save Dockerfile.
  • Confirm the file exists by running:
ls Dockerfile

You should see Dockerfile listed in the output.

Don't see the file?

  • Make sure the file is named exactly Dockerfile with a capital D and no file extension.
  • Confirm you're in the root of your project directory (the folder containing requirements.txt).

Still stuck? Help me find why my Dockerfile isn't appearing in my project directory.

Why Docker instead of just running Python directly?

Running python pipeline/etl.py locally works today. But when you move to CI/CD or a cloud service, you'd need to install Python, pip, and every dependency from scratch each time.

A Docker image bundles everything into a single artifact. The same image runs on your laptop, in GitHub Actions, and on AWS. No "works on my machine" surprises.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file.

ⓧ I'd like to double check the full code

FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY pipeline/ pipeline/

ENTRYPOINT ["python", "pipeline/etl.py"]
Build and test the container

With the Dockerfile in place, you can now build an image and run it. The container needs AWS credentials to upload files to S3, so you'll pass those as environment variables.

  • Build the Docker image by running this command:
docker build -t crypto-pipeline .

What does this command do?

  • docker build reads your Dockerfile and assembles the image layer by layer.
  • -t crypto-pipeline tags the image with the name crypto-pipeline so you can reference it easily.
  • . tells Docker to use the current directory as the build context (where it looks for the Dockerfile and files to copy).

You should see Docker downloading the base image, installing pip packages, and finishing with a success message. This may take a minute or two the first time.

Build failing?

  • Make sure Docker Desktop is running. You can check by looking for the Docker whale icon in your system tray.
  • Verify your requirements.txt is in the same directory as your Dockerfile. Docker can only copy files from the build context (the current directory).
  • If you see a network error, check your internet connection. Docker needs to download the python:3.11-slim base image.

Still stuck? Help me debug my Docker build failure for the crypto-pipeline image.

Now you need to run the container. Docker containers are isolated from your host machine, so they can't access your ~/.aws/credentials file. You'll pass your AWS credentials as environment variables instead.

  • Export your AWS credentials and bucket name as environment variables by running these commands:
export AWS_ACCESS_KEY_ID=your-access-key-id
export AWS_SECRET_ACCESS_KEY=your-secret-access-key
export AWS_DEFAULT_REGION=us-east-1
export S3_BUCKET=[[BUCKET_NAME="crypto-pipeline-data-your-initials"]]

Where do I find my access key and secret key?

These are the same credentials you used when running aws configure in Step 1. If you saved them somewhere safe at the time, grab them from there.

If you've lost them, you can create a new access key from the IAM console for your user. The old key pair cannot be retrieved.

  • Run the containerized pipeline by running this command:
docker run --env AWS_ACCESS_KEY_ID --env AWS_SECRET_ACCESS_KEY --env AWS_DEFAULT_REGION --env S3_BUCKET crypto-pipeline

What does this command do?

The --env VARNAME flag (without a value) passes the environment variable from your host shell into the container. Docker reads the value you just exported and forwards it.

This means your credentials never get baked into the image. They only exist at runtime, which is the production-safe pattern for handling secrets in containers.

  • Watch the terminal output. You should see the pipeline's log messages ending with a line like:
Pipeline complete. Output: s3://crypto-pipeline-data-your-initials/market_data/year=2026/month=07/day=08/crypto_market_20260708_XXXXXX.parquet
  • Verify the new Parquet file landed in S3 by running:
aws s3 ls s3://[[BUCKET_NAME="crypto-pipeline-data-your-initials"]]/ --recursive

You should see at least two Parquet files listed. One from when you ran the pipeline locally in the previous step, and a new one from this containerized run.

Container exiting with an error?

  • If you see a credentials error, double check that you exported all four environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION, S3_BUCKET) in the same terminal session where you ran docker run.
  • If you see a bucket-not-found error, verify your S3_BUCKET value matches the exact bucket name you created (including your initials).
  • If you see a rate-limit (429) error, wait 30 seconds and run the container again. The CoinGecko API has rate limits that reset quickly.

Still stuck? Help me debug why my Docker container is failing to write to S3.

Your pipeline now runs inside a portable container that behaves the same regardless of where it executes. Next up, you'll automate building and pushing this image to AWS ECR using GitHub Actions, so every code change triggers a fresh deployment.

Set Up CI/CD with GitHub Actions

Your pipeline runs inside a Docker container and produces clean Parquet files in S3. The next step is making this repeatable without manual intervention.

Every time you improve your pipeline code, you want the updated Docker image built and stored automatically. In this step, you will create a GitHub Actions workflow that builds your Docker image and pushes it to ECR on every commit to main.

In this step, get ready to:
  • Initialize a Git repository and push your project to GitHub.
  • Create a GitHub Actions workflow that builds and pushes your Docker image to ECR.
  • Verify the CI/CD pipeline runs successfully on push.
Initialize the repository and configure secrets

Before your workflow can run, you need a Git repository on GitHub with your AWS credentials stored securely as secrets.

  • Initialize a Git repository in your project folder by running this command:
git init -b main
  • Stage and commit all your project files by running:
git add .
git commit -m "Initial commit"

What do these commands do?

The git init -b main command creates a new Git repository with main as the default branch name. Then git add . stages all files, and git commit saves them as your first snapshot.

Now create a remote repository on GitHub to host your code.

  • Open your browser and go to github.com/new.
  • Type crypto-pipeline in the Repository name field.
  • Select Public visibility.
  • Do NOT check any initialization options (no README, no .gitignore, no license).
  • Click Create repository.
  • Copy the repository URL from the Quick Setup page, then link your local project to GitHub by running:
git remote add origin [[YOUR_REPO_URL="https://github.com/your-username/crypto-pipeline.git"]]

Next, store your AWS credentials as GitHub Secrets so the workflow can authenticate with AWS.

  • In your new repository on GitHub, click Settings.
  • In the left sidebar under the Security section, click Secrets and variables, then click Actions.
  • Click the Secrets tab.
  • Click New repository secret.
  • In the Name field, type AWS_ACCESS_KEY_ID.
  • In the Secret field, paste your IAM user's access key ID.
  • Click Add secret.
  • Click New repository secret again.
  • In the Name field, type AWS_SECRET_ACCESS_KEY.
  • In the Secret field, paste your IAM user's secret access key.
  • Click Add secret.

Why use GitHub Secrets?

Secrets are encrypted environment variables that GitHub injects into your workflow at runtime. They never appear in logs or in your code, which keeps your AWS credentials safe even in a public repository.

  • Verify you see both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY listed on the Actions secrets page.

Don't see the Settings tab?

The Settings tab only appears if you are the repository owner. Make sure you created the repository under your own account, not an organization you don't have admin access to.

Help me find Settings in my GitHub repository

Write the GitHub Actions workflow

GitHub Actions workflows are YAML files stored in .github/workflows/. Your workflow will check out the code, authenticate with AWS, log in to ECR, then build and push the Docker image.

  • Create the workflow directory and file by running:
mkdir -p .github/workflows
  • Create a new file called deploy.yml inside .github/workflows/ and paste the following workflow:
name: Build and Push to ECR

on:
  push:
    branches:
      - main

env:
  AWS_REGION: us-east-1
  ECR_REPOSITORY: crypto-pipeline

jobs:
  build-and-push:
    name: Build and Push
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v6

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@0e613a0980cbf65ed5b322eb7a1e075d28913a83
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ env.AWS_REGION }}

      - name: Login to Amazon ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@62f4f872db3836360b72999f4b87f1ff13310f3a

      - name: Build, tag, and push image to Amazon ECR
        id: build-image
        env:
          ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
          IMAGE_TAG: ${{ github.sha }}
        run: |
          docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .
          docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
          echo "image=$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG" >> $GITHUB_OUTPUT

What does this workflow do?

  • The on: push: branches: - main block triggers the workflow every time you push to main.
  • The env block sets AWS_REGION and ECR_REPOSITORY as reusable variables across the workflow.
  • The actions/checkout@v6 step clones your repository code into the runner.
  • The configure-aws-credentials step reads your GitHub Secrets to authenticate with AWS.
  • The amazon-ecr-login step logs Docker into your ECR registry.
  • The final step builds the image, tags it with the git commit SHA (github.sha), and pushes it to ECR.

Why pin actions to commit SHAs?

Using full commit SHA references like @0e613a0980cbf65ed5b322eb7a1e075d28913a83 instead of version tags prevents supply-chain attacks. A tag can be moved to point to malicious code, but a commit SHA is immutable.

  • Save the .github/workflows/deploy.yml file.

✔️ Awesome, I've got everything!

Great. Double check you have saved your file before moving on.

ⓧ I'd like to double check the full code

name: Build and Push to ECR

on:
  push:
    branches:
      - main

env:
  AWS_REGION: us-east-1
  ECR_REPOSITORY: crypto-pipeline

jobs:
  build-and-push:
    name: Build and Push
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v6

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@0e613a0980cbf65ed5b322eb7a1e075d28913a83
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ env.AWS_REGION }}

      - name: Login to Amazon ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@62f4f872db3836360b72999f4b87f1ff13310f3a

      - name: Build, tag, and push image to Amazon ECR
        id: build-image
        env:
          ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
          IMAGE_TAG: ${{ github.sha }}
        run: |
          docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .
          docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
          echo "image=$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG" >> $GITHUB_OUTPUT
Push to GitHub and verify the workflow

With the workflow file committed, pushing to main will trigger GitHub Actions to build and push your image automatically.

  • Stage and commit the workflow file by running:
git add .
git commit -m "Add CI/CD workflow"
  • Push your code to GitHub by running:
git push -u origin main

What happens when you push?

GitHub detects the deploy.yml workflow file and sees a push to main. It spins up an Ubuntu runner, checks out your code, authenticates with AWS, and builds then pushes your Docker image to ECR. The image gets tagged with the commit SHA so every version is traceable.

  • Open your repository on GitHub in your browser.
  • Click the Actions tab.
  • You should see your workflow run listed. Wait for it to complete and show a green checkmark.
  • Confirm the Docker image was pushed to ECR by running:
aws ecr list-images --repository-name crypto-pipeline --region us-east-1
  • You should see an image listed with an imageTag matching your latest commit SHA.

Workflow failed with a red X?

  • Click the failed workflow run, then click the job name to see the logs. Look for the step that failed.
  • If the Configure AWS credentials step fails, verify your GitHub Secrets are named exactly AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with no extra spaces.
  • If the Login to Amazon ECR step fails, confirm your IAM user has the AmazonEC2ContainerRegistryFullAccess policy attached.

Help me debug my failed GitHub Actions workflow

Your pipeline now has fully automated CI/CD. Every time you push code to main, a fresh Docker image lands in ECR without you lifting a finger. Next up, you will take this to the next level with automated daily scheduling.

Secret mission

Add Automated Daily Scheduling

Your pipeline works, but you have to run it manually every time. In production, data pipelines run on a schedule without human intervention. Can you wire up AWS EventBridge Scheduler and ECS Fargate to run your containerized pipeline automatically every 24 hours?

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project uses AWS Free Tier services, so costs are negligible for small Parquet files and minimal ECR storage. However, the EventBridge schedule will keep running your pipeline daily if left active.

Cost warning

If you completed the Secret Mission, the EventBridge schedule will trigger your ECS Fargate task once per day. Each run costs a small amount of Fargate compute time. While the Free Tier covers most usage, leaving the schedule running indefinitely could eventually generate charges.

If you are not planning to continue building on this project, choose Pause or Delete below to stop scheduled runs.

Resources you used:

  • S3 bucket (crypto-pipeline-data-{initials}).
  • ECR repository (crypto-pipeline).
  • ECS cluster (crypto-pipeline-cluster).
  • ECS task definition (crypto-pipeline-task).
  • EventBridge schedule (crypto-pipeline-daily).
  • CloudWatch log group (/ecs/crypto-pipeline).
  • IAM user with attached policies and access keys.
  • GitHub repository with Actions workflow.
  • Local project files on your Desktop.

Keep everything running

No action needed. Choose this if you are still actively building on this project or want the daily pipeline to keep running.

  • Your EventBridge schedule will continue triggering daily runs. Each run produces a new Parquet file in S3.
  • Monitor your AWS Free Tier usage in the Billing console to make sure you stay within limits.
  • AWS CLI remains installed on your machine for future projects.

Pause - I'll come back to this later

Stop the scheduled runs to avoid any ongoing compute costs while keeping all your resources intact.

  • Delete the EventBridge schedule by running:
aws scheduler delete-schedule --name crypto-pipeline-daily

This stops all future scheduled runs. Your S3 bucket, ECR repository, ECS cluster, and task definition remain in place. You can recreate the schedule later to resume daily runs.

Delete - I don't want to use this again

Remove all project resources and start fresh. Run each of the following commands in your terminal.

  • Delete the EventBridge schedule:
aws scheduler delete-schedule --name crypto-pipeline-daily
  • Delete the S3 bucket and all its contents:
aws s3 rb s3://[[S3_BUCKET="crypto-pipeline-data-your-initials"]] --force
  • Delete the ECR repository and all images:
aws ecr delete-repository --repository-name crypto-pipeline --force --region us-east-1
  • Deregister the ECS task definition (replace 1 with your revision number):
aws ecs deregister-task-definition --task-definition crypto-pipeline-task:1
  • Delete the ECS cluster:
aws ecs delete-cluster --cluster-name crypto-pipeline-cluster
  • Delete the CloudWatch log group:
aws logs delete-log-group --log-group-name /ecs/crypto-pipeline
  • Delete the IAM user. First, detach the managed policies:
aws iam detach-user-policy --user-name [[IAM_USERNAME="your-iam-username"]] --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess
aws iam detach-user-policy --user-name [[IAM_USERNAME="your-iam-username"]] --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryFullAccess
  • List and delete the access keys for the user:
aws iam list-access-keys --user-name [[IAM_USERNAME="your-iam-username"]]
aws iam delete-access-key --user-name [[IAM_USERNAME="your-iam-username"]] --access-key-id [[ACCESS_KEY_ID="your-access-key-id"]]
  • Delete the IAM user:
aws iam delete-user --user-name [[IAM_USERNAME="your-iam-username"]]
  • Delete the GitHub repository by going to your repository's Settings page, scrolling to the Danger Zone section, and clicking Delete this repository.
  • Delete your local project files:

macOS

cd ~/Desktop
rm -rf crypto-pipeline

Windows

cd ~\Desktop
rmdir /s /q crypto-pipeline

That's a Wrap!

That's a Wrap!

Nice work. You just built a production-grade data pipeline that pulls live cryptocurrency market data, validates its quality, and stores clean analytics-ready files in AWS S3.

You've learned how to:

  • Build a Python ETL pipeline that ingests live cryptocurrency market data from the CoinGecko API, validates data quality with null checks, schema validation, and staleness detection, and stores clean Parquet files in S3 with date partitioning.
  • Containerize the pipeline with Docker for consistent, reproducible execution across any environment.
  • Set up CI/CD with GitHub Actions to automatically build and push your Docker image to AWS ECR on every commit to main.
  • Secret Mission: Added automated daily scheduling with EventBridge and ECS Fargate for hands-off production operation.

Ready to quiz yourself?