Local Data Lakehouse with Medallion

Introduction

30 Second Summary

Empresas processam milhoes de registros diariamente. Antes de virar insights em dashboards, esses dados passam por camadas de refinamento que garantem qualidade e confiabilidade.

Neste projeto, voce vai construir um Data Lakehouse local completo com arquitetura medalha (Bronze/Silver/Gold), orquestrado por um pipeline automatizado que ingere, transforma e testa dados de e-commerce.

What You'll Build

Um pipeline end-to-end containerizado que processa transacoes de e-commerce atraves de tres camadas usando Apache Spark, dbt Core, DuckDB e Apache Airflow.

By the end of this project, you'll have:

• Um Docker Compose multi-servico com LocalStack (S3), cluster Spark, Airflow e Postgres rodando localmente.
• Um job PySpark que ingere CSVs brutos e grava Parquet particionado na camada Bronze, e um projeto dbt com modelos Silver e Gold com testes de data quality.
• Uma DAG Airflow orquestrando o pipeline completo (Spark submit, dbt run, dbt test) com documentacao e lineage gerados automaticamente.
• Secret Mission: adicionar data quality checks avancados com testes dbt customizados e alertas no Airflow para falhas de qualidade.

Quais sao os pre-requisitos?

Este projeto assume familiaridade basica com Docker, Python, SQL e terminal. Experiencia previa com Airflow, Spark e LocalStack e recomendada.

Voce precisa ter o Docker Desktop instalado e pelo menos 8GB de RAM disponivel para rodar todos os containers.

Set Up Project Structure and Docker Compose Base

A Data Lakehouse needs an infrastructure layer before any data flows through it. You need object storage for your medallion layers, a processing engine for ingestion, and a shared network so all services can communicate.

In this step, you'll scaffold the project directories and write a Docker Compose file that spins up LocalStack (S3-compatible storage) and an Apache Spark cluster (master + worker) for distributed processing.

In this step, get ready to:

• Create the project directory structure with folders for DAGs, Spark jobs, dbt models, and raw data.
• Write a Docker Compose file with LocalStack and a Spark cluster.
• Start the containers and validate that all services are healthy.

Create the project directory structure

Your lakehouse project needs dedicated folders for each concern. The dags/ folder holds Airflow orchestration files. The spark/ folder holds PySpark jobs. The dbt/ folder holds transformation models. The data/raw/ folder stores source CSV files. The docker/airflow/ folder holds a custom Dockerfile for Airflow.

• Create all project directories by running this command inside your project folder:

mkdir -p dags spark dbt data/raw docker/airflow

• Confirm the folders exist by running:

ls

You should see dags, data, dbt, docker, and spark listed in the output.

Write the Docker Compose configuration

The Docker Compose file defines all services your lakehouse needs. You'll start by defining a shared Airflow configuration template (used in Step 2), then declare the services that run in this step.

• Create a file called docker-compose.yml in your project root and add the following code:

version: '3.8'

x-airflow-common: &airflow-common
  build:
    context: ./docker/airflow
    dockerfile: Dockerfile
  environment: &airflow-common-env
    AIRFLOW__CORE__EXECUTOR: LocalExecutor
    AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres:5432/airflow
    AIRFLOW__CORE__FERNET_KEY: ''
    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
    AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
    AIRFLOW__WEBSERVER__EXPOSE_CONFIG: 'true'
    AWS_ACCESS_KEY_ID: test
    AWS_SECRET_ACCESS_KEY: test
    AWS_DEFAULT_REGION: us-east-1
    SPARK_HOME: /opt/spark
  volumes:
    - ./dags:/opt/airflow/dags
    - ./spark:/opt/airflow/spark
    - ./dbt/lakehouse:/opt/airflow/dbt
    - ./data:/opt/airflow/data
    - airflow-logs:/opt/airflow/logs
  depends_on:
    postgres:
      condition: service_healthy
    localstack:
      condition: service_healthy
  networks:
    - lakehouse

What does this code do?

The x-airflow-common block is a YAML anchor. It defines a reusable template that Airflow services will inherit from in Step 2.

• The environment section configures Airflow to use LocalExecutor with a Postgres backend, disables example DAGs, and sets AWS credentials for LocalStack access.
• The volumes section mounts your local project folders into the container so Airflow can access DAGs, Spark jobs, dbt models, and raw data.
• The depends_on section ensures Postgres and LocalStack are healthy before any Airflow service starts.

You won't see a change yet. The next chunk completes the file with the actual running services.

• Add the services, networks, and volumes sections below the x-airflow-common block in the same file:

services:
  localstack:
    container_name: localstack
    image: localstack/localstack:3.8
    ports:
      - "4566:4566"
    environment:
      - SERVICES=s3
      - DEBUG=0
      - PERSISTENCE=1
    volumes:
      - localstack-data:/var/lib/localstack
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:4566/_localstack/health"]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - lakehouse

  spark-master:
    image: bitnami/spark:3.5
    container_name: spark-master
    environment:
      - SPARK_MODE=master
      - SPARK_MASTER_HOST=spark-master
    ports:
      - "8081:8080"
      - "7077:7077"
    networks:
      - lakehouse

  spark-worker:
    image: bitnami/spark:3.5
    container_name: spark-worker
    environment:
      - SPARK_MODE=worker
      - SPARK_MASTER_URL=spark://spark-master:7077
      - SPARK_WORKER_MEMORY=1g
      - SPARK_WORKER_CORES=2
    depends_on:
      - spark-master
    networks:
      - lakehouse

networks:
  lakehouse:
    driver: bridge

volumes:
  localstack-data:
  postgres-data:
  airflow-logs:

What does this code do?

This section defines the three services that run in this step.

• The localstack service emulates AWS S3 locally on port 4566. The healthcheck ensures other services wait until it is ready.
• The spark-master service runs the Spark master node. Port 8081 maps to its web UI, and port 7077 is the cluster communication port.
• The spark-worker service connects to the master and provides 1 GB of memory and 2 CPU cores for processing jobs.
• The lakehouse network lets all containers communicate by hostname. The named volumes persist data across restarts.

• Save docker-compose.yml.

YAML syntax error when saving?

YAML is sensitive to indentation. Make sure you use spaces (not tabs) and that each nested level is indented by exactly 2 spaces. Check that the services: key is at the root level (no leading spaces).

Help me fix my YAML syntax error

✔️ Awesome, I've got everything!

Great. Double check you've saved the file before moving on.

ⓧ I'd like to double check the full code

version: '3.8'

x-airflow-common: &airflow-common
  build:
    context: ./docker/airflow
    dockerfile: Dockerfile
  environment: &airflow-common-env
    AIRFLOW__CORE__EXECUTOR: LocalExecutor
    AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres:5432/airflow
    AIRFLOW__CORE__FERNET_KEY: ''
    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
    AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
    AIRFLOW__WEBSERVER__EXPOSE_CONFIG: 'true'
    AWS_ACCESS_KEY_ID: test
    AWS_SECRET_ACCESS_KEY: test
    AWS_DEFAULT_REGION: us-east-1
    SPARK_HOME: /opt/spark
  volumes:
    - ./dags:/opt/airflow/dags
    - ./spark:/opt/airflow/spark
    - ./dbt/lakehouse:/opt/airflow/dbt
    - ./data:/opt/airflow/data
    - airflow-logs:/opt/airflow/logs
  depends_on:
    postgres:
      condition: service_healthy
    localstack:
      condition: service_healthy
  networks:
    - lakehouse

services:
  localstack:
    container_name: localstack
    image: localstack/localstack:3.8
    ports:
      - "4566:4566"
    environment:
      - SERVICES=s3
      - DEBUG=0
      - PERSISTENCE=1
    volumes:
      - localstack-data:/var/lib/localstack
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:4566/_localstack/health"]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - lakehouse

  spark-master:
    image: bitnami/spark:3.5
    container_name: spark-master
    environment:
      - SPARK_MODE=master
      - SPARK_MASTER_HOST=spark-master
    ports:
      - "8081:8080"
      - "7077:7077"
    networks:
      - lakehouse

  spark-worker:
    image: bitnami/spark:3.5
    container_name: spark-worker
    environment:
      - SPARK_MODE=worker
      - SPARK_MASTER_URL=spark://spark-master:7077
      - SPARK_WORKER_MEMORY=1g
      - SPARK_WORKER_CORES=2
    depends_on:
      - spark-master
    networks:
      - lakehouse

networks:
  lakehouse:
    driver: bridge

volumes:
  localstack-data:
  postgres-data:
  airflow-logs:

Start containers and validate services

With the Compose file ready, you can now bring up the three services and confirm they are running correctly.

• Start the containers in detached mode by running:

docker compose up -d localstack spark-master spark-worker

Why specify service names?

The x-airflow-common anchor references a postgres service that does not exist yet. Naming the three services explicitly avoids errors from unresolved dependencies. You will add Postgres and Airflow in the next step.

Docker will pull the images on the first run. This may take a few minutes depending on your connection.

• Once the containers are running, validate that LocalStack is healthy by running:

curl http://localhost:4566/_localstack/health

You should see a JSON response with "s3": "available" in the output.

• Validate the Spark cluster by opening your browser and navigating to http://localhost:8081.

You should see the Spark Master UI with one worker registered, showing 1 GB of memory and 2 cores.

Containers not starting?

• Make sure Docker Desktop is running. You can check by running docker info in your terminal.
• If port 4566 or 8081 is already in use, stop the conflicting process or change the port mapping in docker-compose.yml.
• If you see image pull errors, check your internet connection and try running docker compose pull separately.

Help me debug my Docker containers

Your infrastructure layer is live. LocalStack is emulating S3 and Spark is ready to process data. Next up, you will add Airflow to orchestrate the entire pipeline.

Configure Airflow with Docker Compose

Your Spark cluster and LocalStack S3 are running. Now you need an orchestrator to tie the entire pipeline together.

Apache Airflow schedules and monitors workflows as DAGs (Directed Acyclic Graphs). In this step, you will add Airflow to your Docker Compose stack with a custom image that includes PySpark, dbt, and boto3.

In this step, get ready to:

• Add Postgres and Airflow services (webserver, scheduler, init) to your Docker Compose file.
• Create a custom Airflow Dockerfile with Java, Spark, and Python dependencies.
• Start the full stack and log into the Airflow UI.

Add Postgres and Airflow services to docker-compose.yml

Airflow needs a metadata database to track DAG runs, task states, and connections. Postgres fills that role.

• In docker-compose.yml, find the spark-worker service block. Add the following postgres service below it:

  postgres:
    image: postgres:16
    container_name: postgres
    environment:
      POSTGRES_USER: airflow
      POSTGRES_PASSWORD: airflow
      POSTGRES_DB: airflow
    volumes:
      - postgres-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "airflow"]
      interval: 5s
      retries: 5
    networks:
      - lakehouse

What does this service do?

• postgres:16 runs a PostgreSQL 16 database that Airflow uses to store metadata (DAG run history, task states, connections).
• The healthcheck uses pg_isready so that Airflow services wait until Postgres is accepting connections before starting.
• The postgres-data volume persists database data across container restarts.

Now add the three Airflow services. These handle initialization, the web UI, and task scheduling.

• Below the postgres service you just added, paste these three Airflow service definitions:

  airflow-init:
    <<: *airflow-common
    container_name: airflow-init
    entrypoint: /bin/bash
    command:
      - -c
      - |
        airflow db migrate
        airflow users create \
          --username airflow \
          --password airflow \
          --firstname Admin \
          --lastname User \
          --role Admin \
          --email admin@example.com
    environment:
      <<: *airflow-common-env
    restart: "no"

  airflow-webserver:
    <<: *airflow-common
    container_name: airflow-webserver
    command: webserver
    ports:
      - "8080:8080"
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:8080/health"]
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always

  airflow-scheduler:
    <<: *airflow-common
    container_name: airflow-scheduler
    command: scheduler
    restart: always

What do these services do?

• airflow-init runs once to apply database migrations and create an admin user (airflow/airflow). It exits after setup completes.
• airflow-webserver serves the Airflow UI on port 8080.
• airflow-scheduler monitors DAGs and triggers task runs on schedule.
• All three inherit shared config from the x-airflow-common YAML anchor using <<: *airflow-common. This keeps environment variables, volumes, and dependencies DRY.

• Save docker-compose.yml.

You won't see a visible change yet. The next substep creates the Dockerfile that these services depend on.

YAML indentation errors?

Each service key (postgres, airflow-init, etc.) must be indented at the same level as localstack and spark-master (two spaces under services:).

help me fix YAML indentation in my docker-compose.yml for Airflow services

Create the custom Airflow Dockerfile

The standard Airflow image does not include Java or Spark. Your pipeline needs both to submit PySpark jobs from within Airflow tasks.

• Open the file docker/airflow/Dockerfile (inside the docker/airflow/ folder you created in Step 1) and paste the following:

FROM apache/airflow:2.11.2-python3.11

USER root

# Install Java 17 and curl for Spark
RUN apt-get update && \
    apt-get install -y --no-install-recommends openjdk-17-jdk curl && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# Configure Java and Spark paths
ENV JAVA_HOME=/usr/lib/jvm/java-17-openjdk-arm64
ENV SPARK_HOME=/opt/spark
ENV PATH="${SPARK_HOME}/bin:${PATH}"

# Download and install Spark 3.5.6
RUN mkdir -p ${SPARK_HOME} && \
    curl -fsSL https://dlcdn.apache.org/spark/spark-3.5.6/spark-3.5.6-bin-hadoop3.tgz | \
    tar -xz -C ${SPARK_HOME} --strip-components=1 && \
    chown -R airflow:root ${SPARK_HOME}

USER airflow

# Install Python dependencies
COPY requirements.txt /tmp/requirements.txt
RUN pip install --no-cache-dir -r /tmp/requirements.txt

What does this Dockerfile do?

• Starts from the official apache/airflow:2.11.2-python3.11 image with Python 3.11.
• Installs OpenJDK 17 (required by Spark) and downloads Spark 3.5.6 into /opt/spark.
• Copies a requirements.txt and installs the Python packages needed for dbt, PySpark, and AWS access.

Intel Mac or Linux x86?

The Dockerfile sets JAVA_HOME=/usr/lib/jvm/java-17-openjdk-arm64 for Apple Silicon Macs. If you are on an Intel Mac or Linux x86_64, change arm64 to amd64 in that line.

• Save docker/airflow/Dockerfile.

Now create the requirements file that the Dockerfile copies in.

• Open docker/airflow/requirements.txt and paste the following dependencies:

dbt-core==1.9.4
dbt-duckdb==1.10.1
boto3==1.35.0
pyspark==3.5.6
apache-airflow-providers-apache-spark==4.12.0

Why pin these versions?

Pinning exact versions ensures reproducible builds. Each package serves a specific role:

• dbt-core + dbt-duckdb for SQL transformations on the Silver and Gold layers.
• boto3 for AWS SDK access to LocalStack S3.
• pyspark matches Spark 3.5.6 running on the cluster.
• apache-airflow-providers-apache-spark adds the SparkSubmitOperator used in the DAG.

• Save docker/airflow/requirements.txt.
• Confirm both files exist by running:

ls docker/airflow/

You should see Dockerfile and requirements.txt listed.

Files not showing up?

Make sure you created the files inside docker/airflow/ (not in the project root). The Dockerfile name must be exactly Dockerfile with a capital D and no file extension.

help me find where my Dockerfile should be in the project structure

✔️ Awesome, I've got everything!

Great. Make sure you have saved all three files: docker-compose.yml, docker/airflow/Dockerfile, and docker/airflow/requirements.txt.

ⓧ I'd like to double check the full code

Here is the full docker-compose.yml at this point:

version: '3.8'

x-airflow-common: &airflow-common
  build:
    context: ./docker/airflow
    dockerfile: Dockerfile
  environment: &airflow-common-env
    AIRFLOW__CORE__EXECUTOR: LocalExecutor
    AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres:5432/airflow
    AIRFLOW__CORE__FERNET_KEY: ''
    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
    AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
    AIRFLOW__WEBSERVER__EXPOSE_CONFIG: 'true'
    AWS_ACCESS_KEY_ID: test
    AWS_SECRET_ACCESS_KEY: test
    AWS_DEFAULT_REGION: us-east-1
    SPARK_HOME: /opt/spark
  volumes:
    - ./dags:/opt/airflow/dags
    - ./spark:/opt/airflow/spark
    - ./dbt/lakehouse:/opt/airflow/dbt
    - ./data:/opt/airflow/data
    - airflow-logs:/opt/airflow/logs
  depends_on:
    postgres:
      condition: service_healthy
    localstack:
      condition: service_healthy
  networks:
    - lakehouse

services:
  localstack:
    container_name: localstack
    image: localstack/localstack:3.8
    ports:
      - "4566:4566"
    environment:
      - SERVICES=s3
      - DEBUG=0
      - PERSISTENCE=1
    volumes:
      - localstack-data:/var/lib/localstack
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:4566/_localstack/health"]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - lakehouse

  spark-master:
    image: bitnami/spark:3.5
    container_name: spark-master
    environment:
      - SPARK_MODE=master
      - SPARK_MASTER_HOST=spark-master
    ports:
      - "8081:8080"
      - "7077:7077"
    networks:
      - lakehouse

  spark-worker:
    image: bitnami/spark:3.5
    container_name: spark-worker
    environment:
      - SPARK_MODE=worker
      - SPARK_MASTER_URL=spark://spark-master:7077
      - SPARK_WORKER_MEMORY=1g
      - SPARK_WORKER_CORES=2
    depends_on:
      - spark-master
    networks:
      - lakehouse

  postgres:
    image: postgres:16
    container_name: postgres
    environment:
      POSTGRES_USER: airflow
      POSTGRES_PASSWORD: airflow
      POSTGRES_DB: airflow
    volumes:
      - postgres-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "airflow"]
      interval: 5s
      retries: 5
    networks:
      - lakehouse

  airflow-init:
    <<: *airflow-common
    container_name: airflow-init
    entrypoint: /bin/bash
    command:
      - -c
      - |
        airflow db migrate
        airflow users create \
          --username airflow \
          --password airflow \
          --firstname Admin \
          --lastname User \
          --role Admin \
          --email admin@example.com
    environment:
      <<: *airflow-common-env
    restart: "no"

  airflow-webserver:
    <<: *airflow-common
    container_name: airflow-webserver
    command: webserver
    ports:
      - "8080:8080"
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:8080/health"]
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always

  airflow-scheduler:
    <<: *airflow-common
    container_name: airflow-scheduler
    command: scheduler
    restart: always

networks:
  lakehouse:
    driver: bridge

volumes:
  localstack-data:
  postgres-data:
  airflow-logs:

Here is the full docker/airflow/Dockerfile:

FROM apache/airflow:2.11.2-python3.11

USER root

RUN apt-get update && \
    apt-get install -y --no-install-recommends openjdk-17-jdk curl && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

ENV JAVA_HOME=/usr/lib/jvm/java-17-openjdk-arm64
ENV SPARK_HOME=/opt/spark
ENV PATH="${SPARK_HOME}/bin:${PATH}"

RUN mkdir -p ${SPARK_HOME} && \
    curl -fsSL https://dlcdn.apache.org/spark/spark-3.5.6/spark-3.5.6-bin-hadoop3.tgz | \
    tar -xz -C ${SPARK_HOME} --strip-components=1 && \
    chown -R airflow:root ${SPARK_HOME}

USER airflow

COPY requirements.txt /tmp/requirements.txt
RUN pip install --no-cache-dir -r /tmp/requirements.txt

Here is the full docker/airflow/requirements.txt:

dbt-core==1.9.4
dbt-duckdb==1.10.1
boto3==1.35.0
pyspark==3.5.6
apache-airflow-providers-apache-spark==4.12.0

Start the full stack and access Airflow UI

With all files in place, you can build the custom Airflow image and bring up the entire stack.

• Stop any running containers from Step 1 and rebuild the full stack by running:

docker compose up -d --build

What does this command do?

The --build flag tells Docker Compose to build images from Dockerfiles before starting containers. The -d flag runs everything in detached mode (background).

The first build will take a few minutes because it downloads the Airflow base image, installs Java, and downloads Spark. Subsequent runs will use the cached layers and start in seconds.

• Wait for the build to finish, then verify all containers are running:

docker compose ps

You should see localstack, spark-master, spark-worker, postgres, airflow-webserver, and airflow-scheduler all with status running. The airflow-init container shows Exited (0) because it completes its setup and stops.

• Open your browser and navigate to http://localhost:8080.
• Log in with username airflow and password airflow.

You should see the Airflow dashboard with an empty DAGs list. The DAGs folder is empty for now.

Webserver not responding on port 8080?

The webserver takes 30-60 seconds to become healthy after the container starts. Check its health status:

docker compose logs airflow-webserver --tail 20

• If you see database is locked or migration errors, the airflow-init container may not have finished. Run docker compose logs airflow-init and wait until it exits with code 0.
• If the build failed, check that docker/airflow/Dockerfile and docker/airflow/requirements.txt both exist in the correct path.

help me debug why Airflow webserver is not starting in Docker Compose

Your orchestration layer is live. Next up, you will write the PySpark job that ingests raw CSV data into the Bronze layer on S3.

Build the Spark Bronze Ingestion Job

Your infrastructure is running and Airflow is ready to orchestrate. Now you need data to process and a job to process it.

In a Data Lakehouse, the Bronze layer stores raw data exactly as it arrives from the source. Your PySpark job will read CSV files, add ingestion metadata (timestamps, batch IDs, source file names), and write everything as Parquet to the S3 bucket in LocalStack. This mirrors exactly how production data pipelines ingest raw data before any transformations happen.

In this step, get ready to:

• Create sample e-commerce CSV data (transactions and customers).
• Write a PySpark job that ingests CSVs into the Bronze layer as partitioned Parquet.
• Execute the job via spark-submit and validate the Parquet files in S3.

Create sample e-commerce data

Before building the ingestion job, you need source data. You will create two CSV files that simulate an e-commerce platform: one with transaction orders and another with customer information.

• Create a file called transactions.csv inside the data/raw/ folder with the following content:

order_id,customer_id,product_id,product_name,category,quantity,unit_price,order_date,status,payment_method
ORD001,C101,P001,Wireless Mouse,Electronics,2,29.99,2024-01-15,completed,credit_card
ORD002,C102,P002,USB-C Cable,Electronics,5,12.50,2024-01-15,completed,pix
ORD003,C103,P003,Office Chair,Furniture,1,450.00,2024-01-16,completed,credit_card
ORD004,C101,P004,Mechanical Keyboard,Electronics,1,189.90,2024-01-16,pending,boleto
ORD005,C104,P005,Desk Lamp,Furniture,2,75.00,2024-01-17,completed,pix
ORD006,C105,P001,Wireless Mouse,Electronics,1,29.99,2024-01-17,cancelled,credit_card
ORD007,C102,P006,Monitor Stand,Furniture,1,120.00,2024-01-18,completed,credit_card
ORD008,C106,P007,Webcam HD,Electronics,1,199.00,2024-01-18,completed,pix
ORD009,C103,P008,Notebook Stand,Furniture,1,89.90,2024-01-19,completed,credit_card
ORD010,C107,P002,USB-C Cable,Electronics,3,12.50,2024-01-19,completed,boleto
ORD011,C108,P009,Bluetooth Speaker,Electronics,1,149.90,2024-01-20,completed,pix
ORD012,C101,P003,Office Chair,Furniture,1,450.00,2024-01-20,pending,credit_card
ORD013,C109,P010,Mouse Pad XL,Accessories,2,45.00,2024-01-21,completed,pix
ORD014,C110,P004,Mechanical Keyboard,Electronics,1,189.90,2024-01-21,completed,credit_card
ORD015,C104,P011,USB Hub 7-port,Electronics,1,79.90,2024-01-22,completed,boleto

What does this data represent?

This file contains 15 e-commerce orders spanning 8 days. It includes different statuses (completed, pending, cancelled), multiple payment methods, and three product categories.

The order_date column is important because the Spark job will partition the Bronze layer by this column, mirroring how production data lakes organize data for efficient querying.

• Create a second file called customers.csv inside the same data/raw/ folder:

customer_id,name,email,city,state,created_at
C101,Maria Silva,maria@email.com,Sao Paulo,SP,2023-06-15
C102,Joao Santos,joao@email.com,Rio de Janeiro,RJ,2023-07-20
C103,Ana Oliveira,ana@email.com,Belo Horizonte,MG,2023-08-10
C104,Pedro Costa,pedro@email.com,Curitiba,PR,2023-09-05
C105,Lucas Ferreira,lucas@email.com,Porto Alegre,RS,2023-10-12
C106,Carla Souza,carla@email.com,Salvador,BA,2023-11-08
C107,Rafael Lima,rafael@email.com,Fortaleza,CE,2023-12-01
C108,Juliana Alves,juliana@email.com,Brasilia,DF,2024-01-03
C109,Bruno Martins,bruno@email.com,Recife,PE,2024-01-10
C110,Fernanda Rocha,fernanda@email.com,Manaus,AM,2024-01-15

What does this data do?

The customers file holds 10 customer records across different Brazilian cities. The Silver layer (in a later step) will join this data with transactions to build customer-level analytics in the Gold layer.

• Save both files.
• Confirm the files exist by running:

ls data/raw/

You should see customers.csv and transactions.csv in the output.

Don't see the files?

Make sure you are running the ls command from your project root directory (the folder that contains docker-compose.yml). If you see No such file or directory, navigate back to your project folder first.

help me find my project files

Write the PySpark Bronze ingestion job

The ingestion job connects to the Spark cluster, reads your CSV files, enriches each row with metadata columns, and writes everything as Parquet to the S3 bucket in LocalStack. You will build this function by function.

• Create a file called bronze_ingestion.py inside the spark/ folder. Start with the imports and the Spark session function:

"""\nBronze Ingestion Job - PySpark\nReads raw CSV files and writes Parquet to S3 (LocalStack) with ingestion metadata.\n"""
import sys
from datetime import datetime

from pyspark.sql import SparkSession
from pyspark.sql.functions import lit, current_timestamp, input_file_name


def create_spark_session():
    """Create SparkSession configured for LocalStack S3."""
    return (
        SparkSession.builder
        .appName("BronzeIngestion")
        .master("spark://spark-master:7077")
        .config("spark.hadoop.fs.s3a.endpoint", "http://localstack:4566")
        .config("spark.hadoop.fs.s3a.access.key", "test")
        .config("spark.hadoop.fs.s3a.secret.key", "test")
        .config("spark.hadoop.fs.s3a.path.style.access", "true")
        .config("spark.hadoop.fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
        .config("spark.hadoop.fs.s3a.connection.ssl.enabled", "false")
        .config(
            "spark.jars.packages",
            "org.apache.hadoop:hadoop-aws:3.3.4,com.amazonaws:aws-java-sdk-bundle:1.12.262"
        )
        .getOrCreate()
    )

What does this code do?

The create_spark_session() function builds a SparkSession that connects to the Spark master at spark://spark-master:7077 (the Docker container name on your lakehouse network).

The S3A configurations tell Spark how to talk to LocalStack instead of real AWS. Path-style access (s3a://bucket/key) is required because LocalStack does not support virtual-hosted-style URLs. The hadoop-aws and aws-java-sdk-bundle packages provide the S3A filesystem implementation that Spark needs to read and write to S3.

• You won't see a change yet. Next, add the ingest_to_bronze function below create_spark_session():

def ingest_to_bronze(spark, source_path, table_name, partition_col=None):
    """Read CSV and write to Bronze layer as Parquet."""
    df = (
        spark.read
        .option("header", "true")
        .option("inferSchema", "true")
        .csv(source_path)
    )

    df_with_metadata = (
        df
        .withColumn("_ingested_at", current_timestamp())
        .withColumn("_source_file", input_file_name())
        .withColumn("_batch_id", lit(datetime.now().strftime("%Y%m%d%H%M%S")))
    )

    output_path = f"s3a://lakehouse/bronze/{table_name}"

    writer = df_with_metadata.write.mode("overwrite").format("parquet")

    if partition_col and partition_col in df_with_metadata.columns:
        writer = writer.partitionBy(partition_col)

    writer.save(output_path)

    print(f"[BRONZE] {table_name}: {df_with_metadata.count()} rows written to {output_path}")

What does this function do?

This is the core ingestion logic. It reads a CSV with schema inference, then adds three metadata columns that make the Bronze layer auditable:

• _ingested_at records exactly when this row was processed.
• _source_file captures which file the row came from (useful when ingesting from multiple sources).
• _batch_id groups all rows from the same job execution together.

The function writes Parquet to s3a://lakehouse/bronze/{table_name} and optionally partitions by a date column for efficient downstream queries.

• You won't see a change yet. Finally, add the main() function and entry point below ingest_to_bronze:

def main():
    spark = create_spark_session()

    try:
        ingest_to_bronze(
            spark,
            source_path="/opt/airflow/data/raw/transactions.csv",
            table_name="transactions",
            partition_col="order_date"
        )

        ingest_to_bronze(
            spark,
            source_path="/opt/airflow/data/raw/customers.csv",
            table_name="customers"
        )

        print("[BRONZE] Ingestion completed successfully!")
    finally:
        spark.stop()


if __name__ == "__main__":
    main()

What does this code do?

The main() function orchestrates the ingestion. It ingests transactions partitioned by order_date (so each date gets its own directory), and customers without partitioning (small dataset, no benefit from splitting).

The try/finally block ensures the Spark session is always stopped cleanly, even if an error occurs mid-ingestion. The paths use /opt/airflow/data/raw/ because that is where the Docker volume mounts your local data/ folder inside the container.

• Save spark/bronze_ingestion.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved spark/bronze_ingestion.py before moving on to the execution step.

ⓧ I'd like to double check the full code

"""\nBronze Ingestion Job - PySpark\nReads raw CSV files and writes Parquet to S3 (LocalStack) with ingestion metadata.\n"""
import sys
from datetime import datetime

from pyspark.sql import SparkSession
from pyspark.sql.functions import lit, current_timestamp, input_file_name


def create_spark_session():
    """Create SparkSession configured for LocalStack S3."""
    return (
        SparkSession.builder
        .appName("BronzeIngestion")
        .master("spark://spark-master:7077")
        .config("spark.hadoop.fs.s3a.endpoint", "http://localstack:4566")
        .config("spark.hadoop.fs.s3a.access.key", "test")
        .config("spark.hadoop.fs.s3a.secret.key", "test")
        .config("spark.hadoop.fs.s3a.path.style.access", "true")
        .config("spark.hadoop.fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
        .config("spark.hadoop.fs.s3a.connection.ssl.enabled", "false")
        .config(
            "spark.jars.packages",
            "org.apache.hadoop:hadoop-aws:3.3.4,com.amazonaws:aws-java-sdk-bundle:1.12.262"
        )
        .getOrCreate()
    )


def ingest_to_bronze(spark, source_path, table_name, partition_col=None):
    """Read CSV and write to Bronze layer as Parquet."""
    df = (
        spark.read
        .option("header", "true")
        .option("inferSchema", "true")
        .csv(source_path)
    )

    df_with_metadata = (
        df
        .withColumn("_ingested_at", current_timestamp())
        .withColumn("_source_file", input_file_name())
        .withColumn("_batch_id", lit(datetime.now().strftime("%Y%m%d%H%M%S")))
    )

    output_path = f"s3a://lakehouse/bronze/{table_name}"

    writer = df_with_metadata.write.mode("overwrite").format("parquet")

    if partition_col and partition_col in df_with_metadata.columns:
        writer = writer.partitionBy(partition_col)

    writer.save(output_path)

    print(f"[BRONZE] {table_name}: {df_with_metadata.count()} rows written to {output_path}")


def main():
    spark = create_spark_session()

    try:
        ingest_to_bronze(
            spark,
            source_path="/opt/airflow/data/raw/transactions.csv",
            table_name="transactions",
            partition_col="order_date"
        )

        ingest_to_bronze(
            spark,
            source_path="/opt/airflow/data/raw/customers.csv",
            table_name="customers"
        )

        print("[BRONZE] Ingestion completed successfully!")
    finally:
        spark.stop()


if __name__ == "__main__":
    main()

Execute the job and validate the Bronze layer

With the code written and the data in place, you can now run the job against your Spark cluster. First, you need to create the S3 bucket in LocalStack, then submit the Spark job.

• Create the lakehouse S3 bucket in LocalStack by running:

docker exec localstack awslocal s3 mb s3://lakehouse

What does this command do?

The awslocal command is a thin wrapper around the AWS CLI that automatically points to the LocalStack endpoint. Running s3 mb (make bucket) creates the lakehouse bucket where your Bronze Parquet files will be stored.

You should see make_bucket: lakehouse in the output.

Seeing BucketAlreadyOwnedByYou?

This means the bucket already exists from a previous run. This is fine. The Spark job will still write to it normally.

help me troubleshoot LocalStack S3 bucket creation

• Submit the Bronze ingestion job to your Spark cluster by running:

docker exec airflow-webserver spark-submit \
  --master spark://spark-master:7077 \
  --packages org.apache.hadoop:hadoop-aws:3.3.4,com.amazonaws:aws-java-sdk-bundle:1.12.262 \
  /opt/airflow/spark/bronze_ingestion.py

What does this command do?

You execute spark-submit from inside the airflow-webserver container because it has both Spark installed (from your custom Dockerfile) and the spark/ volume mounted at /opt/airflow/spark/.

The --packages flag downloads the Hadoop AWS and AWS SDK JARs that Spark needs to communicate with S3. The --master flag tells Spark to distribute the work across your cluster (master + worker).

This command takes 1-2 minutes. You will see Spark logs scroll by, followed by output confirming the row counts.

• At the end of the output, look for these two lines confirming success:

[BRONZE] transactions: 15 rows written to s3a://lakehouse/bronze/transactions
[BRONZE] Ingestion completed successfully!

• Verify the Parquet files landed in the Bronze layer by listing the S3 bucket:

docker exec localstack awslocal s3 ls s3://lakehouse/bronze/ --recursive

You should see Parquet files listed under bronze/transactions/order_date=2024-01-15/ (and other dates) plus bronze/customers/.

Spark job failing with connection errors?

• If you see Connection refused to spark-master:7077, confirm the Spark cluster is running with docker ps and check that spark-master and spark-worker are both listed.
• If you see S3-related errors mentioning localstack:4566, make sure the localstack container is healthy by running docker ps and checking its status.
• If packages fail to download, the Spark worker needs internet access to pull the Maven JARs on first run. Retry after confirming your Docker network has internet connectivity.

help me debug my Spark Bronze ingestion job

Your Bronze layer is loaded with raw e-commerce data as partitioned Parquet. Next up, you will build the dbt project that transforms this raw data into clean Silver models and business-ready Gold aggregations.

Build the dbt Silver and Gold Models

Your Bronze layer is loaded with raw Parquet data in S3. The next challenge is turning that raw data into clean, trustworthy tables that answer real business questions.

This is where dbt comes in. You will use dbt Core with the dbt-duckdb adapter to build two transformation layers: Silver (cleaned, typed, deduplicated data) and Gold (aggregated business metrics). DuckDB reads the Parquet files directly from S3 via its httpfs extension, so no data copying is needed.

In this step, get ready to:

• Configure the dbt project with DuckDB connected to LocalStack S3.
• Create Silver models that clean and type raw transactions and customers.
• Create Gold models that aggregate revenue, product rankings, and customer lifetime metrics.

Configure the dbt project

Before dbt can transform anything, it needs a project configuration and a connection profile. The project tells dbt where to find models. The profile tells it how to connect to DuckDB and reach the S3 Parquet files.

• Create the directory structure for your dbt project by running this command:

mkdir -p dbt/lakehouse/models/silver dbt/lakehouse/models/gold

• Confirm the directories exist by running:

ls dbt/lakehouse/models/

You should see gold and silver listed in the output.

• Create dbt_project.yml inside your dbt/lakehouse/ folder with this content:

name: 'lakehouse'
version: '1.0.0'

profile: 'lakehouse'

model-paths: ["models"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
docs-paths: ["docs"]

clean-targets:
  - "target"
  - "dbt_packages"

models:
  lakehouse:
    silver:
      +materialized: table
    gold:
      +materialized: table

What does this configuration do?

The profile: 'lakehouse' line links this project to the connection profile you will create next.

The models section sets all Silver and Gold models to materialize as full tables in DuckDB. This means dbt creates actual tables (not views) so downstream queries run faster.

• Create profiles.yml inside your dbt/lakehouse/ folder with this content:

lakehouse:
  target: dev
  outputs:
    dev:
      type: duckdb
      path: /opt/airflow/dbt/lakehouse.duckdb
      extensions:
        - httpfs
      settings:
        s3_endpoint: localstack:4566
        s3_access_key_id: test
        s3_secret_access_key: test
        s3_use_ssl: false
        s3_url_style: path
      threads: 4

Why these S3 settings?

DuckDB's httpfs extension lets it read files from any S3-compatible endpoint. The s3_endpoint points to LocalStack's internal Docker hostname. s3_url_style: path tells DuckDB to use path-style URLs instead of virtual-hosted style.

The path field uses /opt/airflow/dbt/lakehouse.duckdb because dbt runs inside the Airflow container where dbt/lakehouse/ is mounted at /opt/airflow/dbt.

• Create sources.yml inside your dbt/lakehouse/models/ folder with this content:

version: 2

sources:
  - name: bronze
    description: "Raw data ingested by Spark in Parquet format on S3"
    meta:
      external_location: "s3://lakehouse/bronze/{name}/**/*.parquet"
    tables:
      - name: transactions
        description: "Raw e-commerce transactions"
      - name: customers
        description: "Raw customer data"

What is this sources file?

This file documents the Bronze layer tables that Spark wrote. The external_location metadata records the S3 path pattern. Your SQL models will reference these Parquet paths directly using DuckDB's read_parquet() function.

• Save all three files (dbt_project.yml, profiles.yml, sources.yml).

✔️ Awesome, I've got everything!

Great. Double check you have saved all three files in the correct locations.

ⓧ I'd like to double check the full code

Here is dbt/lakehouse/dbt_project.yml:

name: 'lakehouse'
version: '1.0.0'

profile: 'lakehouse'

model-paths: ["models"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
docs-paths: ["docs"]

clean-targets:
  - "target"
  - "dbt_packages"

models:
  lakehouse:
    silver:
      +materialized: table
    gold:
      +materialized: table

Here is dbt/lakehouse/profiles.yml:

lakehouse:
  target: dev
  outputs:
    dev:
      type: duckdb
      path: /opt/airflow/dbt/lakehouse.duckdb
      extensions:
        - httpfs
      settings:
        s3_endpoint: localstack:4566
        s3_access_key_id: test
        s3_secret_access_key: test
        s3_use_ssl: false
        s3_url_style: path
      threads: 4

Here is dbt/lakehouse/models/sources.yml:

version: 2

sources:
  - name: bronze
    description: "Raw data ingested by Spark in Parquet format on S3"
    meta:
      external_location: "s3://lakehouse/bronze/{name}/**/*.parquet"
    tables:
      - name: transactions
        description: "Raw e-commerce transactions"
      - name: customers
        description: "Raw customer data"

Create the Silver and Gold models

The Silver layer takes raw Bronze data and applies cleaning, type casting, and null filtering. The Gold layer aggregates Silver data into business-ready metrics.

• Create stg_transactions.sql inside dbt/lakehouse/models/silver/ with this content:

{{ config(
    materialized='table',
    description='Cleaned and typed transactions from Bronze layer'
) }}

with source as (
    select * from read_parquet('s3://lakehouse/bronze/transactions/**/*.parquet')
),

cleaned as (
    select
        order_id,
        customer_id,
        product_id,
        trim(product_name) as product_name,
        trim(category) as category,
        cast(quantity as integer) as quantity,
        cast(unit_price as decimal(10,2)) as unit_price,
        cast(order_date as date) as order_date,
        lower(trim(status)) as status,
        lower(trim(payment_method)) as payment_method,
        cast(quantity as decimal(10,2)) * cast(unit_price as decimal(10,2)) as total_amount,
        _ingested_at,
        _batch_id
    from source
    where order_id is not null
      and customer_id is not null
)

select * from cleaned

What does this model do?

The read_parquet() function reads all Parquet files from the Bronze transactions path directly from S3. The glob pattern **/*.parquet picks up files across all date partitions.

The cleaned CTE trims whitespace, casts types to their correct format, normalizes status/payment to lowercase, and computes total_amount. Rows with null order_id or customer_id are filtered out.

• Create stg_customers.sql inside dbt/lakehouse/models/silver/ with this content:

{{ config(
    materialized='table',
    description='Cleaned customer data from Bronze layer'
) }}

with source as (
    select * from read_parquet('s3://lakehouse/bronze/customers/**/*.parquet')
),

cleaned as (
    select
        customer_id,
        trim(name) as customer_name,
        lower(trim(email)) as email,
        trim(city) as city,
        trim(state) as state,
        cast(created_at as date) as created_at,
        _ingested_at
    from source
    where customer_id is not null
)

select * from cleaned

What does this model do?

This model reads raw customer Parquet files and applies the same cleaning pattern: trim whitespace, normalize email to lowercase, cast dates, and filter null keys. The name column is renamed to customer_name for clarity when joining with transactions later.

• Create schema.yml inside dbt/lakehouse/models/silver/ with this content:

version: 2

models:
  - name: stg_transactions
    description: "Staged transactions - cleaned, typed, deduplicated"
    columns:
      - name: order_id
        description: "Unique order identifier"
        tests:
          - not_null
          - unique
      - name: customer_id
        description: "Customer who placed the order"
        tests:
          - not_null
      - name: quantity
        tests:
          - not_null
      - name: unit_price
        tests:
          - not_null
      - name: status
        tests:
          - accepted_values:
              values: ['completed', 'pending', 'cancelled']

  - name: stg_customers
    description: "Staged customers - cleaned and typed"
    columns:
      - name: customer_id
        description: "Unique customer identifier"
        tests:
          - not_null
          - unique
      - name: email
        tests:
          - not_null
          - unique

What are these tests?

This YAML defines data quality tests that dbt will run against your Silver tables. The not_null test fails if any row has a null value in that column. The unique test fails if duplicates exist.

The accepted_values test on status ensures only completed, pending, or cancelled appear. Any unexpected value triggers a test failure.

• Now create the Gold layer. Create gold_revenue_by_category.sql inside dbt/lakehouse/models/gold/ with this content:

{{ config(
    materialized='table',
    description='Revenue metrics aggregated by product category'
) }}

with completed_orders as (
    select *
    from {{ ref('stg_transactions') }}
    where status = 'completed'
)

select
    category,
    count(distinct order_id) as total_orders,
    count(distinct customer_id) as unique_customers,
    sum(total_amount) as total_revenue,
    avg(total_amount) as avg_ticket,
    min(order_date) as first_order_date,
    max(order_date) as last_order_date
from completed_orders
group by category
order by total_revenue desc

What does ref() do?

The {{ ref('stg_transactions') }} function is dbt's way of referencing another model. It creates a dependency link so dbt knows to build Silver tables before Gold tables. It also resolves the correct table name at runtime.

• Create gold_top_products.sql inside dbt/lakehouse/models/gold/ with this content:

{{ config(
    materialized='table',
    description='Top selling products by revenue and quantity'
) }}

with completed_orders as (
    select *
    from {{ ref('stg_transactions') }}
    where status = 'completed'
)

select
    product_id,
    product_name,
    category,
    count(distinct order_id) as times_sold,
    sum(quantity) as total_quantity_sold,
    sum(total_amount) as total_revenue,
    avg(unit_price) as avg_unit_price
from completed_orders
group by product_id, product_name, category
order by total_revenue desc

• Create gold_customer_summary.sql inside dbt/lakehouse/models/gold/ with this content:

{{ config(
    materialized='table',
    description='Customer summary with lifetime metrics'
) }}

with orders as (
    select *
    from {{ ref('stg_transactions') }}
    where status = 'completed'
),

customer_metrics as (
    select
        o.customer_id,
        c.customer_name,
        c.city,
        c.state,
        count(distinct o.order_id) as total_orders,
        sum(o.total_amount) as lifetime_revenue,
        avg(o.total_amount) as avg_order_value,
        min(o.order_date) as first_purchase_date,
        max(o.order_date) as last_purchase_date
    from orders o
    left join {{ ref('stg_customers') }} c
        on o.customer_id = c.customer_id
    group by o.customer_id, c.customer_name, c.city, c.state
)

select
    *,
    last_purchase_date - first_purchase_date as customer_lifetime_days
from customer_metrics
order by lifetime_revenue desc

What does the customer summary model do?

This model joins Silver transactions with Silver customers to compute lifetime metrics per customer: total orders, lifetime revenue, average order value, and the time span between their first and last purchase.

• Create schema.yml inside dbt/lakehouse/models/gold/ with this content:

version: 2

models:
  - name: gold_revenue_by_category
    description: "Revenue aggregated by product category"
    columns:
      - name: category
        tests:
          - not_null
          - unique
      - name: total_revenue
        tests:
          - not_null

  - name: gold_top_products
    description: "Products ranked by revenue"
    columns:
      - name: product_id
        tests:
          - not_null
          - unique
      - name: total_revenue
        tests:
          - not_null

  - name: gold_customer_summary
    description: "Customer lifetime value summary"
    columns:
      - name: customer_id
        tests:
          - not_null
          - unique

• Save all the model files you just created.

✔️ Awesome, I've got everything!

Great. Make sure all files are saved before running dbt.

ⓧ I'd like to double check the full code

Here is dbt/lakehouse/models/silver/stg_transactions.sql:

{{ config(
    materialized='table',
    description='Cleaned and typed transactions from Bronze layer'
) }}

with source as (
    select * from read_parquet('s3://lakehouse/bronze/transactions/**/*.parquet')
),

cleaned as (
    select
        order_id,
        customer_id,
        product_id,
        trim(product_name) as product_name,
        trim(category) as category,
        cast(quantity as integer) as quantity,
        cast(unit_price as decimal(10,2)) as unit_price,
        cast(order_date as date) as order_date,
        lower(trim(status)) as status,
        lower(trim(payment_method)) as payment_method,
        cast(quantity as decimal(10,2)) * cast(unit_price as decimal(10,2)) as total_amount,
        _ingested_at,
        _batch_id
    from source
    where order_id is not null
      and customer_id is not null
)

select * from cleaned

Here is dbt/lakehouse/models/silver/stg_customers.sql:

{{ config(
    materialized='table',
    description='Cleaned customer data from Bronze layer'
) }}

with source as (
    select * from read_parquet('s3://lakehouse/bronze/customers/**/*.parquet')
),

cleaned as (
    select
        customer_id,
        trim(name) as customer_name,
        lower(trim(email)) as email,
        trim(city) as city,
        trim(state) as state,
        cast(created_at as date) as created_at,
        _ingested_at
    from source
    where customer_id is not null
)

select * from cleaned

Here is dbt/lakehouse/models/silver/schema.yml:

version: 2

models:
  - name: stg_transactions
    description: "Staged transactions - cleaned, typed, deduplicated"
    columns:
      - name: order_id
        description: "Unique order identifier"
        tests:
          - not_null
          - unique
      - name: customer_id
        description: "Customer who placed the order"
        tests:
          - not_null
      - name: quantity
        tests:
          - not_null
      - name: unit_price
        tests:
          - not_null
      - name: status
        tests:
          - accepted_values:
              values: ['completed', 'pending', 'cancelled']

  - name: stg_customers
    description: "Staged customers - cleaned and typed"
    columns:
      - name: customer_id
        description: "Unique customer identifier"
        tests:
          - not_null
          - unique
      - name: email
        tests:
          - not_null
          - unique

Here is dbt/lakehouse/models/gold/gold_revenue_by_category.sql:

{{ config(
    materialized='table',
    description='Revenue metrics aggregated by product category'
) }}

with completed_orders as (
    select *
    from {{ ref('stg_transactions') }}
    where status = 'completed'
)

select
    category,
    count(distinct order_id) as total_orders,
    count(distinct customer_id) as unique_customers,
    sum(total_amount) as total_revenue,
    avg(total_amount) as avg_ticket,
    min(order_date) as first_order_date,
    max(order_date) as last_order_date
from completed_orders
group by category
order by total_revenue desc

Here is dbt/lakehouse/models/gold/gold_top_products.sql:

{{ config(
    materialized='table',
    description='Top selling products by revenue and quantity'
) }}

with completed_orders as (
    select *
    from {{ ref('stg_transactions') }}
    where status = 'completed'
)

select
    product_id,
    product_name,
    category,
    count(distinct order_id) as times_sold,
    sum(quantity) as total_quantity_sold,
    sum(total_amount) as total_revenue,
    avg(unit_price) as avg_unit_price
from completed_orders
group by product_id, product_name, category
order by total_revenue desc

Here is dbt/lakehouse/models/gold/gold_customer_summary.sql:

{{ config(
    materialized='table',
    description='Customer summary with lifetime metrics'
) }}

with orders as (
    select *
    from {{ ref('stg_transactions') }}
    where status = 'completed'
),

customer_metrics as (
    select
        o.customer_id,
        c.customer_name,
        c.city,
        c.state,
        count(distinct o.order_id) as total_orders,
        sum(o.total_amount) as lifetime_revenue,
        avg(o.total_amount) as avg_order_value,
        min(o.order_date) as first_purchase_date,
        max(o.order_date) as last_purchase_date
    from orders o
    left join {{ ref('stg_customers') }} c
        on o.customer_id = c.customer_id
    group by o.customer_id, c.customer_name, c.city, c.state
)

select
    *,
    last_purchase_date - first_purchase_date as customer_lifetime_days
from customer_metrics
order by lifetime_revenue desc

Here is dbt/lakehouse/models/gold/schema.yml:

version: 2

models:
  - name: gold_revenue_by_category
    description: "Revenue aggregated by product category"
    columns:
      - name: category
        tests:
          - not_null
          - unique
      - name: total_revenue
        tests:
          - not_null

  - name: gold_top_products
    description: "Products ranked by revenue"
    columns:
      - name: product_id
        tests:
          - not_null
          - unique
      - name: total_revenue
        tests:
          - not_null

  - name: gold_customer_summary
    description: "Customer lifetime value summary"
    columns:
      - name: customer_id
        tests:
          - not_null
          - unique

Run dbt to build and test your models

With all models in place, you can now execute dbt inside the Airflow container. The container already has dbt-core and dbt-duckdb installed, and your local dbt/lakehouse/ folder is mounted at /opt/airflow/dbt.

• Run dbt run to build all Silver and Gold tables by running this command:

docker exec airflow-webserver bash -c "cd /opt/airflow/dbt && dbt run --profiles-dir . --project-dir ."

What does this command do?

The docker exec command runs a shell inside the airflow-webserver container. Inside, dbt run compiles your SQL models and executes them against DuckDB. The --profiles-dir . flag tells dbt to look for profiles.yml in the current directory instead of the default ~/.dbt/ path.

You should see output showing each model being built, ending with a summary line like Completed successfully and PASS=5 (5 models total: 2 Silver + 3 Gold).

Seeing a connection or S3 error?

If dbt reports it cannot connect to the S3 endpoint, make sure your LocalStack container is running. Check with docker ps and confirm the localstack container is healthy.

If you see a "Could not find profile" error, verify that the profile value in dbt_project.yml matches the top-level key in profiles.yml exactly (both should be lakehouse).

help me debug my dbt run error against LocalStack S3

• Now run the data quality tests by running this command:

docker exec airflow-webserver bash -c "cd /opt/airflow/dbt && dbt test --profiles-dir . --project-dir ."

You should see all tests passing with a summary showing PASS for every test and zero errors.

Tests failing?

If a not_null or unique test fails, your Bronze data may have quality issues from a previous ingestion run. Re-run the Spark Bronze ingestion job from Step 3 to refresh the data, then try dbt test again.

help me debug my failing dbt tests

Your dbt project is fully operational. Silver tables hold clean, typed data. Gold tables deliver aggregated business metrics. Next up, you will wire everything together with an Airflow DAG that orchestrates the entire pipeline automatically.

Orchestrate the Pipeline with Airflow

You have a working Spark ingestion job and a tested dbt project. Right now, running the full pipeline means executing each piece manually in the right order.

In production, a data pipeline must run autonomously with clear dependency ordering and failure handling. Airflow solves this by letting you define the entire pipeline as a DAG (Directed Acyclic Graph) where each task knows what must succeed before it can start.

In this step, get ready to:

• Create a DAG that orchestrates the full pipeline from S3 bucket creation through dbt tests.
• Configure the Spark connection in Airflow so the SparkSubmitOperator can reach your cluster.
• Trigger the DAG and verify all five tasks complete successfully.

Create the pipeline DAG

An Airflow DAG defines tasks and their execution order. Your DAG will chain five tasks: create the S3 bucket, run Spark ingestion, run dbt transformations, run dbt tests, and generate dbt docs.

• Create the file dags/lakehouse_pipeline.py and paste the following code:

"""
Data Lakehouse Pipeline DAG
Orchestrates: S3 Bucket Creation -> Spark Bronze Ingestion -> dbt Silver/Gold -> dbt Tests
"""
from datetime import datetime, timedelta

from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.providers.apache.spark.operators.spark_submit import SparkSubmitOperator


# Default arguments applied to every task in the DAG
default_args = {
    "owner": "data-engineering",
    "depends_on_past": False,
    "email_on_failure": False,
    "email_on_retry": False,
    "retries": 1,
    "retry_delay": timedelta(minutes=2),
}

with DAG(
    dag_id="lakehouse_medallion_pipeline",
    default_args=default_args,
    description="E2E Data Lakehouse Pipeline - Bronze/Silver/Gold",
    schedule=None,
    start_date=datetime(2024, 1, 1),
    catchup=False,
    tags=["lakehouse", "medallion", "dbt", "spark"],
) as dag:

    # Task 1: Ensure the S3 bucket exists (idempotent with || true)
    create_s3_bucket = BashOperator(
        task_id="create_s3_bucket",
        bash_command=(
            'aws --endpoint-url=http://localstack:4566 s3 mb s3://lakehouse --region us-east-1 || true'
        ),
        env={
            "AWS_ACCESS_KEY_ID": "test",
            "AWS_SECRET_ACCESS_KEY": "test",
            "AWS_DEFAULT_REGION": "us-east-1",
        },
    )

    # Task 2: Submit PySpark job to ingest CSVs into Bronze (Parquet on S3)
    bronze_ingestion = SparkSubmitOperator(
        task_id="bronze_spark_ingestion",
        application="/opt/airflow/spark/bronze_ingestion.py",
        conn_id="spark_default",
        packages="org.apache.hadoop:hadoop-aws:3.3.4,com.amazonaws:aws-java-sdk-bundle:1.12.262",
        conf={
            "spark.hadoop.fs.s3a.endpoint": "http://localstack:4566",
            "spark.hadoop.fs.s3a.access.key": "test",
            "spark.hadoop.fs.s3a.secret.key": "test",
            "spark.hadoop.fs.s3a.path.style.access": "true",
            "spark.hadoop.fs.s3a.impl": "org.apache.hadoop.fs.s3a.S3AFileSystem",
            "spark.hadoop.fs.s3a.connection.ssl.enabled": "false",
        },
    )

    # Task 3: Run dbt to build Silver and Gold models
    dbt_run = BashOperator(
        task_id="dbt_run_silver_gold",
        bash_command="cd /opt/airflow/dbt && dbt run --profiles-dir . --project-dir .",
    )

    # Task 4: Run dbt tests for data quality validation
    dbt_test = BashOperator(
        task_id="dbt_test_quality",
        bash_command="cd /opt/airflow/dbt && dbt test --profiles-dir . --project-dir .",
    )

    # Task 5: Generate dbt documentation with lineage graph
    dbt_docs = BashOperator(
        task_id="dbt_docs_generate",
        bash_command="cd /opt/airflow/dbt && dbt docs generate --profiles-dir . --project-dir .",
    )

    # Define execution order: bucket -> ingest -> transform -> test -> docs
    create_s3_bucket >> bronze_ingestion >> dbt_run >> dbt_test >> dbt_docs

What does this code do?

• default_args sets shared config for all tasks: 1 retry with a 2-minute delay, no email alerts.
• create_s3_bucket uses the AWS CLI to create the lakehouse bucket. The || true makes it idempotent so re-runs don't fail if the bucket already exists.
• bronze_ingestion uses SparkSubmitOperator to submit your PySpark job to the cluster. It passes the S3A configuration so Spark can write to LocalStack.
• dbt_run, dbt_test, and dbt_docs are BashOperators that run dbt commands from inside the Airflow container.
• The >> operator defines the execution chain. Each task only starts after the previous one succeeds.

• Save dags/lakehouse_pipeline.py.

Why schedule=None?

Setting schedule=None means this DAG only runs when you trigger it manually. For a portfolio project, this avoids accidental repeated runs while you iterate on the pipeline.

DAG not appearing in Airflow UI?

The Airflow scheduler scans the dags/ folder periodically. Wait 30-60 seconds and refresh the page.

If it still doesn't appear, check for import errors by clicking the red warning icon at the top of the Airflow UI. A typo or missing import will prevent the DAG from loading.

help me debug why my DAG isn't showing in Airflow

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

"""
Data Lakehouse Pipeline DAG
Orchestrates: S3 Bucket Creation -> Spark Bronze Ingestion -> dbt Silver/Gold -> dbt Tests
"""
from datetime import datetime, timedelta

from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.providers.apache.spark.operators.spark_submit import SparkSubmitOperator


default_args = {
    "owner": "data-engineering",
    "depends_on_past": False,
    "email_on_failure": False,
    "email_on_retry": False,
    "retries": 1,
    "retry_delay": timedelta(minutes=2),
}

with DAG(
    dag_id="lakehouse_medallion_pipeline",
    default_args=default_args,
    description="E2E Data Lakehouse Pipeline - Bronze/Silver/Gold",
    schedule=None,
    start_date=datetime(2024, 1, 1),
    catchup=False,
    tags=["lakehouse", "medallion", "dbt", "spark"],
) as dag:

    create_s3_bucket = BashOperator(
        task_id="create_s3_bucket",
        bash_command=(
            'aws --endpoint-url=http://localstack:4566 s3 mb s3://lakehouse --region us-east-1 || true'
        ),
        env={
            "AWS_ACCESS_KEY_ID": "test",
            "AWS_SECRET_ACCESS_KEY": "test",
            "AWS_DEFAULT_REGION": "us-east-1",
        },
    )

    bronze_ingestion = SparkSubmitOperator(
        task_id="bronze_spark_ingestion",
        application="/opt/airflow/spark/bronze_ingestion.py",
        conn_id="spark_default",
        packages="org.apache.hadoop:hadoop-aws:3.3.4,com.amazonaws:aws-java-sdk-bundle:1.12.262",
        conf={
            "spark.hadoop.fs.s3a.endpoint": "http://localstack:4566",
            "spark.hadoop.fs.s3a.access.key": "test",
            "spark.hadoop.fs.s3a.secret.key": "test",
            "spark.hadoop.fs.s3a.path.style.access": "true",
            "spark.hadoop.fs.s3a.impl": "org.apache.hadoop.fs.s3a.S3AFileSystem",
            "spark.hadoop.fs.s3a.connection.ssl.enabled": "false",
        },
    )

    dbt_run = BashOperator(
        task_id="dbt_run_silver_gold",
        bash_command="cd /opt/airflow/dbt && dbt run --profiles-dir . --project-dir .",
    )

    dbt_test = BashOperator(
        task_id="dbt_test_quality",
        bash_command="cd /opt/airflow/dbt && dbt test --profiles-dir . --project-dir .",
    )

    dbt_docs = BashOperator(
        task_id="dbt_docs_generate",
        bash_command="cd /opt/airflow/dbt && dbt docs generate --profiles-dir . --project-dir .",
    )

    create_s3_bucket >> bronze_ingestion >> dbt_run >> dbt_test >> dbt_docs

Configure the Spark connection in Airflow

The SparkSubmitOperator references conn_id="spark_default". You need to tell Airflow where your Spark master lives by creating this connection.

• Open your browser and go to http://localhost:8080.
• Log in with username airflow and password airflow.
• Click **Admin** in the top navigation bar.
• Click Connections from the dropdown.
• Click the + button (or Add a new record) to create a new connection.
• Fill in the following fields:

• Connection Id: spark_default.
• Connection Type: select Spark from the dropdown.
• Host: spark://spark-master.
• Port: 7077.

• Click Save.

Why spark://spark-master?

All containers in your Docker Compose stack share the lakehouse network. The Airflow container can reach the Spark master using its container name spark-master as a hostname.

The spark:// prefix tells Airflow to submit jobs to a standalone Spark cluster (as opposed to yarn or local mode).

Don't see Spark in the Connection Type dropdown?

The Spark connection type comes from the apache-airflow-providers-apache-spark package. If it's missing, the package didn't install correctly in your custom Airflow image.

Rebuild your Airflow image by running docker compose build airflow-webserver and then restart the containers.

help me fix my Airflow Spark provider installation

Trigger the DAG and verify all tasks succeed

Your DAG is paused by default (the docker-compose.yml sets AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'). You need to unpause it and trigger a manual run.

• Go back to the Airflow home page (DAGs in the top nav).
• Find lakehouse_medallion_pipeline in the DAG list.
• Toggle the switch on the left from paused (grey) to active (blue).
• Click the play button (triangle icon) on the right side of the row.
• Select Trigger DAG from the dropdown.

The DAG run starts executing. Click on the DAG name to enter the Grid view and watch task progress in real time.

• Wait for all five tasks to turn green (success). This may take 2-5 minutes depending on your machine.

What should I see?

In the Grid view, each task shows as a colored square. Green means success. The tasks run in order: create_s3_bucket finishes first, then bronze_spark_ingestion, then dbt_run_silver_gold, then dbt_test_quality, and finally dbt_docs_generate.

Seeing a red (failed) task?

Click on the red task square, then click Log to see the error details. Common issues:

• bronze_spark_ingestion fails with connection refused: your spark_default connection may have a typo in the host or port. Verify it matches spark://spark-master on port 7077.
• dbt_run_silver_gold fails with a DuckDB or httpfs error: the Bronze layer may not have data. Check that the bronze_spark_ingestion task succeeded first.
• create_s3_bucket fails with command not found: ensure boto3 is in your Airflow requirements.txt and the image was rebuilt.

help me debug my failing Airflow DAG task

Your entire Data Lakehouse pipeline now runs with a single click. Next up, you'll generate documentation and create a professional README to make this project portfolio-ready.

Generate dbt Docs and Write the Portfolio README

Your pipeline runs end-to-end, ingesting CSVs into Bronze, transforming through Silver and Gold, and validating data quality. The final piece is making this project presentable.

A professional portfolio project needs two things: auto-generated documentation that proves your data lineage is traceable, and a README that lets anyone clone and run the project in minutes. In this step, you will generate the dbt documentation site, serve it locally to explore the lineage graph, write the portfolio README, and validate the full stack comes up cleanly from scratch.

In this step, get ready to:

• Generate and serve dbt documentation with a data lineage graph.
• Create a professional README.md with architecture overview and quick start instructions.
• Validate the full project starts from scratch with a single command.

Generate and serve dbt documentation

The dbt_docs_generate task in your DAG already creates the documentation artifacts inside the Airflow container. Since the dbt/lakehouse directory is mounted as a volume, those files are also on your host machine.

• Confirm the documentation files were generated by running this command:

ls dbt/lakehouse/target/catalog.json dbt/lakehouse/target/manifest.json dbt/lakehouse/target/index.html

What are these files?

dbt docs generate creates three key files in the target/ directory.

• catalog.json contains metadata about every table and column in your project.
• manifest.json holds the full graph of models, sources, tests, and their dependencies.
• index.html is the interactive single-page app that renders the lineage graph from the JSON files.

• You should see all three files listed without errors.

✔️ I see all three files

The documentation is ready to serve.

ⓧ File not found

The DAG task may not have run yet. Regenerate the docs manually by running:

docker compose exec airflow-webserver bash -c "cd /opt/airflow/dbt && dbt docs generate --profiles-dir . --project-dir ."

Then check for the files again.

• Serve the documentation site locally on port 8888 by running:

python -m http.server 8888 --directory dbt/lakehouse/target

What does this command do?

Python's built-in HTTP server serves the static dbt documentation files from your host machine. The --directory flag points at the target/ folder where the generated artifacts live.

Because your docker-compose.yml mounts ./dbt/lakehouse into the Airflow container, any files the container writes to /opt/airflow/dbt/target/ appear on your host at dbt/lakehouse/target/.

• Open http://localhost:8888 in your browser.
• Click the lineage graph icon (bottom-right corner of the page) to see the data flow from Bronze sources through stg_transactions and stg_customers to the Gold models.
• Press Ctrl+C in your terminal to stop the server when you are done exploring.

Page shows a directory listing instead of docs?

Make sure you are pointing at the correct directory. The index.html file must be at the root of the served directory. Confirm with ls dbt/lakehouse/target/index.html.

help me debug why my dbt docs page is showing a file listing

Create the portfolio README

A good README lets someone understand what the project does, what technologies it uses, and how to run it in under 60 seconds of reading.

• Create a file called README.md in the root of your project folder (the same level as docker-compose.yml) with the following content:

# Data Lakehouse Local: Pipeline E2E com Arquitetura Medalhao

Portfolio-ready local Data Lakehouse demonstrating Medallion Architecture (Bronze/Silver/Gold) with Spark, dbt, Airflow, and LocalStack S3.

## Architecture

[Architecture diagram with CSV -> Spark -> Bronze (S3/LocalStack) -> dbt+DuckDB -> Silver -> Gold -> Airflow orchestration]

## Technologies
- Docker Compose (6 services)
- Apache Spark 3.5 (bitnami/spark)
- Apache Airflow 2.11.2
- dbt Core + dbt-duckdb
- DuckDB
- LocalStack (S3)
- Python 3.11

## Quick Start

```bash
git clone <repo>
cd lakehouse-local
docker compose up -d
# Trigger DAG at http://localhost:8080
```

## Services
- Airflow UI: http://localhost:8080 (airflow/airflow)
- Spark UI: http://localhost:8081
- LocalStack S3: http://localhost:4566

What makes a good portfolio README?

Recruiters and hiring managers spend less than a minute on each repository. The title states the domain (Data Lakehouse) and the approach (Medallion Architecture).

The Quick Start section proves the project actually runs. The Technologies list maps directly to job requirements for data engineering roles.

• Save README.md.
• Confirm the file exists by running:

ls README.md

• You should see README.md in the output.

Validate the full stack from scratch

The final check for any portfolio project is reproducibility. If someone clones your repo and runs a single command, everything should come up healthy.

• Tear down all containers and volumes, then bring the stack back up by running:

docker compose down -v
docker compose up -d

Why tear down with -v?

The -v flag removes named volumes (localstack-data, postgres-data, airflow-logs). This simulates a fresh clone where no prior state exists.

If the project works after a full reset, anyone can reproduce it.

• Wait about 60 seconds for all services to become healthy.
• Verify all containers are running by executing:

docker compose ps

• You should see localstack, spark-master, spark-worker, postgres, airflow-webserver, and airflow-scheduler all showing a healthy or running state.
• Open http://localhost:8080 in your browser and log in with username airflow and password airflow.
• Unpause and trigger the lakehouse_medallion_pipeline DAG.
• Wait for all 5 tasks to complete with green success indicators.

Tasks failing after the fresh start?

The airflow-init container must finish running before the webserver and scheduler start accepting work. Check its logs:

docker compose logs airflow-init

If you see database migration errors, wait 30 seconds and try triggering the DAG again. The Spark connection (spark_default) may also need to be reconfigured in the Airflow UI under Admin then Connections.

help me debug why my DAG tasks are failing after a fresh docker compose up

✔️ Awesome, I've got everything!

Great. Double check you have saved README.md and that your full stack comes up cleanly.

ⓧ I'd like to double check the full code

Here is the complete README.md file:

# Data Lakehouse Local: Pipeline E2E com Arquitetura Medalhao

Portfolio-ready local Data Lakehouse demonstrating Medallion Architecture (Bronze/Silver/Gold) with Spark, dbt, Airflow, and LocalStack S3.

## Architecture

[Architecture diagram with CSV -> Spark -> Bronze (S3/LocalStack) -> dbt+DuckDB -> Silver -> Gold -> Airflow orchestration]

## Technologies
- Docker Compose (6 services)
- Apache Spark 3.5 (bitnami/spark)
- Apache Airflow 2.11.2
- dbt Core + dbt-duckdb
- DuckDB
- LocalStack (S3)
- Python 3.11

## Quick Start

```bash
git clone <repo>
cd lakehouse-local
docker compose up -d
# Trigger DAG at http://localhost:8080
```

## Services
- Airflow UI: http://localhost:8080 (airflow/airflow)
- Spark UI: http://localhost:8081
- LocalStack S3: http://localhost:4566

Secret mission

Data Quality Avancado com Alertas

Seu pipeline ja funciona. Mas o que acontece quando dados ruins entram silenciosamente no lakehouse? Nesta missao secreta, voce vai adicionar testes de qualidade avancados com dbt, configurar alertas automaticos no Airflow quando a qualidade falha, e criar um modelo Gold de analise de cohort de clientes.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

This project runs entirely locally with Docker, so there are no ongoing cloud costs. However, the Docker containers and volumes do consume memory and disk space while running.

Resources you used:

• Docker containers (LocalStack, Spark Master, Spark Worker, Postgres, Airflow Webserver, Airflow Scheduler).
• Docker volumes (localstack-data, postgres-data, airflow-logs).
• Docker images (localstack/localstack:3.8, bitnami/spark:3.5, postgres:16, custom Airflow image).
• Project folder with all source files (dags/, spark/, dbt/, data/, docker/).

Keep everything running

No action needed. Choose this if you're still actively building or experimenting with the pipeline.

• Your Docker containers will keep running in the background.
• Airflow UI remains available at http://localhost:8080.
• Spark Master UI remains available at http://localhost:8081.
• The containers consume approximately 2-3 GB of RAM while running.

Pause - I'll come back to this later

Stop the running containers to free up memory. Your data and volumes are preserved so you can resume later.

• Navigate to your project folder in the terminal.
• Stop all containers by running:

docker compose down

This stops and removes the containers but keeps your volumes (data) intact.

• To resume later, run docker compose up -d from the same project folder.

Delete - I don't want to use this again

Remove all project resources completely. This frees up disk space and removes all data.

• Navigate to your project folder in the terminal.
• Stop containers, remove volumes, and remove images by running:

docker compose down -v --rmi all

This removes all containers, networks, named volumes, and Docker images created by the project.

• Delete the project folder (macOS/Linux):

cd ~/Desktop
rm -rf data-lakehouse-local

• Delete the project folder (Windows):

cd ~\Desktop
rmdir /s /q data-lakehouse-local

• Verify the containers and images are removed by running:

docker ps -a
docker images

You should see no containers or images related to this project in the output.

That's a Wrap!

That's a Wrap!

Parabens! Voce construiu um Data Lakehouse local completo com arquitetura medalha, processamento Spark, transformacoes dbt e orquestracao Airflow.

Voce aprendeu a:

• Construir um pipeline E2E funcional que leva dados de CSV bruto ate metricas de negocio passando por camadas Bronze (Spark/Parquet), Silver e Gold (dbt+DuckDB).
• Implementar a arquitetura medalha com armazenamento em Parquet no S3 (LocalStack) e orquestrar o pipeline inteiro com uma DAG Airflow.
• Integrar testes de data quality com dbt test e gerar documentacao com data lineage automaticamente.
• Secret Mission: Adicionar data quality checks avancados com testes dbt customizados, alertas no Airflow para falhas de qualidade e analise de cohort de clientes.

Ready to quiz yourself?