Medallion Architecture on Azure Databricks

Introduction

30 Second Summary

Every company collects mountains of raw data from apps, sensors, and transactions. Turning that messy pile into clean numbers that executives actually trust is the core job of a data engineer.

In this project, you will build a complete ETL pipeline on Azure Databricks that transforms raw travel booking data through three layers (Bronze, Silver, Gold) using the industry-standard Medallion Architecture pattern.

What You'll Build

You will query a Gold analytics table showing revenue by destination and prove that your pipeline turned messy JSON files into trustworthy business metrics, end to end.

By the end of this project, you'll have:

• Raw booking data landing in a Bronze Delta table via Auto Loader, capturing every record exactly as it arrived.
• A Silver table with nulls filtered, types cast, and duplicates removed, ready for reliable analysis.
• A Gold analytics table showing total revenue, average booking value, and booking counts grouped by destination.
• Secret Mission: Schedule the pipeline as a Lakeflow Job and add data quality expectations that automatically flag or drop bad records.

Are there any prerequisites?

You need an Azure account with a pay-as-you-go subscription (not a Free Trial subscription). The project uses the Azure Databricks 14-day free trial with up to $400 in credits, so there is no cost to complete it.

No prior Databricks experience is required. This project directly prepares you for the Databricks Data Engineer Associate certification exam, covering 74% of its tested domains.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters for the Databricks Data Engineer Associate certification.

Set Up Your Azure Databricks Workspace

To build a Medallion Architecture pipeline, you need a workspace that supports Unity Catalog, Auto Loader, and Lakeflow Jobs. These three features are heavily tested on the Databricks Data Engineer Associate certification exam.

All three require a Premium-tier Azure Databricks workspace. In this step, you will create one using the free 14-day Premium trial, spin up a compute cluster, and verify everything works with a test notebook.

In this step, get ready to:

• Create an Azure Databricks workspace with the Premium trial pricing tier.
• Create a compute cluster to run your code.
• Create and test your first notebook.

Create the Azure Databricks workspace

Azure Databricks runs inside your Azure subscription as a managed service. You need to create a workspace resource first, which acts as the container for all your notebooks, clusters, and data.

• Sign in to the Azure Portal at portal.azure.com.
• On the home page, click Create a resource.
• In the search bar, type Azure Databricks and press Enter.
• Select Azure Databricks from the results.
• Click Create.

You are now on the workspace creation form. Fill in the required fields:

• Select your Azure subscription from the Subscription dropdown.
• Under Resource group, click Create new and enter a name (e.g., databricks-medallion-rg).
• Enter a workspace name (e.g., medallion-workspace).
• Select a region close to you (e.g., West US 2).
• Under Pricing Tier, select Trial (Premium - 14-Days Free DBUs).

Why the Premium trial tier?

The Premium tier unlocks Unity Catalog, Auto Loader, and Lakeflow Jobs. These are the exact features the certification exam tests, and they are not available on lower tiers.

The 14-day trial gives you up to $400 in free DBU credits. That is more than enough for this entire project.

• Click Review + Create.
• Review your settings, then click Create.

Azure begins deploying your workspace. This takes 2-5 minutes. Wait until you see a deployment complete confirmation.

Deployment failed?

Make sure your Azure subscription is a pay-as-you-go subscription, not an Azure Free Trial subscription. The Databricks trial requires pay-as-you-go billing even though the DBUs themselves are free.

Also confirm you have Contributor or Owner permissions on the subscription.

Help me troubleshoot my Azure Databricks workspace deployment failure

Launch the workspace and create a compute cluster

A compute cluster is the engine that runs your code. Without one attached, your notebooks have no processing power behind them.

• On the deployment complete page, click Go to resource.
• Click Launch Workspace.

Your Databricks workspace opens in a new browser tab. You should see a landing page with a left sidebar showing options including Compute, Catalog, and Workflows.

• Click Compute in the left sidebar.
• Click Create compute.
• In the Compute name field, type learning-cluster.
• Leave all other settings at their defaults.
• Click Create compute.

The cluster takes 3-5 minutes to start. Wait until you see a green circle next to learning-cluster. The green circle means it is running and ready to execute code.

Cluster stuck in pending state?

Some regions have limited VM capacity. If your cluster takes more than 10 minutes to start, try terminating it and creating a new one with a different node type or in a different region.

Help me fix my Databricks compute cluster that will not start

Create your first notebook

Notebooks are where you write and run your pipeline code. Each notebook contains cells that you execute individually or all at once.

• Click New in the left sidebar.
• Click Notebook.
• At the top of the notebook, click the default name and rename it to 01_explore_data.
• In the cluster dropdown at the top of the notebook, select learning-cluster.

Before you run your first cell, what do you expect to see if the notebook is correctly connected to the cluster?

• Type the following code into the first cell:

# Cell 1: Test your setup
print("Hello Databricks!")

• Run the cell by pressing Shift+Enter.

You should see Hello Databricks! printed directly below the cell. This confirms your notebook is connected to the running cluster and can execute PySpark code.

Cell not producing output?

Check that the cluster dropdown at the top shows learning-cluster with a green circle. If it shows a grey circle or says Detached, click the dropdown and select your running cluster.

If the cluster terminated due to inactivity, navigate back to the Compute page and restart it.

Help me connect my Databricks notebook to a compute cluster

Your workspace is live, your cluster is running, and your notebook can execute code. Next up, you will explore the raw travel booking dataset and discover the data quality issues your pipeline will fix.

Explore the Raw Data

Your workspace is live, your cluster is running, and your first notebook printed a friendly greeting. Now it's time to look at the data you'll be transforming throughout this project.

Before building a pipeline, you need to understand what the source data actually looks like. Seeing the messiness firsthand (null values, inconsistent casing, duplicates) gives you the "before" picture that makes every cleaning step meaningful.

In this step, get ready to:

• Query the Wanderbricks sample dataset to see what raw booking data looks like.
• Identify data quality issues like null values and duplicates.
• Export raw data as JSON files to simulate a real data landing zone.

Query the Wanderbricks sample dataset with PySpark

Databricks comes with built-in sample datasets accessible through Unity Catalog. The samples.wanderbricks catalog contains a simulated travel booking platform with tables for bookings, users, properties, and destinations.

• Add a new cell below your existing Cell 1 by hovering below it and clicking + Code.
• Load and display the bookings table by pasting this code into the new cell:

# Explore the bookings table
bookings_df = spark.table("samples.wanderbricks.bookings")
display(bookings_df)

• Run the cell by pressing Shift+Enter.

You should see an interactive table of booking records with columns like booking_id, user_id, total_amount, status, and date columns.

What is a PySpark DataFrame?

The spark.table() call returns a PySpark DataFrame. Think of it as a spreadsheet that lives across a cluster of machines. display() renders it as a rich, sortable table directly in the notebook output.

• Add another new cell and check the schema (column names and data types) by pasting this code:

# Check the schema
bookings_df.printSchema()

• Run the cell with Shift+Enter.

You should see a tree of column names and their types printed below the cell. Notice that date columns may appear as strings rather than proper date types.

• Add another new cell and explore the related tables (users and properties) that you'll join later:

# Explore related tables
users_df = spark.table("samples.wanderbricks.users")
display(users_df.limit(10))

properties_df = spark.table("samples.wanderbricks.properties")
display(properties_df.limit(10))

• Run the cell with Shift+Enter.

You should see a table of user records followed by property listings with details like property type, price, and destination.

Why explore related tables now?

In the Gold layer, you will join bookings with properties and destinations to calculate revenue by location. Seeing these tables now helps you understand what columns are available for joining later.

Seeing an error about the table not found?

Make sure you typed samples.wanderbricks.bookings exactly as shown. The samples catalog is pre-loaded in every Unity Catalog-enabled workspace. If you still see an error, confirm your notebook is attached to your running cluster using the dropdown at the top of the notebook.

help me debug why my Databricks notebook can't find the samples.wanderbricks table

Identify data quality issues

Now that you can see the data, it's time to investigate what's wrong with it. Real-world data always arrives messy. Quantifying the problems tells you exactly what your Silver layer needs to fix.

• Add a new cell and count the null values in every column by pasting this code:

# Identify data quality issues
from pyspark.sql.functions import col, count, when, isnan

# Count nulls per column
null_counts = bookings_df.select([
    count(when(col(c).isNull(), c)).alias(c) 
    for c in bookings_df.columns
])
display(null_counts)

• Run the cell with Shift+Enter.

You should see a single row where each column shows the count of null values. Look for columns like total_amount that have non-zero null counts.

What does this code do?

The select() call builds a list comprehension that loops through every column in the DataFrame. For each column, when(col(c).isNull(), c) returns the column value only when it's null, and count() tallies those non-null results. The output is one row showing how many nulls exist per column.

• Add one more cell and check the distribution of booking statuses:

# Check value distributions
display(bookings_df.groupBy("status").count().orderBy("count", ascending=False))

• Run the cell with Shift+Enter.

You should see a table showing each unique status value and how many times it appears. Look for inconsistencies like mixed casing (e.g., "Confirmed" vs "confirmed").

Mental checklist of issues to fix

From these two cells, you can identify the problems your Silver layer will handle: null values in total_amount, date columns stored as strings, potential duplicates on booking_id, and inconsistent status casing.

Write the raw data to a Volume as JSON files

In production systems, raw data typically arrives as files landing in cloud storage. To simulate this, you'll export a subset of bookings as JSON files into a Unity Catalog Volume. This creates the raw landing zone that Auto Loader will ingest in the next step.

• Add a new cell and paste this code to create a schema and write the raw JSON files:

# Write raw data as JSON to a Volume (simulating raw landing zone)
# First, create a schema for your work
spark.sql("CREATE SCHEMA IF NOT EXISTS main.medallion_project")

# Write bookings as JSON files to simulate raw data arriving
(bookings_df.limit(500)
  .write
  .format("json")
  .mode("overwrite")
  .save("/Volumes/main/medallion_project/raw_data/bookings_json")
)

print("Raw JSON files written successfully!")

What does this code do?

The CREATE SCHEMA IF NOT EXISTS statement creates a namespace called main.medallion_project inside Unity Catalog where all your Delta Lake tables will live.

The .limit(500) grabs 500 booking records and writes them as JSON files to a Volume path. This simulates how data arrives in real pipelines: as raw files landing in a storage location.

Before you run this, what do you think the output will be if the schema creation and file write succeed?

• Run the cell with Shift+Enter.

You should see the message Raw JSON files written successfully! printed below the cell. This confirms your schema was created and the JSON files are now sitting in the Volume path, ready for Auto Loader.

Seeing a permissions error or path not found?

If you see an error about insufficient privileges, make sure you're using a workspace where you have write access to the main catalog. In a fresh Premium trial workspace, you should have full permissions by default. If the error mentions the Volume path not existing, that's expected on first run. The write operation creates the path automatically.

help me fix a permissions error when writing to a Unity Catalog Volume in Databricks

✔️ Awesome, I've got everything!

Great. Confirm you've run all 7 cells and each one produced output without errors.

ⓧ I'd like to double check the full code

Your notebook should have 7 cells in order. Scroll through and confirm each one is present:

• Cell 1: print("Hello Databricks!") (from Step 1).
• Cell 2: Load samples.wanderbricks.bookings into bookings_df and display() it.
• Cell 3: bookings_df.printSchema() to see column types.
• Cell 4: Load and display users_df and properties_df (limited to 10 rows each).
• Cell 5: Import pyspark.sql.functions and count nulls per column.
• Cell 6: groupBy("status").count() to check value distributions.
• Cell 7: Create main.medallion_project schema and write 500 rows as JSON to the Volume path.

You now have a clear picture of the raw data and its problems. Next up, you'll build the Bronze layer using Auto Loader to ingest those JSON files into a proper Delta table.

Build the Bronze Layer with Auto Loader

In the previous step, you explored the raw booking data and spotted nulls, inconsistent types, and other messiness. Now it's time to build the first layer of your pipeline: the Bronze layer.

The Bronze layer captures raw data exactly as it arrives, with no transformations. Auto Loader is the Databricks-native way to incrementally ingest files into Delta Lake tables. Its cloudFiles format is directly tested on the Databricks Data Engineer Associate certification exam.

In this step, get ready to:

• Write Auto Loader ingestion code using the cloudFiles format.
• Ingest raw JSON files into a Bronze Delta table with metadata columns.
• Verify the Bronze table exists with the correct row count and explore it in Catalog Explorer.

Create a new notebook and write the Auto Loader ingestion code

Auto Loader uses structured streaming under the hood. You define a streaming read from your JSON files, enrich the data with metadata columns, and then write the stream to a Delta table.

• In the Databricks sidebar, click New, then click Notebook.
• Name it 02_bronze_ingestion.
• Attach it to your learning-cluster compute using the dropdown at the top of the notebook.
• In the first cell, paste the following Auto Loader read code:

# Cell 1: Bronze Layer - Ingest raw JSON with Auto Loader
from pyspark.sql.functions import current_timestamp, input_file_name

# Read raw JSON files using Auto Loader (cloudFiles format)
bronze_df = (spark.readStream
  .format("cloudFiles")
  .option("cloudFiles.format", "json")
  .option("cloudFiles.inferColumnTypes", "true")
  .option("cloudFiles.schemaLocation", "/Volumes/main/medallion_project/checkpoints/bronze_schema")
  .load("/Volumes/main/medallion_project/raw_data/bookings_json")
)

# Add metadata columns for lineage tracking
bronze_with_metadata = (bronze_df
  .withColumn("ingestion_timestamp", current_timestamp())
  .withColumn("source_file", input_file_name())
)

What does this code do?

This cell defines a streaming DataFrame that reads from your raw JSON landing zone.

• spark.readStream.format("cloudFiles") tells Databricks to use Auto Loader, which tracks which files have already been processed so it only ingests new ones.
• cloudFiles.inferColumnTypes set to true means Auto Loader will figure out the correct data types from your JSON rather than treating everything as strings.
• cloudFiles.schemaLocation stores the inferred schema so Auto Loader can detect schema changes on future runs.
• ingestion_timestamp and source_file are metadata columns that let you trace exactly when and from where each record was ingested.

Running this cell alone won't produce any visible output because it only defines the stream. The actual ingestion happens when you write the stream to a Delta table.

• Add a new cell below (click the + button below the first cell) and paste the write stream code:

# Cell 2: Write to Bronze Delta table
(bronze_with_metadata.writeStream
  .format("delta")
  .option("checkpointLocation", "/Volumes/main/medallion_project/checkpoints/bronze_checkpoint")
  .trigger(availableNow=True)
  .toTable("main.medallion_project.bronze_bookings")
)

print("Bronze ingestion complete!")

What does the write stream do?

• .format("delta") writes to a Delta Lake table, giving you time travel, ACID transactions, and schema enforcement.
• checkpointLocation stores progress so Auto Loader knows which files have been processed if you run it again.
• .trigger(availableNow=True) processes all available files right now and then stops the stream. This is the exam-tested pattern for batch-style Auto Loader runs.
• .toTable("main.medallion_project.bronze_bookings") registers the output as a managed Delta table in Unity Catalog.

• Run Cell 1 by clicking the play button (or pressing Shift+Enter).
• Run Cell 2 immediately after.

You should see Bronze ingestion complete! printed below Cell 2 after a few seconds.

Cell 2 showing an error?

• If you see a path-not-found error, confirm that your 01_explore_data notebook ran successfully in the previous step. The JSON files must exist at the Volume path before Auto Loader can read them.
• If you see a permissions error, make sure your notebook is attached to learning-cluster and that the cluster is running (green circle in the Compute page).
• help me debug my Auto Loader cell error

Verify the Bronze table was created

The data has landed in your Bronze Delta table. Now you need to confirm the row count matches and explore what Auto Loader created.

• Add a third cell and paste this verification query:

# Cell 3: Verify the Bronze table
bronze_table = spark.table("main.medallion_project.bronze_bookings")
print(f"Bronze table row count: {bronze_table.count()}")
display(bronze_table.limit(10))

What does this verification do?

This cell reads back the Delta table you just created and checks the row count. The display() call renders a table preview so you can visually confirm the columns and data look correct.

Before you run this cell, how many rows do you expect to see in the Bronze table?

• Run Cell 3.

You should see Bronze table row count: 500 printed, followed by a table showing your booking records with all original columns plus the ingestion_timestamp and source_file metadata columns you added.

• In the Databricks sidebar, click Catalog.
• Expand main then medallion_project then Tables.
• Click on bronze_bookings to open it.
• Explore the Schema tab to see all column names and types that Auto Loader inferred.
• Click the Sample Data tab to preview rows.
• Click the History tab to see the Delta table's version history. You should see one entry from the write you just performed.

What is Delta table history?

Every write to a Delta table creates a new version. Delta Lake tracks these versions automatically, which enables time travel queries (reading data as it existed at a previous point in time). This is a core Delta Lake feature tested on the certification exam.

Don't see bronze_bookings in Catalog Explorer?

• Click the refresh icon at the top of the Catalog panel. Newly created tables sometimes take a moment to appear.
• Confirm Cell 2 printed Bronze ingestion complete! without errors. If it errored, the table was never written.
• help me find my bronze_bookings table in Catalog Explorer

✔️ Awesome, I've got everything!

Great. Double check you have all three cells saved in your 02_bronze_ingestion notebook.

ⓧ I'd like to double check the full code

# Cell 1: Bronze Layer - Ingest raw JSON with Auto Loader
from pyspark.sql.functions import current_timestamp, input_file_name

# Read raw JSON files using Auto Loader (cloudFiles format)
bronze_df = (spark.readStream
  .format("cloudFiles")
  .option("cloudFiles.format", "json")
  .option("cloudFiles.inferColumnTypes", "true")
  .option("cloudFiles.schemaLocation", "/Volumes/main/medallion_project/checkpoints/bronze_schema")
  .load("/Volumes/main/medallion_project/raw_data/bookings_json")
)

# Add metadata columns for lineage tracking
bronze_with_metadata = (bronze_df
  .withColumn("ingestion_timestamp", current_timestamp())
  .withColumn("source_file", input_file_name())
)

# Cell 2: Write to Bronze Delta table
(bronze_with_metadata.writeStream
  .format("delta")
  .option("checkpointLocation", "/Volumes/main/medallion_project/checkpoints/bronze_checkpoint")
  .trigger(availableNow=True)
  .toTable("main.medallion_project.bronze_bookings")
)

print("Bronze ingestion complete!")

# Cell 3: Verify the Bronze table
bronze_table = spark.table("main.medallion_project.bronze_bookings")
print(f"Bronze table row count: {bronze_table.count()}")
display(bronze_table.limit(10))

Your raw data is now sitting in a proper Delta table with full lineage tracking. Next up, you'll build the Silver layer to clean this data and remove all those nulls and duplicates you spotted earlier.

Build the Silver Layer

Your Bronze table now holds all 500 raw booking records exactly as they arrived. The problem is that raw data is messy.

The Silver layer is where you clean and standardize data. The Databricks Data Engineer Associate exam directly tests your ability to handle nulls, cast data types, and deduplicate records. In this step, you will try to run a calculation on the raw data first, see it break, and then fix it with PySpark transformations.

In this step, get ready to:

• Read from Bronze and discover why raw data needs cleaning.
• Apply PySpark transformations to filter nulls, cast types, deduplicate, and standardize.
• Write the cleaned data to a Silver Delta table and verify the results.

Create notebook "03_silver_transform" and read from Bronze

Before applying any transformations, you need to see what happens when you try to use raw Bronze data directly. This is the moment that makes the Silver layer's value obvious.

• In the Databricks sidebar, click New, then click Notebook.
• Name the notebook 03_silver_transform.
• Attach it to your learning-cluster using the compute dropdown at the top of the notebook.

• In the first cell, load the Bronze table and attempt to calculate an average booking value by pasting this code:

# Cell 1: Silver Layer - Read from Bronze
from pyspark.sql.functions import col, to_date, lower, round, current_date

bronze_df = spark.table("main.medallion_project.bronze_bookings")
print(f"Bronze rows: {bronze_df.count()}")

# Try calculating average without cleaning - see the problem
print("Average total_amount (before cleaning):")
display(bronze_df.agg({"total_amount": "avg"}))

What does this code do?

• The import line brings in all the PySpark functions you will use across the entire notebook.
• spark.table("main.medallion_project.bronze_bookings") loads the Bronze Delta table you created in the previous step.
• The .agg({"total_amount": "avg"}) line attempts to calculate the average booking amount from raw, uncleaned data.

Before you run this cell, think about what you expect to happen. If some total_amount values are null, how might that affect the average calculation?

• Run the cell by pressing Shift+Enter.

You will see an average value, but it is calculated while silently skipping nulls. The result is misleading because you do not know how many records were excluded. Dates are also stored as strings, and duplicate bookings inflate the count. This is exactly why the Silver layer exists.

Seeing an error instead of results?

• Confirm your compute cluster shows a green circle (running) in the top dropdown.
• Check that you typed the table name exactly as main.medallion_project.bronze_bookings. A typo in the schema or table name causes a TABLE_OR_VIEW_NOT_FOUND error.
• Help me debug my Bronze table read.

Apply cleaning transformations with PySpark

Now that you have seen the problem, you can fix it. The Silver layer applies a chain of transformations that filter bad records, cast types, and remove duplicates.

• Create a new cell below Cell 1 and paste the cleaning transformation code:

# Cell 2: Apply cleaning transformations
silver_df = (bronze_df
  # Remove records with null total_amount
  .filter(col("total_amount").isNotNull())
  # Remove records with null booking_id  
  .filter(col("booking_id").isNotNull())
  # Cast date columns to proper DATE type
  .withColumn("check_in", to_date(col("check_in")))
  .withColumn("check_out", to_date(col("check_out")))
  # Standardize status to lowercase
  .withColumn("status", lower(col("status")))
  # Remove duplicate bookings
  .dropDuplicates(["booking_id"])
  # Select only the columns we need (drop metadata)
  .select(
    "booking_id", "user_id", "property_id", 
    "status", "guests_count", "total_amount",
    "check_in", "check_out"
  )
)

print(f"Silver rows (after cleaning): {silver_df.count()}")

What does each transformation do?

• .filter(col("total_amount").isNotNull()) drops any record where the payment amount is missing. This ensures every row has a valid number.
• .withColumn("check_in", to_date(col("check_in"))) converts the string date into a proper DATE type so you can do date arithmetic later.
• .withColumn("status", lower(col("status"))) standardizes status values (e.g. "Confirmed" and "confirmed" become the same value).
• .dropDuplicates(["booking_id"]) keeps only one record per booking, removing any accidental duplicates.
• .select(...) drops the metadata columns (ingestion_timestamp, source_file) that are only needed for auditing in Bronze.

• Run Cell 2 by pressing Shift+Enter.

You should see a "Silver rows (after cleaning)" count printed below the cell. This number should be less than or equal to the Bronze row count, because duplicates and null records were removed.

• Add the data quality assertions in a new Cell 3 to confirm the cleaning worked:

# Cell 3: Data quality assertions
assert silver_df.filter(col("total_amount").isNull()).count() == 0, "Found null total_amount!"
assert silver_df.filter(col("booking_id").isNull()).count() == 0, "Found null booking_id!"
print("All data quality checks passed!")

What are these assertions?

Each assert statement checks a condition and crashes the notebook if it fails. If your cleaning left any null values behind, you would see an AssertionError instead of the success message.

This pattern is how production pipelines guard data quality at each layer boundary.

• Run Cell 3.

You should see "All data quality checks passed!" printed. If you see an AssertionError, your filter step in Cell 2 has a typo.

Seeing an AssertionError?

• Go back to Cell 2 and check that your filter uses .isNotNull() (with parentheses at the end). A missing () means the filter never actually runs.
• Make sure you re-ran Cell 2 after making any corrections. The assertions check the silver_df variable, which only updates when Cell 2 executes.
• Help me fix my data quality assertions.

Write the cleaned data to the Silver Delta table

With your data cleaned and validated, you can now persist it as a Delta Lake table that downstream consumers (like the Gold layer) can trust.

• Create Cell 4 and paste the write code:

# Cell 4: Write to Silver Delta table
(silver_df.write
  .format("delta")
  .mode("overwrite")
  .saveAsTable("main.medallion_project.silver_bookings")
)

print("Silver table written successfully!")
display(spark.table("main.medallion_project.silver_bookings").limit(10))

What does this code do?

• .write.format("delta") saves the DataFrame as a Delta table, giving you time travel and ACID transactions.
• .mode("overwrite") replaces the table contents each time you run. This is safe for a batch Silver layer that reprocesses from Bronze.
• .saveAsTable("main.medallion_project.silver_bookings") registers the table in Unity Catalog so other notebooks and users can discover it.

Before running this cell, what do you expect to see in the displayed output? Think about which columns should appear and what the status values should look like.

• Run Cell 4.

You should see "Silver table written successfully!" followed by a table showing 10 rows with exactly 8 columns: booking_id, user_id, property_id, status, guests_count, total_amount, check_in, and check_out. The status column should show all-lowercase values. The date columns should display as proper dates (e.g. 2024-03-15 format). No ingestion_timestamp or source_file columns should be present.

Seeing extra columns or wrong data types?

• If you still see metadata columns like ingestion_timestamp, check that Cell 2 ends with the .select(...) call listing only the 8 columns you want.
• If dates still look like strings, confirm the .withColumn("check_in", to_date(col("check_in"))) line is present in Cell 2 and that you re-ran both Cell 2 and Cell 4.
• Help me debug my Silver table output.

✔️ Awesome, I've got everything!

Great. Double check that you have saved all four cells in your notebook.

ⓧ I'd like to double check the full code

Your notebook should have four cells in order. Scroll through and confirm each one is present:

• Cell 1: imports, Bronze table read, and the average calculation.
• Cell 2: the full cleaning transformation chain ending with a Silver row count print.
• Cell 3: two assert statements and the "All data quality checks passed!" print.
• Cell 4: the Delta write with .saveAsTable("main.medallion_project.silver_bookings") and a display of the first 10 rows.

Why does the Silver layer matter for the exam?

The Data Transformation and Modeling domain is 22% of the Databricks Data Engineer Associate exam. It tests your ability to filter, cast, deduplicate, and standardize data using PySpark DataFrame operations.

The pattern you just built (read from one layer, transform, write to the next layer) is exactly how exam questions frame pipeline problems.

Your data is now clean, typed, and deduplicated. Next up, you will build the Gold layer where you join this cleaned data with property and destination information to produce business-ready revenue analytics.

Build the Gold Layer

Your Silver table now holds clean, deduplicated booking data. But a business stakeholder asking "which destinations generate the most revenue?" cannot get that answer from individual booking rows. They need summarized, aggregated metrics.

The Gold layer is where your pipeline delivers real business value. In this step, you will join your Silver bookings with dimension tables (properties and destinations), compute revenue aggregations using PySpark DataFrames, and write the results to a final Delta Lake table. Computing complex aggregations is a direct exam objective for the Databricks Data Engineer Associate certification.

In this step, get ready to:

• Join Silver bookings with property and destination dimension tables.
• Compute revenue aggregations grouped by destination and property type.
• Write the Gold Delta table and validate your end-to-end pipeline.

Create your Gold notebook and join dimension tables

Your Silver table has booking amounts, but it does not include the destination name or property type. Those details live in separate dimension tables within the samples.wanderbricks catalog. Joining them together creates the enriched dataset you need for meaningful aggregations.

• In your Databricks workspace, click New in the sidebar, then click Notebook.
• Name it 04_gold_aggregations.
• Attach it to your learning-cluster compute resource using the dropdown at the top of the notebook.
• Paste the following code into the first cell to read Silver bookings and join with the properties and destinations tables:

# Cell 1: Gold Layer - Join Silver with dimension tables
from pyspark.sql.functions import col, sum, avg, count, round

silver_df = spark.table("main.medallion_project.silver_bookings")
properties_df = spark.table("samples.wanderbricks.properties")
destinations_df = spark.table("samples.wanderbricks.destinations")

enriched_df = (silver_df
  .join(properties_df, silver_df.property_id == properties_df.property_id, "inner")
  .join(destinations_df, properties_df.destination_id == destinations_df.destination_id, "inner")
  .select(
    silver_df.booking_id,
    silver_df.total_amount,
    silver_df.status,
    silver_df.guests_count,
    properties_df.property_type,
    destinations_df.name.alias("destination_name")
  )
)

print(f"Enriched records: {enriched_df.count()}")

What does this code do?

This cell performs two inner joins to combine your cleaned booking data with property and destination details.

• The first join connects silver_bookings with samples.wanderbricks.properties on property_id, pulling in the property type.
• The second join connects properties with samples.wanderbricks.destinations on destination_id, pulling in the destination name.
• The .select() at the end keeps only the columns you need for aggregation, avoiding ambiguous column references from the joins.

• Run the cell by pressing Shift+Enter.
• You should see a printed count showing how many enriched records were produced from the join.

Seeing zero enriched records?

• Make sure your Silver table exists. Run spark.table("main.medallion_project.silver_bookings").count() in a new cell to confirm it has rows.
• Verify the join columns match. The property_id column must exist in both your Silver table and the properties table.
• help me debug why my Gold layer join is returning zero records

Compute business metrics with PySpark aggregations

Now that you have an enriched dataset combining bookings with destination names, you can answer the business question: which destinations generate the most revenue? The exam tests your ability to use groupBy, agg, and aggregate functions like sum and avg on PySpark DataFrames.

• Create a new cell below and paste this aggregation code:

# Cell 2: Compute business metrics
gold_df = (enriched_df
  .filter(col("status") == "confirmed")
  .groupBy("destination_name", "property_type")
  .agg(
    round(sum("total_amount"), 2).alias("total_revenue"),
    round(avg("total_amount"), 2).alias("avg_booking_value"),
    count("booking_id").alias("booking_count")
  )
  .orderBy(col("total_revenue").desc())
)

display(gold_df)

What does this code do?

• .filter(col("status") == "confirmed") keeps only completed bookings. Cancelled or pending bookings should not count toward revenue.
• .groupBy("destination_name", "property_type") creates groups so you can see revenue broken down by both destination and property type.
• The .agg() block computes three metrics per group: total revenue, average booking value, and booking count. The round(..., 2) ensures clean decimal formatting.
• .orderBy(col("total_revenue").desc()) sorts results so the highest-revenue combinations appear at the top.

• Run the cell.
• You should see a table displayed with columns for destination_name, property_type, total_revenue, avg_booking_value, and booking_count, sorted by revenue from highest to lowest.

Why filter to confirmed only?

A Gold table serves business users who need accurate revenue figures. Including cancelled bookings would inflate revenue numbers and mislead decision-making. Filtering at the Gold layer keeps the Silver layer as a complete historical record while ensuring Gold delivers trustworthy metrics.

Seeing an empty table or an error?

• If the table is empty, check that your Silver data actually has rows with status confirmed. Run enriched_df.groupBy("status").count().display() to see the distribution.
• If you see a NameError for enriched_df, make sure you ran the first cell (Cell 1) in this notebook before running Cell 2.
• help me debug my Gold layer aggregation

Write the Gold table and validate end-to-end

The aggregation looks correct in the display output. Now you will persist it as a Delta table so other tools and users can query it directly. This final write completes your three-layer Medallion Architecture pipeline.

• Create a new cell and paste this code to write your Gold table:

# Cell 3: Write Gold table
(gold_df.write
  .format("delta")
  .mode("overwrite")
  .saveAsTable("main.medallion_project.gold_revenue_by_destination")
)

print("Gold table written successfully!")
print(f"Gold table rows: {gold_df.count()}")

What does this code do?

• .write.format("delta") saves as a Delta table, giving you versioning, time travel, and ACID transactions.
• .mode("overwrite") replaces the table contents each time. This is appropriate for Gold tables that are fully recomputed from Silver.
• .saveAsTable("main.medallion_project.gold_revenue_by_destination") registers the table in Unity Catalog so it is governed, discoverable, and queryable by others.

• Run the cell. You should see the messages Gold table written successfully! and a row count printed.

Now for the final proof that your entire pipeline works. Before you run the validation query, think about this: what do you expect to see in the output? You started with messy raw JSON, cleaned it in Silver, and aggregated confirmed bookings in Gold. What columns and sort order should appear?

• Create a new cell and paste this final validation query:

# Cell 4: Final validation - Top 10 destinations by revenue
print("Top 10 Destinations by Revenue:")
display(spark.table("main.medallion_project.gold_revenue_by_destination").limit(10))

• Run the cell.

You should see a clean summary table showing destination names, property types, total revenue, average booking value, and booking count. The rows are sorted by total revenue descending. This is your finished product: clean business intelligence produced from raw, messy JSON data.

Gold table not showing results?

• Make sure Cell 3 completed without errors. Look for the Gold table written successfully! confirmation before running Cell 4.
• If the count is 0, revisit Cell 2. Your Silver data may not have any rows with status confirmed.
• help me figure out why my Gold table query returns no data

✔️ Awesome, I've got everything!

Great. Double check you have saved all four cells in your 04_gold_aggregations notebook and that each cell ran without errors.

ⓧ I'd like to double check the full code

Scroll through your 04_gold_aggregations notebook and confirm these four cells are present in order:

• Cell 1: Imports, reads three tables (silver_bookings, properties, destinations), performs two inner joins, selects columns, and prints the enriched record count.
• Cell 2: Filters to confirmed status, groups by destination_name and property_type, computes total_revenue/avg_booking_value/booking_count, orders by total_revenue descending, and calls display(gold_df).
• Cell 3: Writes gold_df as a Delta table named main.medallion_project.gold_revenue_by_destination with mode overwrite, and prints the row count.
• Cell 4: Reads the Gold table back with spark.table() and displays the top 10 rows.

Your Medallion Architecture pipeline is now complete. Raw JSON became Bronze, Bronze became Silver, and Silver became Gold. Next up, you will take this pipeline further in the Secret Mission by scheduling it as a Lakeflow Job and adding data quality expectations.

Secret mission

Schedule as a Lakeflow Job and Add Data Quality

In production, pipelines run on a schedule and enforce data quality rules that automatically flag or drop bad records. Can you wire your three notebooks into a single Lakeflow Job and add quality gates to the Silver layer?

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project uses an Azure Databricks Premium trial that provides $400 in free DBU credits for 14 days, but your compute cluster charges DBUs while it is running.

Cost warning

Your Azure Databricks workspace is free during the 14-day trial period (up to $400 in credits). After the trial expires, any running compute will incur charges per DBU.

If you are not actively using the workspace, terminate your compute cluster or delete the workspace entirely to avoid unexpected billing.

Resources you used:

• Azure Databricks workspace (Premium trial): free for 14 days, then charged per DBU usage.
• Compute cluster (learning-cluster): charges DBUs while running. Auto-terminates after 120 minutes of inactivity by default.
• Delta tables (bronze_bookings, silver_bookings, gold_revenue_by_destination): stored in Unity Catalog managed storage within your workspace.
• Notebooks (01_explore_data, 02_bronze_ingestion, 03_silver_transform, 04_gold_aggregations): stored in the Databricks workspace.
• Lakeflow Job: the scheduled workflow with three sequential tasks, stored in the Databricks workspace.

Keep everything running

No action needed. Choose this if you are still actively building or experimenting with your pipeline.

• Your workspace persists after the 14-day trial, but compute will incur DBU charges once the $400 credit is exhausted.
• The trial gives you $400 in free DBUs, which is more than enough for continued learning and experimentation.
• Your compute cluster auto-terminates after 120 minutes of inactivity, so leaving it idle will not accumulate charges indefinitely.
• All notebooks, Delta tables, and the Lakeflow Job remain available in your workspace.

Pause - I'll come back to this later

Shut down your compute cluster to stop DBU charges while keeping all your work intact.

• In your Databricks workspace, click Compute in the left sidebar.
• Find learning-cluster in the list and click the stop icon (square) to terminate it.
• Wait for the cluster status to show Terminated.
• Your notebooks, Delta tables, Unity Catalog schema, and Lakeflow Job all persist at no additional compute cost.
• When you return, click the start icon on your cluster and it will spin back up in a few minutes.

Delete - I don't want to use this again

Remove the entire Azure Databricks workspace to stop all billing. This deletes all notebooks, tables, jobs, and compute inside the workspace.

• Open the Azure Portal at portal.azure.com.
• Click Resource groups in the left sidebar (or search for it in the top search bar).
• Click on the resource group that contains your Databricks workspace.
• Find the Azure Databricks resource in the list and click on it.
• Click Delete in the top toolbar.
• Type the workspace name to confirm deletion, then click Delete.
• Wait for the deletion to complete. This removes the workspace, all notebooks, all Delta tables (bronze_bookings, silver_bookings, gold_revenue_by_destination), the Lakeflow Job, the compute cluster, and all Volume data.
• This stops all billing immediately.

That's a Wrap!

That's a Wrap!

Nice work! You just built a production-style Medallion Architecture pipeline on Azure Databricks that transforms raw travel booking data into clean, business-ready analytics tables.

You've learned how to:

• Build a complete Medallion Architecture pipeline (Bronze, Silver, Gold) that incrementally ingests raw JSON with Auto Loader's cloudFiles format and writes to Delta Lake tables governed by Unity Catalog.
• Clean and transform data using PySpark DataFrame operations including filtering nulls, casting types, deduplicating records, joining dimension tables, and computing aggregated business metrics.
• Query a Gold analytics table showing total revenue, average booking value, and booking counts grouped by destination and property type, proving the entire pipeline works end-to-end.
• Secret Mission: Scheduled the entire pipeline as a multi-task Lakeflow Job and added data quality assertions, covering two more exam domains (Lakeflow Jobs 16% and Governance 15%).

Ready to quiz yourself?