Medallion Architecture ETL on Databricks

Build a Bronze/Silver/Gold ETL pipeline on Databricks using pure SQL.

Introduction

30 Second Summary

Every online purchase, customer signup, and transaction generates raw data that piles up fast. The real challenge is turning that mess into something a business can actually use.

In this project, you will build an ETL pipeline on Databricks that processes raw retail sales data through three organized layers into analytics-ready tables. You will write about 25 lines of SQL to create the entire pipeline.

What You'll Build

A SQL-based ETL pipeline on Databricks that processes retail sales data through the Medallion architecture, from raw ingestion to analytics-ready tables.

By the end of this project, you'll have:

  • A Bronze layer streaming table that ingests raw data with built-in quality rules to automatically reject bad records.
  • Silver and Gold layer materialized views that clean, join, and aggregate data into analytics-ready tables.
  • A scheduled job that automatically refreshes your entire pipeline on a daily cadence.
  • A Secret Mission challenge where you build a top customers analysis and tighten your data quality rules.

Are there any prerequisites?

You need basic SQL knowledge including SELECT, JOIN, GROUP BY, and WHERE. No Databricks, Spark, Python experience, or cloud provider account is needed.

Sign Up for Databricks Free Edition

Before you can build your ETL pipeline, you need a workspace to run it in. Databricks Free Edition gives you a fully managed, serverless workspace at no cost. No cloud provider account is needed.

In this step, you will create your Databricks account, get familiar with the workspace layout, and locate the catalog where your pipeline will store its data.

In this step, get ready to:
  • Create a Databricks Free Edition account.
  • Explore the workspace navigation.
  • Verify your default catalog and schema.
Create a Databricks Free Edition account

Databricks Free Edition is a perpetually free, serverless workspace. It includes everything you need for this project, including pipelines, jobs, and a built-in data catalog.

  • Open the Databricks Free Edition signup page in your browser.
  • Choose your preferred signup method: email, Sign in with Google, or Sign in with Microsoft.
  • Follow the on-screen prompts to complete your registration.

Databricks automatically creates a serverless workspace for you. Once the setup finishes, you land on the workspace homepage.

What is Databricks Free Edition?

Databricks Free Edition is a no-cost version of the full Databricks platform, designed for learning and experimentation. It uses serverless compute, so there are no clusters to configure. It does have daily usage quotas, but they are more than enough for this project.

Signup not working?

Make sure you are on the Free Edition signup page, not the paid trial page. The Free Edition page does not ask for a credit card or cloud account.

If you already have a Databricks account from a previous project, log in at login.databricks.com and use your existing workspace.

Help me troubleshoot my Databricks Free Edition signup.

Explore the workspace

The workspace homepage is your central hub for everything in Databricks. The left sidebar gives you quick access to the sections you will use throughout this project.

  • Look at the left sidebar. You should see the following categories at the top:
  • Click Workspace in the sidebar. This is where your notebooks, files, and pipeline source code live.
  • Click Catalog in the sidebar. This is the data governance layer where your tables and schemas are organized.
  • Click Jobs & Pipelines in the sidebar. This is where you will manage your ETL pipeline and scheduled jobs later in the project.

What are the other sidebar sections?

You will also see Recents, Compute, and Marketplace in the sidebar. You do not need these for this project. Free Edition handles compute automatically with serverless resources.

Verify your default catalog

Every table your pipeline creates needs a home. In Databricks, tables are organized inside a catalog and a schema. You need to confirm which catalog and schema your workspace uses by default.

  • Click Catalog in the left sidebar.
  • Look for your default catalog in the catalog list. It is typically named main.
  • Click on your catalog name to expand it.
  • Locate the default schema inside it.

This catalog and schema combination is where your pipeline will store the Bronze, Silver, and Gold tables you create in the next steps.

What is a catalog and schema?

A catalog is the top-level container for all your data objects in Databricks. Inside a catalog, schemas group related tables together. Think of it like a filing cabinet (catalog) with drawers (schemas) that hold individual files (tables).

Pro tip: note your catalog name

Write down your catalog name (e.g. main). You will need it later when writing ad-hoc queries in Step 5, because exploration files require fully qualified table names like main.default.daily_orders_by_state_gold.

Don't see a catalog?

If the Catalog page is empty or you see an error, your workspace may still be provisioning. Wait a minute and refresh the page.

If you see a catalog but cannot find the default schema, click the catalog name to expand it. The schema list appears as a nested level under the catalog.

Help me find my default catalog and schema in Databricks.

Your workspace is ready. Next up, you will create your first ETL pipeline and open the pipeline editor where all the action happens.

Create Your ETL Pipeline

Your Databricks workspace is set up and ready to go. Now it's time to create the pipeline that transforms raw sales data into analytics-ready tables.

Databricks uses a feature called Lakeflow Spark Declarative Pipelines for ETL work. You write SQL to declare what your data should look like, and Databricks handles all the orchestration and compute under the hood.

Why declarative pipelines instead of notebooks?

Databricks also supports building ETL in regular notebooks using Apache Spark directly. That approach requires Python or Scala, manual orchestration, and manual checkpoint management.

Declarative pipelines let you focus on what your data should look like instead of how to process it. You get built-in data quality checks, automatic retries, and a visual graph of your data flow.

In this step, get ready to:
  • Create a new ETL pipeline in Databricks.
  • Configure the pipeline's target catalog and schema.
  • Open the pipeline editor with a blank SQL source file.
Create a new ETL pipeline

Every pipeline needs a name and a destination for its output tables. The destination is defined by a catalog and a schema, which organize your data inside Databricks.

  • Click New in the left sidebar.
  • Select ETL Pipeline.
  • Enter retail-medallion-pipeline as the pipeline name.
  • Just below the name, select your default catalog (typically main).
  • Select default as the schema.

What are catalogs and schemas?

A catalog is the top-level container for your data in Databricks. A schema (also called a database) is a folder inside a catalog that holds your tables and views.

Setting these as your pipeline target means every table you create in this pipeline lands in that location automatically.

Don't see your catalog or schema?

  • Make sure you completed Step 1 and verified your default catalog in the Catalog sidebar.
  • If you see a different catalog name than main, that's fine. Select whatever catalog appeared in Step 1.
  • Help me find my catalog and schema in Databricks
Set up your SQL source file
  • Select Start with an empty file.
  • Accept the default folder path for your source files.
  • Select SQL as the language for your first source file.
  • Click Select.

The pipeline editor opens with a blank SQL source file. This is where you'll write all of your pipeline transformations.

Where does this file live?

Your source file is created inside a transformations folder. Any .sql or .py files in this folder are automatically included when the pipeline runs.

Your pipeline is created and ready for code. Next up, you'll write your first SQL transformation to build the Bronze layer.

Build the Bronze Layer

Your pipeline is created and the SQL editor is open. Now it's time to write your first transformation.

Every pipeline built on the Medallion architecture starts with the Bronze layer, where raw data enters the system. You'll ingest sales order data and add a rule that automatically catches bad records before they flow downstream.

In this step, get ready to:
  • Write a streaming table that ingests raw sales order data.
  • Add a data quality constraint that rejects records with invalid dates.
  • Run the pipeline and preview your Bronze table.
Write your Bronze streaming table

The Bronze layer uses a streaming table to pull in raw data files. A streaming table processes files incrementally, meaning it picks up new files each time the pipeline runs without reprocessing old ones.

What is a streaming table?

A streaming table is a special table backed by Delta Lake that remembers which files it has already read. When new data arrives, it only processes the new files. Under the hood, Databricks uses a feature called Auto Loader to handle this automatically.

  • In your empty SQL source file, paste the following code:
-- =============================================
-- Bronze Layer: Ingest raw sales order data
-- =============================================
CREATE OR REFRESH STREAMING TABLE orders_bronze (
  CONSTRAINT valid_order_date
    EXPECT (order_datetime IS NOT NULL AND length(order_datetime) > 0)
    ON VIOLATION DROP ROW
)
AS SELECT *
FROM STREAM read_files(
  "/databricks-datasets/retail-org/sales_orders"
);

What does this code do?

  • CREATE OR REFRESH STREAMING TABLE orders_bronze creates a streaming table named orders_bronze. The _bronze suffix makes the layer explicit in the table name.
  • CONSTRAINT valid_order_date EXPECT (...) defines a data quality rule. It checks that every record has a non-null, non-empty order_datetime value.
  • ON VIOLATION DROP ROW tells the pipeline to silently drop bad records instead of failing the entire run.
  • STREAM read_files("/databricks-datasets/retail-org/sales_orders") reads raw sales order files from a built-in sample dataset. The STREAM keyword enables incremental processing through Auto Loader.
  • SELECT * brings in every column from the raw data files without any transformations. That is exactly what the Bronze layer is for.

Awesome, I've got everything!

Great. Double check you've got the code pasted into your SQL source file before moving on.

I'd like to double check the full code

Here is the full content of your retail_medallion.sql file at this point.

-- =============================================
-- Bronze Layer: Ingest raw sales order data
-- =============================================
CREATE OR REFRESH STREAMING TABLE orders_bronze (
  CONSTRAINT valid_order_date
    EXPECT (order_datetime IS NOT NULL AND length(order_datetime) > 0)
    ON VIOLATION DROP ROW
)
AS SELECT *
FROM STREAM read_files(
  "/databricks-datasets/retail-org/sales_orders"
);
Run the pipeline and preview your data

Your SQL is ready. Time to let Databricks process those raw files and load them into your Bronze table.

  • Click Run file at the top of the pipeline editor to start a pipeline update.

Why Run file and not Run pipeline?

Run file processes only the SQL in the current file. Run pipeline processes all source files in the pipeline. Right now you only have one file, so both would do the same thing. You'll use Run pipeline in the next step when you add more code.

Databricks spins up serverless compute to run your pipeline. This can take 1-2 minutes on Free Edition.

When the update completes, you should see the orders_bronze table appear in the pipeline graph (DAG) on the right side of the editor.

  • Click the orders_bronze table in the DAG to preview the ingested data in the bottom pane.

You should see rows of raw sales order data with columns like customer_id, order_datetime, and order_number. This is your raw, unprocessed Bronze data.

Pipeline update failed?

  • Check for missing parentheses or commas in your SQL. The CONSTRAINT block sits inside the parentheses right after orders_bronze, not after the AS keyword.
  • Make sure you typed STREAMING TABLE, not MATERIALIZED VIEW. Streaming tables require the STREAM keyword in the SELECT statement.
  • If the pipeline seems stuck, give it up to 2 minutes. Free Edition uses serverless compute that needs time to spin up.
  • Still stuck? Help me debug my Bronze streaming table pipeline error.

Your Bronze layer is live and ingesting raw data with built-in quality checks. Next up, you'll build the Silver and Gold layers to clean, join, and aggregate this data into something ready for analytics.

Build the Silver and Gold Layers

Your Bronze layer is live. Raw sales order data is streaming into your pipeline with a quality rule that automatically rejects bad records. That's the foundation.

Raw data on its own is not useful for analysis. In this step, you'll build the Silver and Gold layers to clean, join, and aggregate your data into something a business analyst could actually query.

In this step, get ready to:
  • Create Silver layer materialized views that load customer data and join it with orders.
  • Create a Gold layer materialized view that aggregates daily order counts by state.
  • Run the full pipeline and verify data flows through all four tables in the DAG.
Create the Silver layer materialized views

The Silver layer is where raw data gets cleaned and enriched. You'll create two materialized views: one that loads customer reference data, and one that joins customers with orders to produce a useful dataset.

Why materialized views instead of streaming tables?

Streaming tables are designed for continuously ingesting new data from files. Materialized views are designed for transformations like joins and aggregations that recompute on each pipeline refresh. Since the Silver and Gold layers transform data rather than ingest it, materialized views are the right fit.

  • In the pipeline editor, find your Bronze table code in the SQL file.
  • Below the Bronze table definition, add these two Silver layer materialized views:
-- =============================================
-- Silver Layer: Load customer reference data
-- =============================================
CREATE OR REFRESH MATERIALIZED VIEW customers_silver
AS SELECT *
FROM read_files(
  "/databricks-datasets/retail-org/customers"
);

-- =============================================
-- Silver Layer: Join orders with customers
-- =============================================
CREATE OR REFRESH MATERIALIZED VIEW customer_orders_silver
AS SELECT
  c.customer_id,
  o.order_number,
  c.state,
  date(timestamp(int(o.order_datetime))) AS order_date
FROM orders_bronze o
INNER JOIN customers_silver c
  ON o.customer_id = c.customer_id;

What do these two views do?

The first view (customers_silver) loads raw customer data from the built-in dataset. It uses read_files without the STREAM keyword because materialized views do batch reads, not streaming reads.

The second view (customer_orders_silver) joins the Bronze orders with the customer data on customer_id. It also converts the raw order_datetime (stored as an integer timestamp) into a proper date with date(timestamp(int(...))). This is the core cleaning step of the Silver layer.

You won't see a change in the DAG yet. The pipeline needs to run before these views appear.

Create the Gold layer materialized view

The Gold layer produces analytics-ready tables. Each Gold view answers a specific business question. This one answers: how many orders does each state get per day?

  • Below the Silver views, add this Gold layer materialized view:
-- =============================================
-- Gold Layer: Daily order counts by state
-- =============================================
CREATE OR REFRESH MATERIALIZED VIEW daily_orders_by_state_gold
AS SELECT
  state,
  order_date,
  count(*) AS order_count
FROM customer_orders_silver
GROUP BY state, order_date;

What does this view do?

This view reads from the Silver customer_orders_silver table and groups every order by state and order_date. The count(*) gives you the total number of orders for each state on each day. A business analyst can query this directly without touching the raw data.

You won't see a change yet. The next substep runs the full pipeline to bring everything to life.

Awesome, I've got everything!

Double check you've saved your file. Your SQL file should now contain the Bronze streaming table from the previous step, the two Silver materialized views, and the Gold materialized view.

I'd like to double check the full code

Here is the complete retail_medallion.sql file. Make sure your code matches this exactly.

-- =============================================
-- Bronze Layer: Ingest raw sales order data
-- =============================================
CREATE OR REFRESH STREAMING TABLE orders_bronze (
  CONSTRAINT valid_order_date
    EXPECT (order_datetime IS NOT NULL AND length(order_datetime) > 0)
    ON VIOLATION DROP ROW
)
AS SELECT *
FROM STREAM read_files(
  "/databricks-datasets/retail-org/sales_orders"
);

-- =============================================
-- Silver Layer: Load customer reference data
-- =============================================
CREATE OR REFRESH MATERIALIZED VIEW customers_silver
AS SELECT *
FROM read_files(
  "/databricks-datasets/retail-org/customers"
);

-- =============================================
-- Silver Layer: Join orders with customers
-- =============================================
CREATE OR REFRESH MATERIALIZED VIEW customer_orders_silver
AS SELECT
  c.customer_id,
  o.order_number,
  c.state,
  date(timestamp(int(o.order_datetime))) AS order_date
FROM orders_bronze o
INNER JOIN customers_silver c
  ON o.customer_id = c.customer_id;

-- =============================================
-- Gold Layer: Daily order counts by state
-- =============================================
CREATE OR REFRESH MATERIALIZED VIEW daily_orders_by_state_gold
AS SELECT
  state,
  order_date,
  count(*) AS order_count
FROM customer_orders_silver
GROUP BY state, order_date;
Run the full pipeline

In Step 3, you clicked Run file to run just the Bronze table. This time you'll run the entire pipeline so all four tables get processed together.

  • Click Run pipeline at the top of the pipeline editor.
  • Wait for the pipeline update to complete. This may take a minute or two as Databricks processes all four tables.

The DAG on the right side of the editor now shows all four tables: orders_bronze, customers_silver, customer_orders_silver, and daily_orders_by_state_gold. Arrows connect them, showing how data flows from Bronze through Silver to Gold.

  • Click orders_bronze in the DAG to preview the raw ingested data in the bottom pane.
  • Click customer_orders_silver in the DAG. You should see columns for customer_id, order_number, state, and a clean order_date column.
  • Click daily_orders_by_state_gold in the DAG. You should see state, order_date, and order_count columns with aggregated data.

What is the DAG showing me?

The DAG (Directed Acyclic Graph) is a visual map of your pipeline. Each box is a table or view. Each arrow shows a dependency. customer_orders_silver depends on both orders_bronze and customers_silver, so Databricks knows to process those first. This is the Medallion architecture in action.

Pipeline failed or a table is missing?

  • Check for typos in table names. The FROM orders_bronze and INNER JOIN customers_silver references in customer_orders_silver must match the exact names you used in the Bronze and Silver definitions.
  • Make sure each SQL statement ends with a semicolon (;). A missing semicolon causes Databricks to read two statements as one, which produces a syntax error.
  • Verify that you used CREATE OR REFRESH MATERIALIZED VIEW for the Silver and Gold layers. Using STREAMING TABLE for a view that reads from another table (instead of from files) causes an error.
  • If your daily quota has been exceeded, compute is unavailable until the next day. Try again tomorrow.

Still stuck? Help me debug my pipeline error.

Your data pipeline is complete. Raw sales data flows through four tables, getting cleaned, joined, and aggregated along the way. Next up, you'll query your Gold layer to pull out real business insights.

Query Your Transformed Data

Your pipeline is fully built. Data flows from raw files through Bronze, Silver, and Gold layers.

Now it's time to put that transformed data to work. In this step, you'll create an exploration file to run ad-hoc queries against your Gold layer table and answer real business questions.

In this step, get ready to:
  • Create an exploration file for ad-hoc analysis.
  • Query the Gold layer to find the top states by order volume.
  • Analyze order trends over time.
Create an exploration file

Your pipeline's transformations folder contains the SQL that runs during every pipeline update. For ad-hoc analysis, you need a separate space that won't interfere with your pipeline.

  • In the pipeline assets browser sidebar on the left, click Add.
  • Select Exploration.
  • Enter retail-analysis as the name.
  • Select SQL as the language.
  • Click Select.

This creates a SQL file inside an explorations folder. Files in this folder are not executed during pipeline updates.

Why a separate folder?

Databricks runs every .sql file in the transformations folder as part of your pipeline. If you put exploratory queries there, they would execute on every pipeline update.

The explorations folder keeps your analysis separate from your pipeline logic.

Query your Gold layer table

Exploration files don't inherit your pipeline's catalog and schema settings. You need to use fully qualified table names to reference your pipeline's tables.

Pro tip: check your catalog name

The queries below use main.default as the catalog and schema. If your catalog has a different name (you verified this in Step 1), replace main.default with your actual catalog and schema.

  • In your retail-analysis exploration file, write a query to find the top 10 states by total orders:
-- Top 10 states by total orders
SELECT
  state,
  SUM(order_count) AS total_orders
FROM main.default.daily_orders_by_state_gold
GROUP BY state
ORDER BY total_orders DESC
LIMIT 10;

What does this query do?

  • It reads from your Gold layer table daily_orders_by_state_gold, which already has order counts grouped by state and date.
  • SUM(order_count) rolls up the daily counts into a single total per state.
  • ORDER BY total_orders DESC LIMIT 10 sorts states from highest to lowest and keeps only the top 10.
  • Run the query by pressing Shift+Enter.
  • You should see a table of 10 states with their total order counts, sorted from highest to lowest.

Seeing a table not found error?

  • Check that your catalog and schema names match what you see under Catalog in the left sidebar. Replace main.default with your actual catalog and schema if they differ.
  • Make sure your pipeline ran successfully in Step 4. The Gold table only exists after a successful pipeline update.
  • Verify you're in the exploration file, not the transformations file. Exploration files require fully qualified table names.

Help me fix a table not found error in my Databricks exploration query.

  • Click into the next cell below your results. Write a second query to analyze order volume over time:
-- Order volume over time
SELECT
  order_date,
  SUM(order_count) AS total_orders
FROM main.default.daily_orders_by_state_gold
GROUP BY order_date
ORDER BY order_date;

What does this query do?

  • This query sums up all order counts across every state for each date.
  • ORDER BY order_date sorts the results chronologically so you can spot trends and patterns in order volume.
  • Run the query by pressing Shift+Enter.
  • You should see a table of dates with total order counts, sorted from earliest to latest.

Query returns unexpected results?

  • Make sure you're querying daily_orders_by_state_gold, not customer_orders_silver. The Gold table has the pre-aggregated order_count column you need for SUM().
  • Check for typos in column names. The column is order_count (singular), not order_counts.

Help me debug my exploration query in Databricks.

Awesome, I've got everything!

Double check you've run both queries and can see results for each one.

I'd like to double check the full code

Here is the complete exploration file with both queries:

-- Top 10 states by total orders
SELECT
  state,
  SUM(order_count) AS total_orders
FROM main.default.daily_orders_by_state_gold
GROUP BY state
ORDER BY total_orders DESC
LIMIT 10;

-- Order volume over time
SELECT
  order_date,
  SUM(order_count) AS total_orders
FROM main.default.daily_orders_by_state_gold
GROUP BY order_date
ORDER BY order_date;

Your pipeline's data is now queryable and delivering real insights. Next up, you'll schedule the pipeline to run automatically so the data stays fresh without any manual work.

Schedule Your Pipeline as an Automated Job

Your pipeline transforms raw sales data into analytics-ready tables. You've queried those results to uncover top-performing states and order trends over time.

Right now, the pipeline only runs when you manually click Run pipeline. In this step, you'll schedule it to run automatically on a daily cadence as a Databricks job.

In this step, get ready to:
  • Add an automated daily schedule to your pipeline.
  • Verify the scheduled job runs correctly from the Jobs page.
Add a schedule to your pipeline

The pipeline editor includes a built-in scheduling feature that turns your pipeline into a recurring job. Once scheduled, the pipeline refreshes automatically at your chosen interval.

  • At the top of the pipeline editor, click the Schedule button.
  • In the dialog that appears, click Add schedule.
  • Give the job a name or accept the default.

What's the default schedule?

The schedule defaults to running once per day. To change the frequency or set a specific run time, expand the Advanced section in the schedule dialog.

Save and verify the scheduled job
  • Click Create to save the schedule.
  • Click the schedule name that appears to navigate to the Jobs page.

The Jobs page shows your job details, run history, and next scheduled run time.

  • Click the Run now button to trigger a manual run and verify the job works.

The job triggers a full pipeline refresh. After it completes, you should see a successful run appear in the run history section.

Don't see your schedule?

  • Make sure you clicked Create in the schedule dialog. If you closed the dialog without clicking Create, the schedule was not saved.
  • If the manual run fails, go back to the pipeline editor and click Run pipeline to check that the pipeline itself runs successfully first.
  • Help me troubleshoot my Databricks pipeline schedule.

Secret mission

Build a Top Customers Analysis and Strengthen Data Quality

Your pipeline already cleans and aggregates data, but what about deeper insights? Add a new Gold layer table that identifies your highest-volume customers. Then tighten your Bronze layer with an additional data quality rule that catches even more bad records.

Before you go

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs on Databricks Free Edition, so there are no monetary costs. However, Free Edition has daily usage quotas, and your scheduled job consumes quota each time it runs.

Resources you used:

  • Databricks Free Edition account.
  • ETL pipeline (retail-medallion-pipeline).
  • Bronze streaming table (orders_bronze).
  • Silver materialized views (customers_silver and customer_orders_silver).
  • Gold materialized view (daily_orders_by_state_gold).
  • Exploration file (retail_analysis.sql).
  • Scheduled daily job.

Keep everything running

No action needed. Choose this if you are still actively building on your pipeline or want to keep the scheduled job running.

  • Your pipeline and all tables will remain in your workspace.
  • The scheduled job will continue running daily, consuming some of your Free Edition usage quota each time.
  • If you hit your daily quota limit, compute will be unavailable for the rest of the day but resets the next day.

Pause - I'll come back to this later

Disable the scheduled job so it stops consuming your daily usage quota. Your pipeline and tables stay intact for when you return.

  • Navigate to Jobs in the left sidebar.
  • Find your pipeline's scheduled job and click on it.
  • Click the schedule trigger and select Pause to disable the automatic runs.
  • Your pipeline, tables, and exploration files all stay in your workspace. You can resume the schedule any time.

Delete - I don't want to use this again

Remove all project resources from your workspace and start fresh.

  • Navigate to Jobs in the left sidebar. Find your scheduled job, click the three-dot menu, and select Delete.
  • Navigate to Pipelines in the left sidebar. Find retail-medallion-pipeline, click the three-dot menu, and select Delete.
  • Navigate to Catalog in the left sidebar. Open your catalog and the default schema. Delete the tables: orders_bronze, customers_silver, customer_orders_silver, and daily_orders_by_state_gold.
  • Optionally, navigate to Workspace and delete the transformations and explorations folders.
  • Your Databricks Free Edition account can remain open at no cost. If you want to remove it entirely, visit your account settings.

Mission accomplished

Nice Work!

You just built a production-style ETL pipeline on Databricks. Starting from a blank workspace, you transformed raw retail sales data into analytics-ready tables using the Medallion architecture. The pipeline cleans, joins, and aggregates data automatically, and it runs on a daily schedule without you lifting a finger.

Here's what you accomplished:

  • Built a Medallion architecture pipeline with Bronze, Silver, and Gold layers using pure SQL on Databricks.
  • Added data quality expectations that automatically catch and reject bad records before they reach your analytics tables.
  • Scheduled your pipeline as an automated Databricks job that refreshes your data on a daily cadence.
  • Built a Top Customers Gold layer analysis and added multiple data quality rules to tighten your pipeline.

Ready to quiz yourself?