Build an ETL Pipeline with Databricks

Build an ETL pipeline that processes NYC taxi data into Delta Lake tables.

Introduction

30 Second Summary

Every time you check a weather app, scroll a social feed, or get a product recommendation, a data pipeline is working behind the scenes to collect raw data, clean it up, and deliver it in a useful format.

In this project, you will build an ETL (Extract, Transform, Load) data pipeline on Databricks that processes millions of New York City taxi trip records, cleans the data, engineers new features, and loads the results into a production-ready table you can query and visualize.

What You'll Build

An end-to-end ETL data pipeline on Databricks that transforms raw NYC taxi trip data into a clean, enriched Delta Lake table with SQL analysis and visualizations.

By the end of this project, you'll have:

  • A Databricks notebook containing your complete ETL pipeline in Python.
  • A cleaned and enriched Delta Lake table stored in Unity Catalog with new features like trip duration, cost per mile, and time-of-day categories.
  • SQL queries that analyze trip patterns, costs, and trends, plus built-in visualizations showing your results as bar charts.
  • A secret mission to schedule your pipeline as an automated Databricks job that runs on a recurring schedule.

Are there any prerequisites?

You just need basic familiarity with Python (variables, functions, basic data types) and a web browser. No Databricks experience is needed. The project uses Databricks Free Edition, which is permanently free and does not require a credit card or cloud account.

Sign Up for Databricks Free Edition

Before you can build an ETL pipeline, you need a place to build it. Databricks is the industry-standard data platform that companies use for data engineering, and its Free Edition gives you the full workspace at no cost.

In this step, you will create your Databricks account, explore the workspace, and set up the notebook where your entire pipeline will live.

In this step, get ready to:
  • Create your Databricks Free Edition account.
  • Tour the Databricks workspace and locate key features.
  • Create your first notebook for the ETL pipeline.
Create your Databricks Free Edition account

Databricks Free Edition is a permanently free, serverless workspace designed for learning and experimentation. No credit card or cloud provider account is required.

  • Open the Databricks Free Edition signup page in your browser.
  • Choose your preferred signup method: email, Sign in with Google, or Sign in with Microsoft.
  • Complete the signup form and verify your email if prompted.
  • Wait for Databricks to create your workspace. This typically takes under five minutes.

Once your workspace is ready, Databricks takes you directly to the homepage.

What is Databricks Free Edition?

Databricks Free Edition is a no-cost version of the same platform used by data teams at thousands of companies. It includes serverless compute, Unity Catalog for data governance, notebooks, a SQL editor, and built-in sample datasets. The workspace runs entirely in the cloud, so there is nothing to install on your computer.

Signup not working?

If you see a 401 error or the page hangs after clicking Continue, try opening the signup page in an incognito or private browser window. Disable any VPN or ad blocker temporarily.

Make sure you are using the most recent verification code from your email, not an older one.

Help me troubleshoot my Databricks Free Edition signup.

Tour the Databricks workspace

The Databricks workspace is your central hub for everything in this project. Take a minute to get oriented before you start building.

  • Look at the left sidebar. The top-level categories you will use most are Workspace, Catalog, and Compute.
  • Click Compute in the sidebar.

You should see that serverless compute is available. Free Edition uses serverless compute exclusively, so there is no cluster to configure or manage.

  • Click Catalog in the sidebar to open the Catalog Explorer.
  • Find the samples catalog in the catalog panel and click to expand it.
  • Expand the nyctaxi schema. You should see a table called trips.

This is the NYC taxi trip dataset you will use for your ETL pipeline. It is pre-loaded in every Databricks workspace through Unity Catalog, so there is nothing to download or upload.

What is Unity Catalog?

Unity Catalog is Databricks' data governance layer. It organizes all your data into a three-level hierarchy: catalog, schema, and table. The samples catalog is a built-in collection of sample datasets that Databricks provides for learning. You will also get your own workspace catalog where you can create tables later.

What is the samples catalog?

The samples catalog contains several built-in datasets. The nyctaxi schema has taxi trip records for New York City. Other schemas include tpch (a benchmark dataset) and wanderbricks (simulated travel booking data).

Don't see the samples catalog?

Make sure you are looking in the Catalog Explorer (click Catalog in the sidebar). The samples catalog should appear alongside your workspace catalog. If your workspace is still being provisioned, wait a few minutes and refresh the page.

Help me find the samples catalog in my Databricks workspace.

Create your first notebook

Databricks notebooks are interactive documents where you write and run code cell by cell. Your entire ETL pipeline will live in a single notebook.

  • Click + New in the left sidebar.
  • Click Notebook from the menu.

A blank notebook opens in your workspace. Databricks automatically attaches it to serverless compute.

  • Click the notebook title at the top of the page (it will say something like Untitled Notebook).
  • Replace the title with NYC Taxi ETL Pipeline.
  • Check the language selector at the top of the notebook. Make sure it is set to Python. If it shows a different language, click the dropdown and select Python.

Why Python?

Databricks Free Edition supports Python and SQL. You will use Python (specifically PySpark) for the ETL pipeline because it gives you full control over data transformations. You will switch to SQL later when querying the finished table.

Notebook not connecting to compute?

Free Edition uses serverless compute, which starts automatically when you run a cell. If you see a message about compute being unavailable, you may have exceeded your daily quota. Wait until the next day for the quota to reset, then try again.

Help me connect my Databricks notebook to compute.

Your workspace is ready and your notebook is set up. Next up, you will write your first PySpark code to extract the NYC taxi trip data from Unity Catalog.

Extract Raw Data from Unity Catalog

In the previous step, you set up your Databricks Free Edition account and created a notebook called NYC Taxi ETL Pipeline.

Now it is time to start building your pipeline. Every ETL pipeline begins with the Extract step, where you pull raw data from a source. Your source is a built-in NYC taxi trip dataset that Databricks provides through Unity Catalog. You will load this dataset into a PySpark DataFrame and explore its structure to understand what you are working with.

In this step, get ready to:
  • Load the NYC taxi trip dataset from Unity Catalog into a PySpark DataFrame.
  • Profile the raw data to understand its shape, quality, and what needs cleaning.
Load the NYC taxi dataset into a DataFrame

Databricks comes with several sample datasets pre-loaded in the samples catalog. The one you will use is samples.nyctaxi.trips, which contains records of taxi rides across New York City.

  • Click into the first empty cell in your NYC Taxi ETL Pipeline notebook.
  • Copy the following code into the cell to load the taxi data into a DataFrame:
# Load the NYC taxi trip dataset from Unity Catalog
df = spark.read.table("samples.nyctaxi.trips")

# Display the first few rows to preview the data
display(df)

What does this code do?

  • spark.read.table("samples.nyctaxi.trips") reads the entire taxi trips table from Unity Catalog into a PySpark DataFrame called df.
  • display(df) is a Databricks-specific function that renders the DataFrame as an interactive table directly in your notebook.
  • Run the cell by pressing Shift+Enter.

You should see a table appear below the cell with columns like tpep_pickup_datetime, tpep_dropoff_datetime, trip_distance, fare_amount, pickup_zip, and dropoff_zip.

What do these columns mean?

  • tpep_pickup_datetime and tpep_dropoff_datetime are the timestamps for when each taxi trip started and ended.
  • trip_distance is the distance of the trip in miles.
  • fare_amount is the fare charged for the trip in US dollars.
  • pickup_zip and dropoff_zip are the ZIP codes where the passenger was picked up and dropped off.

Not seeing any data?

  • Make sure you typed the table name exactly as samples.nyctaxi.trips with no extra spaces or typos.
  • If you see a "connecting to compute" message, wait a moment. Databricks serverless compute can take 10-30 seconds to start up on the first run.
  • Help me debug why my notebook cell is not returning data.
Check the schema of the DataFrame

Before profiling the data, it helps to see the exact column names and data types that PySpark inferred from the table.

  • Add a new cell below the previous one and paste the following code to print the schema:
# Print the column names and data types
df.printSchema()
  • Run the cell by pressing Shift+Enter.

You should see a tree-like output listing each column name, its data type (like timestamp, double, integer), and whether it is nullable.

What is a schema?

In PySpark, a schema describes the structure of your data. It lists every column name and its data type. Checking the schema early helps you catch issues before they cause errors later in the pipeline.

Profile the raw data

Profiling means running quick checks to understand the quality and shape of your data. You need to know how many rows you have, whether any values are missing, and whether there are obviously bad records.

  • Add a new cell and paste the following code to count the total number of rows:
# Count the total number of rows in the dataset
print(f"Total rows: {df.count()}")
  • Run the cell by pressing Shift+Enter.

You should see the total row count printed below the cell. This tells you the scale of the dataset you are working with.

  • Add another new cell and paste the following code to check for null values and bad records:
from pyspark.sql.functions import col, sum as spark_sum

# Count null values in each column
null_counts = df.select(
    [spark_sum(col(c).isNull().cast("int")).alias(c) for c in df.columns]
)
print("Null values per column:")
display(null_counts)

# Count records with zero or negative trip distance
bad_distance = df.filter((col("trip_distance") <= 0)).count()
print(f"Rows with zero or negative trip distance: {bad_distance}")

# Count records with zero or negative fare
bad_fare = df.filter((col("fare_amount") <= 0)).count()
print(f"Rows with zero or negative fare amount: {bad_fare}")

What does this code do?

  • The first block counts how many null (missing) values exist in each column. It loops through every column, checks if each value is null, and sums the results.
  • The second block counts rows where trip_distance is zero or negative. A taxi trip with no distance is invalid data.
  • The third block does the same check for fare_amount. A trip that costs nothing or has a negative fare is also invalid.
  • These counts tell you exactly what needs cleaning in the Transform step.
  • Run the cell by pressing Shift+Enter.

You should see the null counts displayed as a table, followed by the counts of bad distance and bad fare records printed below it.

Seeing an import error?

  • Make sure the import line reads from pyspark.sql.functions import col, sum as spark_sum exactly. The as spark_sum alias avoids a conflict with Python's built-in sum function.
  • Help me fix my PySpark import error.

Awesome, I've got everything!

Great. Double check you have saved your notebook before moving on.

I'd like to double check the full code

Here is the complete code you should have across your notebook cells so far:

# Cell 1: Load the NYC taxi trip dataset from Unity Catalog
df = spark.read.table("samples.nyctaxi.trips")

# Display the first few rows to preview the data
display(df)
# Cell 2: Print the column names and data types
df.printSchema()
# Cell 3: Count the total number of rows in the dataset
print(f"Total rows: {df.count()}")
# Cell 4: Profile the data for quality issues
from pyspark.sql.functions import col, sum as spark_sum

# Count null values in each column
null_counts = df.select(
    [spark_sum(col(c).isNull().cast("int")).alias(c) for c in df.columns]
)
print("Null values per column:")
display(null_counts)

# Count records with zero or negative trip distance
bad_distance = df.filter((col("trip_distance") <= 0)).count()
print(f"Rows with zero or negative trip distance: {bad_distance}")

# Count records with zero or negative fare
bad_fare = df.filter((col("fare_amount") <= 0)).count()
print(f"Rows with zero or negative fare amount: {bad_fare}")

You now have a clear picture of the raw data. You know how many rows you are working with, what the columns look like, and where the quality issues are. Next up, you will use PySpark to clean this data and engineer new features.

Transform the Data with PySpark

In the previous step, you loaded the raw NYC taxi dataset and profiled it to spot quality issues like null values and invalid records.

Raw data is rarely ready for analysis. Trips with zero distance, negative fares, or missing values would skew any results you build on top of them. In this step, you will clean out those bad records and engineer new columns that make the data more useful.

In this step, get ready to:
  • Clean and filter out invalid taxi trip records.
  • Engineer new features like trip duration, cost per mile, and time-of-day category.
  • Verify the transformations by comparing before and after row counts.
Clean and filter the data

The profiling you did in Step 2 revealed records with null values, zero distances, and negative fares. These invalid records need to be removed before you can trust any analysis built on this data.

  • Add a new code cell by clicking the + Code button at the bottom of your last cell.
  • Import the PySpark functions you need and clean the data by pasting this code:
# Import PySpark functions for transformations
from pyspark.sql.functions import col, when, hour, round, unix_timestamp

# Remove rows with null values in key columns
df_clean = df.dropna(subset=["trip_distance", "fare_amount"])

# Filter out trips with zero or negative distance
df_clean = df_clean.filter(col("trip_distance") > 0)

# Filter out trips with zero or negative fare
df_clean = df_clean.filter(col("fare_amount") > 0)

# Preview the cleaned data
display(df_clean)

What does this code do?

  • The from pyspark.sql.functions import ... line loads the helper functions you will use throughout this step. These are pre-installed in every Databricks notebook.
  • dropna(subset=[...]) removes any row where trip_distance or fare_amount is null.
  • The two .filter() calls keep only rows where distance and fare are both positive. A trip with zero distance or a negative fare is a data entry error.
  • Press Shift+Enter to run the cell.
  • You should see a table preview where every row has a positive trip_distance and a positive fare_amount.

Seeing a NameError for df?

This means the cell from Step 2 that created the df variable has not been run yet. Scroll up to the cell containing df = spark.read.table("samples.nyctaxi.trips") and run it first, then come back to this cell.

Help me fix a NameError in my Databricks notebook.

Engineer new features

Clean data is a good start, but the raw columns alone do not answer questions like "how long was this trip?" or "was this trip cost-efficient?" You will now add calculated columns that unlock those insights.

  • Add a new code cell and calculate trip duration in minutes by pasting this code:
# Calculate trip duration in minutes from pickup and dropoff timestamps
df_transformed = df_clean.withColumn(
    "trip_duration_minutes",
    round(
        (unix_timestamp(col("tpep_dropoff_datetime")) -
         unix_timestamp(col("tpep_pickup_datetime"))) / 60,
        2
    )
)

What does this code do?

  • unix_timestamp() converts each timestamp into the number of seconds since January 1, 1970. Subtracting pickup from dropoff gives the trip length in seconds.
  • Dividing by 60 converts seconds to minutes. round(..., 2) keeps the result to two decimal places for readability.
  • Press Shift+Enter to run the cell.

You will not see output yet because withColumn() defines the transformation without displaying results. The next cell adds the remaining columns and previews everything together.

  • Add a new code cell and create the cost-per-mile and time-of-day columns by pasting this code:
# Calculate cost per mile (fare divided by distance)
df_transformed = df_transformed.withColumn(
    "cost_per_mile",
    round(col("fare_amount") / col("trip_distance"), 2)
)

# Categorize trips by time of day based on pickup hour
df_transformed = df_transformed.withColumn(
    "time_of_day",
    when(hour(col("tpep_pickup_datetime")).between(6, 11), "Morning")
    .when(hour(col("tpep_pickup_datetime")).between(12, 17), "Afternoon")
    .when(hour(col("tpep_pickup_datetime")).between(18, 21), "Evening")
    .otherwise("Night")
)

# Preview the transformed data with all new columns
display(df_transformed)

What does this code do?

  • cost_per_mile divides the fare by the trip distance. This tells you how expensive each mile of a trip was.
  • time_of_day uses when() to check the pickup hour and assign a label: Morning (6-11), Afternoon (12-17), Evening (18-21), or Night (everything else).
  • hour() extracts the hour (0-23) from the pickup timestamp. The .between() method checks if the hour falls within a range.
  • Press Shift+Enter to run the cell.
  • You should see a table preview with three new columns at the end: trip_duration_minutes, cost_per_mile, and time_of_day. Scroll right in the table if needed.

New columns showing null values?

If trip_duration_minutes shows null for every row, check that you used tpep_pickup_datetime and tpep_dropoff_datetime as the column names. A typo like pickup_datetime would silently produce nulls instead of raising an error.

Help me debug null values in my PySpark transformation.

What is feature engineering?

Feature engineering is the process of creating new columns from existing data to make it more useful for analysis or machine learning. The raw dataset only had timestamps and a fare. By computing trip_duration_minutes, cost_per_mile, and time_of_day, you turned raw numbers into columns that directly answer business questions.

Verify the transformations

Before moving on, you should confirm exactly how many records survived the cleaning step. Comparing the raw and cleaned row counts tells you how much bad data was in the original dataset.

  • Add a new code cell and compare the row counts by pasting this code:
# Count rows before and after cleaning
raw_count = df.count()
cleaned_count = df_transformed.count()
removed_count = raw_count - cleaned_count

print(f"Raw records: {raw_count}")
print(f"Cleaned records: {cleaned_count}")
print(f"Records removed: {removed_count}")

What does this code do?

  • .count() returns the total number of rows in a DataFrame. Running it on both the raw and transformed DataFrames lets you see exactly how many bad records were removed.
  • The difference between the two counts is the number of invalid trips that would have skewed your analysis.
  • Press Shift+Enter to run the cell.
  • You should see three printed lines showing the raw count, cleaned count, and how many records were removed.

Cell taking a long time to run?

The .count() operation triggers a full scan of the data. On serverless compute, this can take 30-60 seconds the first time as the cluster warms up. Wait for the cell to finish before moving on.

Help me understand why my Databricks cell is slow.

Take a look at your full notebook so far. You can verify that every cell from this step has been run successfully.

Awesome, I've got everything!

Double check you have run every cell in this step before moving on.

I'd like to double check the full code

Here is all the code from this step combined. Each block should be in its own notebook cell.

# --- Cell 1: Import functions and clean data ---
from pyspark.sql.functions import col, when, hour, round, unix_timestamp

# Remove rows with null values in key columns
df_clean = df.dropna(subset=["trip_distance", "fare_amount"])

# Filter out trips with zero or negative distance
df_clean = df_clean.filter(col("trip_distance") > 0)

# Filter out trips with zero or negative fare
df_clean = df_clean.filter(col("fare_amount") > 0)

# Preview the cleaned data
display(df_clean)

# --- Cell 2: Trip duration ---
# Calculate trip duration in minutes from pickup and dropoff timestamps
df_transformed = df_clean.withColumn(
    "trip_duration_minutes",
    round(
        (unix_timestamp(col("tpep_dropoff_datetime")) -
         unix_timestamp(col("tpep_pickup_datetime"))) / 60,
        2
    )
)

# --- Cell 3: Cost per mile and time of day ---
# Calculate cost per mile (fare divided by distance)
df_transformed = df_transformed.withColumn(
    "cost_per_mile",
    round(col("fare_amount") / col("trip_distance"), 2)
)

# Categorize trips by time of day based on pickup hour
df_transformed = df_transformed.withColumn(
    "time_of_day",
    when(hour(col("tpep_pickup_datetime")).between(6, 11), "Morning")
    .when(hour(col("tpep_pickup_datetime")).between(12, 17), "Afternoon")
    .when(hour(col("tpep_pickup_datetime")).between(18, 21), "Evening")
    .otherwise("Night")
)

# Preview the transformed data with all new columns
display(df_transformed)

# --- Cell 4: Verify row counts ---
# Count rows before and after cleaning
raw_count = df.count()
cleaned_count = df_transformed.count()
removed_count = raw_count - cleaned_count

print(f"Raw records: {raw_count}")
print(f"Cleaned records: {cleaned_count}")
print(f"Records removed: {removed_count}")

Your data is clean, enriched, and ready to be saved. Next up, you will load this transformed DataFrame into a Delta Lake table so it becomes a permanent, queryable dataset in your workspace.

Load into a Delta Lake Table

In the previous step, you cleaned your raw taxi data and engineered new features like trip duration, cost per mile, and time-of-day categories.

Your transformed DataFrame is sitting in memory right now, but it disappears the moment your notebook session ends. To make the data permanent and queryable, you need to save it as a table. In this step, you will write the DataFrame to a Delta Lake table in Unity Catalog, completing the "Load" phase of your ETL pipeline.

In this step, get ready to:
  • Save your transformed DataFrame as a Delta Lake table.
  • Verify the new table appears in Unity Catalog and inspect its schema and sample data.
Save the DataFrame as a Delta table

Delta Lake is the default storage format for all tables in Databricks. Every table you create automatically gets ACID transactions, schema enforcement, and versioning. You do not need any special configuration.

What is Delta Lake?

Delta Lake is an open-source storage layer that extends Parquet files with a transaction log. This transaction log tracks every change made to a table, giving you reliable writes and the ability to query previous versions of your data (called "time travel"). Every table on Databricks is a Delta table by default.

Unity Catalog organizes data in a three-level namespace: catalog.schema.table. In Databricks Free Edition, your workspace comes with a workspace catalog (named after your workspace) and a default schema that you already have full permissions on.

  • Create a new cell in your notebook and write the transformed DataFrame to a Delta table by pasting the following code:
# Save the transformed DataFrame as a Delta Lake table in Unity Catalog
# "overwrite" mode replaces the table if it already exists
df_transformed.write.mode("overwrite").saveAsTable("default.nyc_taxi_cleaned")

What does this code do?

  • df_transformed.write accesses the DataFrame's write interface, which controls how data gets saved.
  • .mode("overwrite") tells Databricks to replace the table if it already exists. This is useful when re-running your pipeline with updated logic.
  • .saveAsTable("default.nyc_taxi_cleaned") writes the data to a table called nyc_taxi_cleaned inside the default schema of your workspace catalog. Because you omit the catalog name, Databricks uses your workspace's default catalog automatically.
  • Run the cell by pressing Shift+Enter.

The cell takes a few seconds to run. Once it finishes, your transformed data is permanently stored as a Delta Lake table.

Seeing a permission error?

  • Make sure you are using the default schema. Free Edition users have full privileges on the workspace catalog and default schema by default.
  • Check that the table name in your code matches default.nyc_taxi_cleaned exactly. Typos in the schema or table name can cause errors.
  • If your compute was shut down due to the daily quota, wait until the next day and try again.

Why am I getting a permission or catalog error when saving my Delta table?

Verify the table in Unity Catalog

Your data is saved, but you should always confirm the table landed correctly. The Catalog Explorer in Databricks lets you browse all tables, view their schemas, and preview sample data.

  • Click Catalog in the left sidebar.
  • Expand your workspace catalog (it shares the same name as your workspace).
  • Expand the default schema.
  • Confirm that the nyc_taxi_cleaned table appears in the list.
  • Click on the nyc_taxi_cleaned table to open its details.
  • Click the Overview tab to view the table's column names and data types.
  • Click Sample Data to preview 100 rows from the table.

You should see all your original columns plus the new ones you engineered: trip duration, cost per mile, and time-of-day category.

Why verify in the Catalog instead of the notebook?

Checking the Catalog Explorer confirms that your table is registered in Unity Catalog and accessible to anyone in the workspace. A notebook display() call only shows data in your current session. The Catalog is the source of truth for what is permanently stored.

Don't see your table in the Catalog?

  • Make sure you are looking inside the correct catalog. Your workspace catalog shares the same name as your workspace. It is not the samples catalog.
  • Confirm you expanded the default schema, not information_schema.
  • Go back to your notebook and check that the saveAsTable cell ran without errors. If it shows an error, fix the issue and re-run.

Help me find my saved Delta table in the Databricks Catalog Explorer.

Your ETL pipeline is complete. You extracted raw data, transformed it, and loaded it into a production-ready Delta Lake table. Next up, you will query this table with SQL and build visualizations to uncover patterns in the data.

Query and Visualize the Results

Your cleaned and enriched data is now sitting in a Delta Lake table, ready to be queried. The "Load" phase of your ETL pipeline is complete.

Now comes the payoff. You will switch from Python to SQL to query your transformed table and uncover patterns in the taxi trip data. Then you will turn those query results into visual charts directly inside your notebook.

In this step, get ready to:
  • Write SQL queries to analyze pickup zones, fares, and cost efficiency.
  • Create a bar chart visualization from your query results.
  • Review your complete ETL pipeline from start to finish.
Write a query to find the busiest pickup zones

Your notebook's default language is Python, but Databricks lets you run SQL in any cell by adding a %sql magic command as the first line. This makes it easy to query your Delta table without leaving the notebook.

  • Click the + Code button at the bottom of your last cell to add a new cell.
  • Find the top 10 busiest pickup zip codes by pasting the following SQL query into the new cell:
%sql

-- Find the top 10 pickup zip codes with the most trips
SELECT
  pickup_zip,
  COUNT(*) AS trip_count,
  ROUND(AVG(fare_amount), 2) AS avg_fare
FROM nyc_taxi_cleaned
GROUP BY pickup_zip
ORDER BY trip_count DESC
LIMIT 10

What does this query do?

  • %sql on the first line tells Databricks to treat this cell as SQL instead of Python.
  • GROUP BY pickup_zip groups all trips by their starting zip code.
  • COUNT(*) counts the total number of trips from each zip code.
  • AVG(fare_amount) calculates the average fare for trips originating from each zone.
  • ORDER BY trip_count DESC LIMIT 10 sorts by trip count and returns only the top 10 busiest zones.
  • Run the cell by pressing Shift+Enter.

You should see a table with 10 rows showing zip codes, their trip counts, and average fares.

Seeing "Table or view not found"?

  • Make sure you ran the cell in Step 4 that saved the table with saveAsTable(). If that cell was not run, the table does not exist yet.
  • Check that the table name in your query matches exactly what you used in Step 4. The name is case-sensitive.
  • Try running SHOW TABLES in a new SQL cell to see which tables exist in your default schema.
  • Still stuck? Help me fix a "table not found" error in my Databricks SQL query.
Query average fare and distance by time of day

The time_of_day column you engineered in Step 3 categorizes each trip as Morning, Afternoon, Evening, or Night. Now you can use it to compare travel patterns across different parts of the day.

  • Add a new code cell and paste the following query to break down fares and distances by time of day:
%sql

-- Compare average fare, distance, and duration across time-of-day categories
SELECT
  time_of_day,
  COUNT(*) AS trip_count,
  ROUND(AVG(fare_amount), 2) AS avg_fare,
  ROUND(AVG(trip_distance), 2) AS avg_distance,
  ROUND(AVG(trip_duration_minutes), 2) AS avg_duration_min
FROM nyc_taxi_cleaned
GROUP BY time_of_day
ORDER BY trip_count DESC

What does this query do?

  • GROUP BY time_of_day splits the results into your four time categories.
  • The averages for fare, distance, and duration let you compare how trips differ between morning rush hour and late-night rides.
  • Sorting by trip_count puts the busiest time period first.
  • Run the cell by pressing Shift+Enter.

You should see a table with 4 rows, one for each time-of-day category, showing trip counts, average fares, distances, and durations.

Seeing an error about an unknown column?

  • If you see an error mentioning time_of_day or trip_duration_minutes, go back to Step 3 and confirm that your transformation cell added these columns before saving the table.
  • Need help? Help me debug a column not found error in my SQL query.
Create a bar chart visualization

Databricks notebooks let you create charts directly from query results. You do not need any external visualization tools.

  • Scroll up to the results of your time-of-day query (the one you just ran).
  • Above the result table, click the + button.
  • Select Visualization.

The visualization editor opens with a default chart. Now configure it to show average fare by time of day.

  • In the Visualization Type drop-down, verify that Bar is selected.
  • Select time_of_day for the X column.
  • Select avg_fare for the Y column.
  • Click Save.

You should see a bar chart with four bars, one for each time-of-day category, showing how average fares compare across Morning, Afternoon, Evening, and Night.

Why use the built-in visualization editor?

The visualization editor lets you add aggregation directly in the chart without modifying your SQL query. You can swap between SUM, COUNT, AVERAGE, and other aggregations to explore the data from different angles.

Don't see the + button above the results?

  • Make sure the cell has finished running and results are displayed. The + button only appears after a result set is available.
  • If you accidentally closed the results, re-run the cell by clicking inside it and pressing Shift+Enter.
  • Still having trouble? Help me create a visualization from my SQL query results in Databricks.
Review your complete ETL pipeline
  • Scroll to the top of your notebook and read through each cell from top to bottom.

Your notebook now contains a complete ETL pipeline. Each cell represents a stage of the workflow that data engineers follow in production every day.

What are the stages of your pipeline?

  • Extract: You read raw taxi trip data from the samples.nyctaxi.trips table in Unity Catalog.
  • Transform: You cleaned the data by removing invalid records, then engineered new features like trip duration, cost per mile, and time-of-day categories.
  • Load: You saved the enriched DataFrame as a Delta Lake table with ACID transaction support.
  • Analyze: You queried the table with SQL and visualized the results to uncover trip patterns.

This is the exact workflow data engineers use to move raw data into production-ready tables. The only difference in a real job is scale and scheduling, which is exactly what the Secret Mission covers.

Secret mission

Schedule Your Pipeline as a Job

Your ETL pipeline works perfectly when you run it manually. But real data pipelines run on autopilot. In this secret mission, you will turn your notebook into a scheduled Databricks job that runs automatically on a recurring cadence.

Before you go

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs on Databricks Free Edition, so there are no ongoing costs. Your workspace is free to keep.

Resources you used:

  • Databricks Free Edition account and workspace.
  • NYC Taxi ETL Pipeline notebook.
  • Delta Lake table (nyc_taxi_cleaned) in Unity Catalog.
  • Scheduled job (if you completed the Secret Mission).

Keep everything running

No action needed. Choose this if you are still actively building or want to experiment further with your pipeline.

  • Your Databricks Free Edition workspace stays available indefinitely at no cost.
  • Your notebook, Delta table, and scheduled job (if created) will remain in place.
  • Serverless compute shuts down automatically when idle, so there is nothing to turn off.

Pause - I'll come back to this later

Disable your scheduled job to prevent unnecessary runs while keeping all your work intact.

  • If you created a scheduled job in the Secret Mission, click Workflows in the sidebar.
  • Select your job and pause or disable the schedule so it does not run while you are away.
  • Your notebook and Delta table will remain untouched and ready for when you return.
  • Serverless compute is already managed for you. There is nothing else to shut down.

Delete - I don't want to use this again

Remove all project resources from your workspace. Your Databricks account will remain active.

  • If you created a scheduled job, click Workflows in the sidebar, select your job, and delete it.
  • Drop the Delta table by running this SQL command in a notebook cell:
%sql
DROP TABLE IF EXISTS nyc_taxi_cleaned;
  • Delete the notebook by right-clicking it in the workspace sidebar and selecting Delete.
  • If you want to delete your Databricks account entirely, you can do so from your account settings. This is permanent and cannot be undone.

Mission accomplished

That's a Wrap

Nice work! You just built a complete ETL data pipeline on Databricks, taking millions of raw NYC taxi trip records and turning them into clean, queryable, analysis-ready data.

You've learned how to:

  • Build an ETL pipeline that extracts data from Unity Catalog, transforms it with PySpark, and loads the results into a Delta Lake table.
  • Clean and enrich raw data by filtering invalid records and engineering new features like trip duration, cost per mile, and time-of-day categories.
  • Query your pipeline's output with SQL and create notebook visualizations to uncover trip patterns and trends.
  • Schedule your pipeline as an automated Databricks job that runs on a recurring schedule.

Ready to quiz yourself?