Build an Incremental Sensor Pipeline

Ingest IoT sensor data into Apache Iceberg with time travel and schema evolution.

Introduction

30 Second Summary

Every time you save a document, your computer keeps the previous version so you can undo mistakes. Data pipelines processing thousands of records need the same safety net, but most databases don't offer it.

In this project, you will build a sensor telemetry pipeline on Apache Iceberg that snapshots every data load automatically. You'll simulate a real-world data corruption, recover in one command using time travel, and handle a schema change mid-stream without rewriting a single row.

What You'll Build

A working Iceberg table in Databricks that you can query at any point in its history, roll back after a simulated disaster, and expand with new columns while old data stays untouched.

By the end of this project, you'll have:

  • A time-travel capable table where you can query any previous version of your sensor data by snapshot, seeing exactly what it looked like before and after each batch load.
  • A proven disaster recovery workflow where you roll back a corrupted table to a clean state in one command, without losing the ability to inspect what went wrong.
  • A live schema evolution demonstration where you add a new column without rewriting existing data, and old rows gracefully show NULL for the new field.
  • Secret Mission: Dive into Iceberg's metadata layer by querying snapshot tables directly and comparing how Iceberg tracks history versus Delta Lake.

Are there any prerequisites?

You need a Databricks Free Edition account with Iceberg table support confirmed working. If you've already created a test Iceberg table (like CREATE TABLE test_ice (id INT) USING ICEBERG) and it succeeded, you're ready to go.

Before We Start

Before We Start...

Before we dive in, let's lock in what you're about to build and why it matters.

Create the Iceberg Table and Load Initial Data

Every operation on an Apache Iceberg table creates a new snapshot. To demonstrate time travel, recovery, and schema evolution in later steps, you first need a clean starting point that establishes the baseline snapshot.

In this step, you'll create a managed Iceberg table in Unity Catalog and populate it with initial sensor readings. This first INSERT becomes version 1 in the table's history, and all future time travel references back to it.

In this step, get ready to:
  • Create a managed Iceberg table with an explicit schema.
  • Insert an initial batch of 5 sensor readings.
  • Verify the data and inspect the table's version history.
Create the managed Iceberg table

Databricks creates Delta Lake tables by default. The USING ICEBERG keyword explicitly tells Databricks to use the Iceberg table format instead, which gives you snapshot-based versioning from the very first operation.

  • In the first cell of your notebook, create the table structure by running this SQL:
-- Cell 1: Create the managed Iceberg table
CREATE TABLE sensor_readings (
  device_id STRING,
  temperature DOUBLE,
  reading_time TIMESTAMP
) USING ICEBERG;

What does this code do?

This creates a managed table in Unity Catalog with three columns designed for IoT sensor data.

  • device_id identifies which sensor sent the reading.
  • temperature stores the measurement as a decimal number.
  • reading_time records when the measurement was taken.
  • USING ICEBERG switches the format from the default Delta Lake to Apache Iceberg.
  • Run the cell. You should see a confirmation message indicating the table was created successfully.

Seeing an error about Iceberg not being supported?

This means predictive optimization is not enabled for your account. Managed Iceberg tables require it. If your account was created after November 11, 2024, it should be enabled by default.

Help me debug my Iceberg table creation error.

Insert the first batch of sensor data

This initial batch represents 5 temperature readings from two IoT devices over a 10-minute window. It simulates the first data arriving in your pipeline.

  • Add a new cell below and insert the first batch of sensor data by running this SQL:
-- Cell 2: Load initial batch
INSERT INTO sensor_readings VALUES
  ('sensor_01', 22.5, '2026-08-01 08:00:00'),
  ('sensor_01', 23.1, '2026-08-01 08:05:00'),
  ('sensor_02', 19.8, '2026-08-01 08:00:00'),
  ('sensor_02', 20.2, '2026-08-01 08:05:00'),
  ('sensor_01', 22.9, '2026-08-01 08:10:00');

What does this data represent?

You're inserting readings from sensor_01 and sensor_02 at 5-minute intervals. The temperatures hover around 20-23 degrees, representing normal indoor conditions.

This INSERT creates the table's first data snapshot (version 1). Every future INSERT adds another snapshot on top of this one.

  • Run the cell. You should see a confirmation that rows were inserted.

Seeing a syntax error?

Check that each row's values are wrapped in parentheses and separated by commas. The timestamps must be in single quotes with the format YYYY-MM-DD HH:MM:SS.

Help me fix my INSERT statement syntax error.

Verify the table and inspect its first version

Before you run the verification query, predict what you expect to see. How many rows should be in the table? What columns will appear?

  • Add a new cell and query the full table by running:
-- Cell 3: Verify initial data
SELECT * FROM sensor_readings;

You should see 5 rows of temperature data with device_id, temperature, and reading_time columns.

  • Now inspect the table's version history by running this in a new cell:
-- Cell 4: Check table history
DESCRIBE HISTORY sensor_readings;

The history output shows two versions. Version 0 corresponds to the CREATE TABLE operation, and version 1 corresponds to the INSERT (shown as a WRITE operation).

Why does the history matter?

Each row in the history represents a snapshot of your table at that point in time. In the next steps, you'll add more snapshots and then use these version numbers to travel back to any previous state.

The version column gives you an integer you can reference directly with VERSION AS OF queries. The timestamp column gives you a point-in-time reference for the same purpose.

✔️ Awesome, I've got everything!

Great. Confirm you see 5 rows in the SELECT output and 2 versions in DESCRIBE HISTORY before moving on.

ⓧ I'd like to double check the full code

Here is the full code across all four cells in your notebook so far:

-- Cell 1: Create the managed Iceberg table
CREATE TABLE sensor_readings (
  device_id STRING,
  temperature DOUBLE,
  reading_time TIMESTAMP
) USING ICEBERG;

-- Cell 2: Load initial batch
INSERT INTO sensor_readings VALUES
  ('sensor_01', 22.5, '2026-08-01 08:00:00'),
  ('sensor_01', 23.1, '2026-08-01 08:05:00'),
  ('sensor_02', 19.8, '2026-08-01 08:00:00'),
  ('sensor_02', 20.2, '2026-08-01 08:05:00'),
  ('sensor_01', 22.9, '2026-08-01 08:10:00');

-- Cell 3: Verify initial data
SELECT * FROM sensor_readings;

-- Cell 4: Check table history
DESCRIBE HISTORY sensor_readings;

Your Iceberg table is live with its first snapshot recorded. Next up, you'll simulate a real-world pipeline by loading additional batches and watching the snapshot history grow with each one.

Simulate Incremental Batch Arrivals

Your Iceberg table is live with its first batch of sensor data. In a real pipeline, data doesn't arrive once and stop. It flows in continuously, batch after batch.

Each INSERT you run creates a new snapshot in the Iceberg table. These snapshots accumulate over time, forming a complete timeline of your data. When something goes wrong later, you'll need to know exactly which version to roll back to.

In this step, get ready to:
  • Load a second batch of sensor readings from a new time window.
  • Load a third batch that introduces a new device.
  • Inspect the growing version history to see each snapshot.
Load batch 2 (normal readings)

The next 5-minute window of sensor data has arrived. This batch covers the 08:15 and 08:20 readings from your two existing devices.

  • Add a new SQL cell below your existing cells and load the second batch by running this code:
-- Cell 5: Load batch 2
INSERT INTO sensor_readings VALUES
  ('sensor_01', 23.5, '2026-08-01 08:15:00'),
  ('sensor_02', 20.8, '2026-08-01 08:15:00'),
  ('sensor_01', 23.8, '2026-08-01 08:20:00'),
  ('sensor_02', 21.1, '2026-08-01 08:20:00');

What does this code do?

This INSERT adds 4 new rows representing the 08:15 and 08:20 time windows for sensor_01 and sensor_02. Notice the temperatures are climbing slightly, simulating a warming trend through the morning.

Behind the scenes, Iceberg writes these rows as a new set of data files and creates a fresh snapshot pointing to both the old and new files. This is version 2 of your table.

  • Run the cell. You should see confirmation that 4 rows were affected.

Seeing a table not found error?

Make sure you ran the Cell 1 (CREATE TABLE) and Cell 2 (first INSERT) cells from Step 1 in this same notebook session. If your session timed out, re-run those cells first.

Also check that you haven't accidentally changed the table name. It should be exactly sensor_readings.

Help me debug a table not found error in my Databricks notebook

Load batch 3 (more normal readings)

Another time window arrives. This batch is interesting because it introduces a third device, sensor_03, reporting for the first time at 08:25.

  • Add another new SQL cell and load the third batch by running this code:
-- Cell 6: Load batch 3
INSERT INTO sensor_readings VALUES
  ('sensor_01', 24.0, '2026-08-01 08:25:00'),
  ('sensor_02', 21.5, '2026-08-01 08:25:00'),
  ('sensor_03', 18.3, '2026-08-01 08:25:00');

What does this code do?

This INSERT adds 3 rows for the 08:25 window. All three devices now report in the same time slot. This creates version 3 of your table.

Your table now has 12 total rows spread across 3 snapshots of actual data (versions 1, 2, and 3). Each snapshot captures the exact state of the table at that point in time.

  • Run the cell. You should see confirmation that 3 rows were affected.
Inspect the growing history

You've now run three separate INSERT operations. Each one created a distinct Iceberg snapshot. Time to see the full timeline that Iceberg has been building behind the scenes.

Before you run the next query, think about this: how many versions do you expect to see in the history? Remember that the CREATE TABLE operation also counts as a version.

  • Inspect the full version history by adding a new cell and running this code:
-- Cell 7: Inspect history
DESCRIBE HISTORY sensor_readings;

You should see 4 versions (0 through 3) with distinct timestamps. Version 0 is the CREATE TABLE. Versions 1, 2, and 3 are your three INSERT operations, each labeled as a WRITE operation.

Why do version numbers matter?

Each version number is a handle you can use to travel back in time. If version 3 is clean but version 4 turns out to be corrupted, you can restore to version 3 in a single command. This is exactly what you'll do in the next step.

  • Confirm all 12 rows are present by adding one more cell and running:
-- Cell 8: Confirm row count
SELECT COUNT(*) as total_rows FROM sensor_readings;

The result should show 12 total rows. That's your 5 from batch 1, 4 from batch 2, and 3 from batch 3.

Seeing fewer than 12 rows?

If your count is 9, you likely missed batch 2 (Cell 5). If it's 5, you missed both batch 2 and batch 3. Scroll up and run any INSERT cells that show no output.

Help me figure out why my row count is wrong

✔️ Awesome, I've got everything!

Great. Make sure all your cells from Cell 5 through Cell 8 show output in your notebook.

ⓧ I'd like to double check the full code

Your notebook should now have 8 cells total. Scroll through and confirm you have:

  • Cells 1-4 from Step 1 (CREATE TABLE, initial INSERT, SELECT, DESCRIBE HISTORY).
  • Cell 5: INSERT with 4 rows for the 08:15 and 08:20 time windows.
  • Cell 6: INSERT with 3 rows including sensor_03 for the 08:25 window.
  • Cell 7: DESCRIBE HISTORY sensor_readings; showing 4 versions.
  • Cell 8: SELECT COUNT(*) as total_rows FROM sensor_readings; showing 12.

Your table now has a rich history of snapshots. Next up, you'll see why that history matters. A corrupted batch is about to arrive, and you'll use Iceberg's time travel to undo the damage.

Inject Bad Data and Recover with Time Travel

Your pipeline is humming along with 12 rows across three batches, each one neatly tracked by its own Iceberg snapshot. But what happens when something goes wrong?

In a traditional database, recovering from bad data means restoring from a backup. With Iceberg's snapshot architecture, every previous state of your table is still accessible. You'll inject corrupted data, watch it pollute your table, and then restore the clean state in one command using time travel.

In this step, get ready to:
  • Inject a corrupted batch of impossible sensor values into your table.
  • Use time travel to inspect the last clean version without modifying anything.
  • Restore the table to its clean state and verify the recovery in the history.
Inject a corrupted batch

Imagine one of your sensors malfunctions and starts sending impossible temperature values. This is exactly the kind of problem that breaks pipelines in production.

  • Create a new SQL cell in your notebook and simulate the corrupted batch by running:
-- Cell 9: Inject corrupted batch
INSERT INTO sensor_readings VALUES
  ('sensor_01', -999.0, '2026-08-01 08:30:00'),
  ('sensor_ERROR', 9999.0, '2026-08-01 08:30:00'),
  ('sensor_02', -273.2, '2026-08-01 08:30:00');

What does this simulate?

These values represent a sensor malfunction. Temperatures of -999 and 9999 are physically impossible, and sensor_ERROR is a garbled device ID. In real pipelines, this kind of garbage data arrives more often than you'd expect.

Before you check the damage, what do you think a query filtering for extreme temperatures will return?

  • Spot the corruption by running this query in a new cell:
-- Cell 10: Spot the corruption
SELECT * FROM sensor_readings WHERE temperature < -100 OR temperature > 100;

You should see 3 rows of garbage data. Your table is now polluted. In a traditional system, you'd be reaching for a backup. With Iceberg, you have a better option.

Query returns no rows?

Make sure you ran the INSERT cell (Cell 9) first and it completed without errors. Check that you're querying the same table name sensor_readings.

Help me figure out why my corruption query returns no rows.

Use time travel to inspect the clean state

You know from the previous step that version 3 was your last clean state (the third INSERT batch). Iceberg lets you query any previous version without modifying the current table.

  • Query the table at version 3 to confirm it was clean by running this in a new cell:
-- Cell 11: Time travel to inspect clean state
SELECT * FROM sensor_readings VERSION AS OF 3;

What is VERSION AS OF?

The VERSION AS OF clause lets you read the table as it existed at a specific snapshot version. The current table remains unchanged. You're peeking into the past without affecting the present.

You should see 12 rows of clean data with no impossible values. Now let's make the contrast obvious.

  • Compare the current row count against the clean version by running this in a new cell:
-- Cell 12: Compare counts
SELECT 'current' as state, COUNT(*) as rows FROM sensor_readings
UNION ALL
SELECT 'version_3' as state, COUNT(*) as rows FROM sensor_readings VERSION AS OF 3;

You should see 15 rows in the current state and 12 in version 3. The difference of 3 is your corrupted batch.

Seeing an error about time travel not supported?

Iceberg requires that a snapshot has been previously read (with a SELECT) before you can time travel to it. Since you ran DESCRIBE HISTORY and queried the table in the previous step, version 3 should be accessible. If you see an error, try running DESCRIBE HISTORY sensor_readings first, then retry the time travel query.

Help me troubleshoot my Iceberg time travel query.

Restore the table to the clean version

You've confirmed version 3 is clean. Now you can roll back the entire table to that state with a single command. The corrupted batch disappears from the current view, but the history of what happened is preserved.

  • Restore the table to version 3 by running this in a new cell:
-- Cell 13: Restore to clean version
RESTORE TABLE sensor_readings TO VERSION AS OF 3;

What does RESTORE do differently than deleting rows?

RESTORE doesn't delete the bad snapshot. It creates a NEW snapshot (version 5) whose contents match version 3. The corruption at version 4 remains in the history for auditing. You can always go back and inspect what went wrong.

Before you verify, how many rows do you expect to see in the restored table?

  • Confirm the row count is back to 12 by running:
-- Cell 14: Verify recovery
SELECT COUNT(*) as total_rows FROM sensor_readings;

You should see 12. The 3 corrupted rows are gone from the current state.

  • Check the full history to see the RESTORE operation recorded by running:
-- Cell 15: Confirm restore in history
DESCRIBE HISTORY sensor_readings;

You should see versions 0 through 5. Version 4 is the corrupted INSERT. Version 5 is the RESTORE operation. The entire audit trail of what happened is preserved, even though the current data is clean.

Still seeing 15 rows after RESTORE?

Make sure the RESTORE cell ran successfully (check for a confirmation message). If you accidentally ran version 4 instead of 3, you restored to the corrupted state. Run RESTORE TABLE sensor_readings TO VERSION AS OF 3 again with the correct version number.

Help me fix my RESTORE TABLE command.

✔️ Awesome, I've got everything!

Great. Make sure your notebook has cells 9 through 15 from this step, and that your final row count shows 12.

ⓧ I'd like to double check the full code

Here are all the cells you added in this step. Confirm each one matches your notebook:

-- Cell 9: Inject corrupted batch
INSERT INTO sensor_readings VALUES
  ('sensor_01', -999.0, '2026-08-01 08:30:00'),
  ('sensor_ERROR', 9999.0, '2026-08-01 08:30:00'),
  ('sensor_02', -273.2, '2026-08-01 08:30:00');

-- Cell 10: Spot the corruption
SELECT * FROM sensor_readings WHERE temperature < -100 OR temperature > 100;

-- Cell 11: Time travel to inspect clean state
SELECT * FROM sensor_readings VERSION AS OF 3;

-- Cell 12: Compare counts
SELECT 'current' as state, COUNT(*) as rows FROM sensor_readings
UNION ALL
SELECT 'version_3' as state, COUNT(*) as rows FROM sensor_readings VERSION AS OF 3;

-- Cell 13: Restore to clean version
RESTORE TABLE sensor_readings TO VERSION AS OF 3;

-- Cell 14: Verify recovery
SELECT COUNT(*) as total_rows FROM sensor_readings;

-- Cell 15: Confirm restore in history
DESCRIBE HISTORY sensor_readings;

You just recovered from data corruption without losing any audit trail. The bad data is gone from your current table, but the full story of what happened lives in the version history. Next up, you'll handle a different kind of pipeline challenge: an upstream schema change where sensors start reporting a brand new field.

Handle an Upstream Schema Change

Your table is clean again after the time travel recovery. Now your sensor fleet just received a firmware update, and devices are reporting a new field: humidity.

In many data systems, this upstream schema change breaks the pipeline because the existing table can't accept the new column. With Iceberg's schema evolution, you add the column without rewriting existing data.

In this step, get ready to:
  • Add a new humidity column to the Iceberg table using schema evolution.
  • Insert new sensor data that includes humidity values alongside the original fields.
  • Query across the schema boundary to see old and new data coexist gracefully.
Evolve the schema by adding a column

Your sensors now emit a humidity reading with every temperature sample. The table needs a new column to accept this field.

  • Add the humidity column to sensor_readings by running this SQL in a new cell:
-- Add humidity column (schema evolution)
ALTER TABLE sensor_readings ADD COLUMN humidity DOUBLE;

What does this do?

ALTER TABLE ... ADD COLUMN registers a new column in the Iceberg table's schema metadata.

Iceberg does not rewrite any existing Parquet files. Old data files simply don't have the column, so queries return NULL for those rows. This is what makes schema evolution non-destructive.

  • Confirm the schema change took effect by running this in a new cell:
-- Verify new schema
DESCRIBE sensor_readings;

You should see four columns listed: device_id, temperature, reading_time, and the new humidity column with type DOUBLE.

Don't see the humidity column?

Make sure you ran the ALTER TABLE cell before the DESCRIBE cell. Cells must run in order.

Check that you typed humidity DOUBLE with no typos in the column name or data type.

Why isn't my new column showing up after ALTER TABLE?

Insert new data with the expanded schema

The next batch of sensor data includes humidity readings. Each row now has four values instead of three.

  • Insert the new batch with humidity values by running this in a new cell:
-- Insert data with new schema
INSERT INTO sensor_readings VALUES
  ('sensor_01', 24.5, '2026-08-01 08:35:00', 45.2),
  ('sensor_02', 22.0, '2026-08-01 08:35:00', 52.1),
  ('sensor_03', 18.9, '2026-08-01 08:35:00', 61.8);

What's different about this INSERT?

Each row now supplies four values matching the expanded schema: device_id, temperature, reading_time, and humidity.

The 12 rows already in the table were written before humidity existed. They will show NULL for that column. No backfill is needed.

The cell completes successfully, adding 3 new rows to bring the table to 15 total.

Query across the schema boundary

This is the moment of truth. Old rows (inserted before the schema change) and new rows (inserted after) now live in the same table.

Before you run the next query, what do you think the humidity column will show for rows that were inserted before the column existed?

  • Query the full table sorted by time to see both eras of data side by side:
-- Query across schema boundary
SELECT device_id, temperature, humidity, reading_time
FROM sensor_readings
ORDER BY reading_time DESC;

The top 3 rows (at 08:35) show humidity values like 45.2, 52.1, and 61.8. Every older row shows NULL for humidity.

Why NULL instead of an error?

Iceberg tracks schema at the metadata level, not the file level. Old Parquet files don't contain a humidity column, so Iceberg fills in NULL at read time.

No existing data files were rewritten. This is what makes Iceberg schema evolution safe for production pipelines with terabytes of historical data.

  • Run DESCRIBE HISTORY one final time to see the complete story of every operation on this table:
-- Final history
DESCRIBE HISTORY sensor_readings;

You should see the full history from version 0 through the latest version. The recent entries include the ALTER TABLE operation (schema evolution) and the final INSERT with humidity data.

Seeing an error on the INSERT?

If you get a column count mismatch, make sure you have exactly four values per row in Cell 18. The order must be device_id, temperature, reading_time, humidity.

Check that Cell 16 (ALTER TABLE) ran successfully before Cell 18. The column must exist before you insert data into it.

Help me fix a column mismatch error on my Iceberg INSERT.

✔️ Awesome, I've got everything!

Great. You've just proved that Iceberg can evolve a table's schema without touching historical data.

ⓧ I'd like to double check the full code

Confirm these cells are in your notebook for this step, in order:

  • Cell 16: ALTER TABLE sensor_readings ADD COLUMN humidity DOUBLE;
  • Cell 17: DESCRIBE sensor_readings;
  • Cell 18: INSERT INTO sensor_readings VALUES with 3 rows including humidity values (45.2, 52.1, 61.8).
  • Cell 19: SELECT device_id, temperature, humidity, reading_time FROM sensor_readings ORDER BY reading_time DESC;
  • Cell 20: DESCRIBE HISTORY sensor_readings;

Secret mission

Explore Iceberg's Metadata Layer

You've used DESCRIBE HISTORY to see your table's version timeline. But Iceberg tracks far more under the hood. In this mission, you'll query the raw snapshot metadata directly via PySpark and compare how Iceberg and Delta Lake track history differently.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project uses managed tables in Unity Catalog on Databricks Free Edition, so there are no ongoing costs beyond the storage included in your free account.

Resources you used:

  • sensor_readings (Iceberg table in Unity Catalog).
  • sensor_readings_delta (Delta table in Unity Catalog, created in Secret Mission).
  • test_ice (Iceberg test table from your earlier setup confirmation).

Keep everything running

No action needed. Choose this if you plan to keep experimenting with Iceberg features or want to reference your snapshot history later.

  • Your tables persist in Unity Catalog at no extra cost. Storage is included in the Free Edition.
  • Your notebook remains in your workspace and you can rerun cells any time.
  • Snapshot history remains queryable until the default retention period expires (7 days for deleted files, 30 days for logs).

Pause - I'll come back to this later

Free Edition serverless compute shuts down automatically when idle. Your tables and notebook stay in your workspace at no cost.

  • Close your notebook tab. Your work is auto-saved.
  • When you return, open the notebook from your workspace and run cells again. The tables will still be there.

Delete - I don't want to use this again

Remove all project tables from your workspace. This is permanent and cannot be undone.

  • Drop the main Iceberg table by running this command in a new notebook cell:
DROP TABLE sensor_readings;
  • Drop the Delta comparison table (created in the Secret Mission) by running:
DROP TABLE sensor_readings_delta;
  • Drop the test table from your initial Iceberg setup confirmation by running:
DROP TABLE test_ice;
  • Delete the notebook from your workspace by right-clicking it in the sidebar and selecting Move to Trash.

Nice Work!

Nice Work!

You just built a complete incremental sensor pipeline on Apache Iceberg, handling real-world data challenges that production data engineers face every day.

You've learned how to:

  • Simulated an incremental batch ingestion pipeline with multiple loads creating distinct Iceberg snapshots, giving you a complete audit trail of every change.
  • Recovered from data corruption using Iceberg's snapshot-based time travel with RESTORE TABLE ... TO VERSION AS OF, rolling back to a clean state in one command.
  • Evolved the table schema mid-stream by adding a humidity column without rewriting any existing data, proving Iceberg's graceful schema evolution model.
  • Secret Mission: Explored Iceberg's internal metadata layer and compared snapshot tracking between Iceberg and Delta Lake.

Ready to quiz yourself?