Build an E-Commerce Analytics Lakehouse

Build a local Apache Iceberg lakehouse with Trino, Spark, and Polaris.

Introduction

30 Second Summary

Every time you check an order status, browse product recommendations, or get a shipping notification, a data system somewhere is storing, updating, and querying millions of rows in real time.

In this project, you will build a production-style Apache Iceberg lakehouse on your own machine, using Cursor IDE as your AI engineering partner to generate every file, debug every error, and document the entire system through natural language prompts.

What You'll Build

An e-commerce analytics lakehouse with five interconnected services (MinIO, Apache Polaris, Trino, Apache Spark, Jupyter) running locally in Docker, processing realistic order data with full ACID guarantees.

By the end of this project, you'll have:

  • A fully working Iceberg lakehouse with five Docker services that you built entirely through Cursor natural language prompts.
  • A realistic e-commerce data platform with orders, customers, and products tables that supports inserts, updates, deletes, and analytical queries.
  • Automated maintenance jobs (compaction and snapshot expiration) with before-and-after measurements proving they work, plus a Cursor-generated validation report and industry comparison.
  • A time-travel disaster recovery test where you recover "accidentally deleted" orders from a previous Iceberg snapshot.

Are there any prerequisites?

You need basic familiarity with SQL and terminal commands. No prior Iceberg or data engineering experience is needed. You also need Docker Desktop installed on Windows (with 32 GB RAM) and a Cursor IDE Pro+ plan ($60/mo) with Composer 2, Agent Mode, and Parallel Agents.

Set Up Your Development Environment

Before you can build a lakehouse, you need a workspace that can handle five interconnected services running simultaneously.

This step configures Docker Desktop with enough memory to run the full stack, sets up your project folder in Cursor IDE, and creates a .cursorrules file that shapes every AI-generated artifact for the rest of the project.

In This Step, Get Ready To:
  • Configure Docker Desktop with 16 GB of RAM and verify it is running.
  • Create your project workspace in Cursor and enable Agent Mode settings.
  • Generate a .cursorrules file and an architecture plan using Cursor's Plan Mode.
Configure Docker Desktop for the Lakehouse Stack

The full lakehouse stack runs five services inside Docker containers. By default, Docker Desktop does not allocate enough memory to run all of them at once. You need to bump it to 16 GB so that Spark and Trino do not get killed by the operating system mid-query.

  • Open Docker Desktop on Windows.
  • Click the gear icon to open Settings.
  • Select Resources in the left sidebar.
  • Set Memory to 16 GB.
  • Set CPUs to at least 4.
  • Click Apply & restart.

Why 16 GB?

You have 32 GB of total RAM. Giving Docker 16 GB leaves plenty for Windows and Cursor to run comfortably.

The minimum for the Iceberg stack is 8 GB, but 16 GB is recommended when running Spark alongside the other services.

Now verify that Docker is working correctly.

  • Open PowerShell.
  • Run the following commands to confirm Docker is installed and responsive.
docker --version
docker compose version

Both commands should print version numbers.

I see version numbers

Docker is ready. Move on to the next substep.

I see an error

If you see "command not found" or "docker daemon is not running," make sure Docker Desktop is fully started.

  • Check the Docker Desktop icon in your system tray. It should show a green "running" status.
  • If Docker Desktop is not installed, download it from docker.com/products/docker-desktop and install it before continuing.

Still Not Working?

If Docker Desktop is installed but the commands still fail, help me troubleshoot Docker Desktop on Windows.

Create Your Project Workspace in Cursor

Cursor IDE is the primary way you build, debug, and document the entire lakehouse. Before you start prompting, you need a project folder and the right settings enabled so Cursor can run terminal commands and create files autonomously.

  • Open Cursor IDE.
  • Create a new folder called iceberg-lakehouse on your machine.
  • Open it in Cursor via File, then Open Folder.

Now configure Cursor's Agent Mode settings.

  • Open Settings with Ctrl+,.
  • Navigate to Models and set Composer 2 as the default model.
  • Navigate to Features, then Agent.
  • Enable Agent Mode.
  • Enable Auto-run terminal commands.
  • Keep Allow file deletion set to OFF for safety.

What Is Agent Mode?

Agent Mode lets Cursor work autonomously in a loop: it plans what to do, takes action (creates files, runs commands), observes the result, and fixes any errors.

With Auto-run terminal commands enabled, Cursor can execute Docker commands and read their output without you manually approving each one.

What Is Composer 2?

Composer 2 is Cursor's own frontier coding model, released in March 2026. It is optimized for code editing and runs 4x faster than comparable models.

Your Pro+ plan gives you unlimited Composer 2 usage, so you can prompt as many times as you need without worrying about request limits.

Create the .cursorrules File and Architecture Plan

A .cursorrules file sits at the root of your project and tells Cursor the rules of your stack. Without it, Cursor might generate Linux-only commands on your Windows machine, use the wrong port numbers, or reference paths that do not exist. This file prevents all of that by defining your stack constraints upfront.

  • Open Cursor Composer with Ctrl+I.
  • Switch to Agent Mode using the toggle in the top-right of the Composer panel.
  • Paste the following prompt.
Create a .cursorrules file in the project root for an Apache Iceberg lakehouse project. The stack is: Docker Compose on Windows, MinIO for S3-compatible storage, Apache Polaris as the Iceberg REST catalog, Trino 465 for SQL queries, Apache Spark 4.0.1 with Jupyter for data processing. Rules: always use docker compose (not docker-compose), all shell commands must work in PowerShell on Windows, never modify files in the ./data/ directory, use port 8181 for Polaris, 8080 for Trino, 9000/9001 for MinIO, 8888 for Jupyter. File conventions: Docker configs in project root, Trino catalog configs in ./trino/catalog/, Polaris bootstrap scripts in ./polaris/, SQL files in ./sql/, notebooks in ./notebooks/, documentation in ./docs/. When writing SQL, always use the iceberg catalog prefix. When debugging Docker issues, always run docker compose logs with the specific service name. When generating documentation, use plain English at a 7th-grade reading level.

What does this prompt do?

This prompt tells Cursor to create a rules file that defines your entire stack: which services you use, which ports they run on, where files should go, and what OS you are on. Every future Cursor prompt in this project reads the .cursorrules file automatically.

It acts as a guardrail. When Cursor generates a Docker command, it checks this file and uses docker compose instead of the deprecated docker-compose. When it writes a bootstrap script, it uses port 8181 for Polaris instead of guessing.

  • Review the generated .cursorrules file in the diff view.
  • Click Accept to save it.

Now use Plan Mode to create an architecture plan before writing any code.

Plan Mode is different from Agent Mode. Instead of immediately creating files, Cursor reads your codebase, asks clarifying questions, and produces a structured plan for your approval. This plan becomes the single source of truth for the rest of the build.

  • In the Composer input, press Shift+Tab to activate Plan Mode.
  • Paste the following prompt.
I am building an Apache Iceberg lakehouse for e-commerce analytics. The stack is MinIO (object storage), Apache Polaris (REST catalog), Trino (query engine), and Spark (maintenance engine). Plan the architecture: (1) What files and folders need to exist in this project? (2) What is the data flow from MinIO through Polaris to Trino? (3) What are the service dependencies in Docker Compose? (4) What ports does each service need? (5) Save the plan to .cursor/plans/lakehouse-architecture.md.

What does this prompt do?

This prompt asks Cursor to research your project and produce a structured architecture plan. Cursor reads your .cursorrules file, may ask clarifying questions, and outputs a document covering file structure, data flow, service dependencies, and port assignments.

Cursor produces the plan and saves it to .cursor/plans/lakehouse-architecture.md.

  • Review the plan carefully.
  • Verify the port assignments match what you defined in .cursorrules: 8181 for Polaris, 8080 for Trino, 9000/9001 for MinIO, 8888 for Jupyter.
  • Edit anything that looks off, then accept the plan.

What Is Plan Mode?

Plan Mode (activated with Shift+Tab) tells Cursor to research before acting. The agent reads your codebase, creates a structured plan, and waits for your approval before writing any files.

Plans are saved to the .cursor/plans/ directory so you can reference them later. In future steps, every agent prompt can pick up this architecture plan through @Codebase context.

Your development environment is fully configured. Docker has enough memory, Cursor knows your stack constraints, and you have an architecture plan guiding every future prompt. Next up, you will use Cursor's parallel agents to generate all of your infrastructure files at the same time.

Scaffold the Lakehouse Infrastructure with Parallel Agents

Your architecture plan is ready. Now it is time to turn that plan into actual files. A lakehouse stack has many moving parts: a Docker Compose file defining all services, a Trino catalog config pointing at the right catalog, and a bootstrap script that initializes Polaris with the correct credentials. Generating these one at a time would be slow.

Cursor's parallel agents let you launch multiple agent sessions at the same time. Each agent works in its own isolated git worktree, so they do not interfere with each other. In this step, you will launch three agents simultaneously to generate all infrastructure files, then use Cursor to cross-check them for consistency.

In This Step, Get Ready To:
  • Launch three parallel Cursor agents to generate Docker Compose, Trino config, and Polaris bootstrap files simultaneously.
  • Cross-check all generated files for credential and port consistency using a single Cursor prompt.
  • Generate architecture documentation and a project README in parallel.
Launch Parallel Agents for Infrastructure Generation

You are about to generate three files at the same time using three separate Cursor agent sessions. Each agent receives a detailed prompt and works independently. Cursor's Pro+ plan supports up to 8 simultaneous agents, so three is well within the limit.

  • Open the Agents sidebar in Cursor.
  • Click New Agent or use the command palette (Ctrl+Shift+P on Windows) and select New Agent Session.
  • Set each agent to Agent Mode with Composer 2.
  • Create three agent sessions total. You will paste a different prompt into each one.

Paste the following prompt into Agent 1 (Docker Compose):

Create a docker-compose.yml for an Apache Iceberg lakehouse with these services:

(1) minio using quay.io/minio/minio:latest with ports 9000 and 9001, root user admin password password123, server command with console address 9001, healthcheck using curl on the minio health endpoint, data volume at ./data/minio.

(2) setup-bucket using quay.io/minio/mc:latest that depends on minio being healthy, creates a bucket called warehouse using mc commands.

(3) polaris using apache/polaris:latest with ports 8181 and 8182, environment variables for AWS credentials pointing to MinIO (admin/password123), AWS_REGION us-east-1, AWS_ENDPOINT_URL_S3 http://minio:9000, POLARIS_BOOTSTRAP_CREDENTIALS set to default-realm,root,s3cr3t, healthcheck on port 8182 path /q/health, depends on minio healthy.

(4) polaris-setup using alpine/curl that depends on polaris healthy, runs a bootstrap script from ./polaris/bootstrap-catalog.sh, mounts ./polaris as /setup.

(5) trino using trinodb/trino:465 with port 8080, mounts ./trino/catalog to /etc/trino/catalog, depends on polaris-setup completed.

(6) jupyter using quay.io/jupyter/pyspark-notebook:spark-4.0.1 with port 8888, no password, Spark configured for Iceberg with Polaris catalog.

All services on a shared network called lakehouse-net. Use path-style S3 access everywhere.

Why Path-Style S3 Access?

S3 has two URL styles. Virtual-hosted style puts the bucket name in the hostname (like warehouse.minio:9000). Path style puts it in the URL path (like minio:9000/warehouse). Docker's internal DNS cannot resolve virtual-hosted bucket names, so path-style access is required for all services in this stack.

Paste the following prompt into Agent 2 (Trino Config):

Create the file trino/catalog/iceberg.properties for Trino to connect to the Polaris Iceberg catalog. Use connector.name=iceberg, iceberg.catalog.type=rest, rest catalog URI pointing to http://polaris:8181/api/catalog, warehouse s3://warehouse, enable native S3 filesystem, S3 endpoint http://minio:9000, path-style access true, access key admin, secret key password123, region us-east-1.

Paste the following prompt into Agent 3 (Polaris Bootstrap):

Create a bash script at polaris/bootstrap-catalog.sh that bootstraps a Polaris Iceberg catalog. The script should:

(1) Get an OAuth2 access token from http://polaris:8181/api/catalog/v1/oauth/tokens using client credentials grant with client_id root and client_secret s3cr3t.

(2) Create a catalog called lakehouse with storage type S3, default base location s3://warehouse/lakehouse, using the Polaris REST management API at http://polaris:8181/api/management/v1/catalogs.

(3) Create a catalog role called admin_role with full privileges.

(4) Grant the admin_role to the root principal.

Include error handling, echo progress messages, and make it executable. Use curl and jq.

What Is the Polaris Bootstrap Process?

Polaris uses OAuth2 for authentication. Before any service can use the catalog, a bootstrap sequence must run: get a token, create the catalog, create roles, and assign those roles. The POLARIS_BOOTSTRAP_CREDENTIALS environment variable in Docker Compose provides the initial credentials in the format REALM,CLIENT_ID,CLIENT_SECRET.

Pro tip: watch the agents work

All three agents run simultaneously. You can switch between agent tabs to watch each one generate its file. When each agent finishes, review its output in the diff view and click Accept. If any agent produces an error or conflict, Cursor flags it in the change list.

Agent session not starting?

Make sure Agent Mode is toggled on in each session (not Plan Mode or Normal Mode). Also confirm you have Composer 2 selected as the model. If sessions still fail to start, help me troubleshoot Cursor parallel agent sessions.

Cross-Check Generated Files with Cursor

Three agents generated three files independently. Credentials, ports, and service hostnames must all match across these files. A mismatch in even one value (like a port number or password) will cause silent failures when you start the stack. Instead of checking manually, let Cursor do the code review.

  • Open a new Cursor Composer session in Agent Mode.
  • Paste the following cross-check prompt:
Review @docker-compose.yml, @trino/catalog/iceberg.properties, and @polaris/bootstrap-catalog.sh. Check that all credentials match (admin/password123 for MinIO, root/s3cr3t for Polaris). Check that all ports are consistent. Check that service names in Docker Compose match hostnames used in config files. List any inconsistencies.

What Does This Prompt Do?

The @ syntax tells Cursor to read specific files from your project. Cursor compares the credentials, ports, and hostnames across all three files and reports any mismatches. This is the same job a human code reviewer would do, but faster.

  • Review the consistency report Cursor produces.
  • If Cursor finds mismatches, accept its proposed fixes.
  • If Cursor reports no inconsistencies, your infrastructure files are ready.

What Should I See?

After the cross-check, your project folder should contain these files:

  • docker-compose.yml in the project root with 6 service definitions.
  • trino/catalog/iceberg.properties with Trino's connection to Polaris.
  • polaris/bootstrap-catalog.sh with the OAuth2 bootstrap sequence.
Generate a README and Architecture Doc in Parallel

The infrastructure files are fresh in Cursor's context. This is the best time to generate documentation, because Cursor can reference the actual file contents for accuracy. You will launch two more parallel agents to create both docs simultaneously.

  • Open two new agent sessions in the Agents sidebar, both set to Agent Mode with Composer 2.

Paste the following prompt into Agent A (Architecture Doc):

Create docs/architecture.md that explains this lakehouse architecture at a 7th-grade reading level. Include: what each service does (MinIO, Polaris, Trino, Spark), how data flows from storage through catalog to query engine, a text-based architecture diagram showing the five services and their connections, and why this architecture matters for real companies. Reference @docker-compose.yml for accuracy.

Paste the following prompt into Agent B (Project README):

Create a README.md for this project. Include: project title, one-paragraph description, architecture overview, prerequisites (Docker Desktop 16GB RAM, Cursor IDE), quick start commands (docker compose up -d), service URLs and ports, how to run queries in Trino, how to run maintenance in Jupyter, and teardown instructions. Keep it concise and professional. Reference @docker-compose.yml for port mappings.
  • Wait for both agents to finish.
  • Review each document in the diff view.
  • Accept both when you are satisfied with the content.

Why Generate Docs Now?

Documentation generated alongside infrastructure is more accurate than documentation written after the fact. Cursor's @file references pull real port numbers, service names, and credentials directly from your config files. You will update these docs later as the project evolves.

You just generated five project files using parallel agents in a fraction of the time it would take to write them manually. Your infrastructure is defined, your documentation is started, and everything has been cross-checked for consistency. Next up, you will bring this entire stack online and verify that every service is healthy.

Launch, Verify, and Debug the Lakehouse Stack

Your infrastructure files are ready. Now it is time to bring the entire lakehouse stack online and confirm every service is healthy.

Docker Compose reads your configuration and starts all six services in the correct dependency order. If anything fails, you will use Cursor as your debugging partner. Cursor reads Docker logs, identifies the root cause, and applies fixes. This is faster than manually reading log output and searching for solutions.

In This Step, Get Ready To:
  • Start the full lakehouse stack with Docker Compose and pull all required images.
  • Run automated health checks against every service using Cursor Agent Mode.
  • Troubleshoot and resolve any container failures with Cursor's debugging workflow.
Start the Stack

Docker Compose reads your docker-compose.yml and starts all six services in the correct dependency order. The first run pulls every container image from its registry, which takes 3-8 minutes depending on your internet speed.

  • Open Cursor Composer (Ctrl+I) and make sure Agent Mode is toggled on.
  • Paste this prompt into Cursor Agent Mode:
Run docker compose up -d in the project root, wait 10 seconds, then run docker compose ps to show container status. If any container shows unhealthy or exited with a non-zero code, run docker compose logs for that specific container and explain the error.

What does this prompt do?

This prompt tells Cursor to start all containers in detached mode (-d), wait for them to initialize, and then check their status. If any container has a problem, Cursor automatically pulls that container's logs and explains the issue. You do not need to read raw log output yourself.

Cursor runs the commands in your terminal and displays the output. You should see all six services listed in the docker compose ps output.

What about the containers that show "exited"?

Two containers are designed to exit after completing their one-time setup tasks. The setup-bucket container creates the MinIO bucket, and the polaris-setup container bootstraps the Polaris catalog. Both should show "exited (0)", meaning they completed successfully.

If either shows "exited (1)" or any non-zero code, that means the setup failed. Cursor will catch this and explain the error.

Run Automated Health Checks via Cursor

Containers running does not always mean services are healthy. Each service has its own health endpoint that confirms it is actually ready to accept requests. Cursor can hit every endpoint in sequence and report the results.

  • Paste this debugging and verification prompt into Cursor Agent Mode:
Run these health checks in sequence and report results:
(1) curl http://localhost:8182/q/health to check Polaris.
(2) Open a Trino CLI session with docker exec -it trino trino --catalog iceberg and run SHOW CATALOGS then exit.
(3) curl http://localhost:9000/minio/health/live to check MinIO.
(4) curl http://localhost:8888/api/status to check Jupyter.
For any check that fails, read the docker compose logs for that service and explain what went wrong and how to fix it.

What does each health check verify?

  • Polaris uses a Quarkus health endpoint on port 8182. A healthy response confirms the REST catalog is ready to serve Iceberg metadata.
  • Trino is verified by opening its CLI and running SHOW CATALOGS. If the iceberg catalog appears, Trino has successfully connected to Polaris.
  • MinIO exposes a liveness endpoint on port 9000. This confirms the object storage layer is accepting requests.
  • Jupyter has a status API on port 8888. This confirms the notebook server is running and you can access it for Spark maintenance later.

Cursor runs each check, reads the output, and reports the results. If any service is unhealthy, it diagnoses the issue and proposes a fix.

All checks passed

All four health checks returned healthy responses. Your lakehouse stack is fully operational. Move on to the next substep.

One or more checks failed

Cursor should have already identified the failing service and explained the error. Common issues include:

  • Port conflicts: Another application is using port 8080, 8181, or 9000. Close the conflicting app or change the port mapping in docker-compose.yml.
  • Insufficient memory: A container was OOM-killed because Docker Desktop does not have enough RAM allocated. Go back to Docker Desktop Settings and confirm Memory is set to 16 GB.
  • S3 path-style misconfiguration: If Trino cannot connect to MinIO, confirm that s3.path-style-access=true is set in trino/catalog/iceberg.properties. Without this setting, Trino tries to resolve bucket names as hostnames, which fails inside Docker networking.

Follow Cursor's proposed fix and proceed to the troubleshooting substep below.

Troubleshoot Any Failures

If Cursor identified a problem in the previous substep, it will have already proposed a fix. This substep applies that fix and re-verifies the stack.

  • Accept any changes Cursor proposed in the previous substep.
  • Paste this prompt into Cursor Agent Mode to rebuild and re-verify:
Run docker compose down, then docker compose up -d, then docker compose ps to verify all services are healthy.

What does this command sequence do?

The docker compose down command stops and removes all containers. Then docker compose up -d rebuilds them from scratch with your updated configuration. Finally, docker compose ps confirms every service is running. This clean restart ensures your fixes take effect.

If everything passed in the health check substep and you had no failures, you can skip the rebuild. The stack is already running.

Pro tip: the universal debugging pattern

If you hit an unexpected error at any point later in the project, the pattern is always the same. Paste the error message into Cursor Agent Mode and add "explain this error and fix it." Cursor reads your .cursorrules, understands the stack, and applies targeted fixes. You never need to manually search for solutions.

Still seeing unhealthy containers after rebuilding?

If the stack still will not start after applying Cursor's fix, try running docker compose down -v to remove all volumes, then docker compose up -d for a completely fresh start. The -v flag removes persistent data volumes that might contain corrupted state from a failed first run.

If the problem persists, help me debug my Docker Compose stack that won't start.

Your five-service lakehouse stack is running. MinIO is storing files, Polaris is managing the catalog, Trino is ready for queries, and Jupyter is standing by for Spark maintenance. Next up, you will design and create your e-commerce data model with properly partitioned Iceberg tables.

Design and Create E-Commerce Iceberg Tables

Your lakehouse infrastructure is running. Five services are healthy and talking to each other. Now you need something for them to actually work with: tables.

Every e-commerce business tracks the same core entities: orders, customers, and products. In this step, you will design a schema that models these entities as Iceberg tables, with partitioning and compression settings that mirror what production data teams use. You will use Cursor's Plan Mode for the design phase, then Agent Mode to generate and execute the SQL.

In This Step, Get Ready To:
  • Design an e-commerce data model using Cursor Plan Mode for architecture decisions.
  • Generate and execute Iceberg table DDL with partitioning and compression settings.
  • Verify that all three tables exist in Trino and that metadata files appear in MinIO.
Design the Schema with Cursor Plan Mode

Before writing any SQL, you want Cursor to think through the design decisions. Which columns does each table need? Which table should be partitioned, and by what? What Iceberg-specific properties should you set? Plan Mode lets Cursor research your codebase and produce a reasoned plan before generating code.

  • Open Cursor Composer and press Shift+Tab to activate Plan Mode.
  • Paste the following system design prompt:
I need to design an e-commerce data model for an Iceberg lakehouse. The business has orders, customers, and products. Orders have statuses (pending, shipped, delivered, returned) and change frequently. Customers update their addresses occasionally. Products can be discontinued. Plan: (1) What columns does each table need? (2) Which table should be partitioned, and by what column? (3) What Iceberg table properties should I set for compression and metadata cleanup? (4) How should the tables relate to each other? Explain your reasoning for each decision.

Cursor produces a structured plan with rationale for each decision. Review it carefully, especially the partitioning recommendation. Orders should be partitioned by month of order_date because most analytical queries filter by time range. Monthly partitions group related data together so the query engine can skip irrelevant files entirely.

What Is Partitioning?

Partitioning splits a table's data into separate groups based on a column value. When you partition orders by month, all January orders go into one group, February orders into another, and so on. When a query asks for "revenue in March," the engine reads only the March partition instead of scanning every file in the table.

Iceberg handles partitioning differently from traditional systems. You define a partition transform (like month(order_date)) and Iceberg manages the rest automatically. You never need to manually create or manage partitions.

  • Edit anything in the plan that looks off. When satisfied, tell Cursor to proceed.
Generate and Execute Table DDL

With the design plan approved, you can now have Cursor generate the actual SQL and run it against Trino. The DDL includes Iceberg-specific table properties for compression and automatic metadata cleanup.

  • Switch to Agent Mode in the Composer panel.
  • Paste the following prompt:
Based on the plan, create a SQL file at sql/create_tables.sql with Iceberg table DDL for Trino. Create schema iceberg.ecommerce. Then create three tables: (1) orders with columns order_id BIGINT, customer_id BIGINT, product_id BIGINT, quantity INT, unit_price DECIMAL(10,2), total_amount DECIMAL(10,2), order_status VARCHAR, order_date DATE, updated_at TIMESTAMP(6), partitioned by month(order_date). (2) customers with customer_id BIGINT, name VARCHAR, email VARCHAR, city VARCHAR, state VARCHAR, created_at TIMESTAMP(6), updated_at TIMESTAMP(6). (3) products with product_id BIGINT, product_name VARCHAR, category VARCHAR, price DECIMAL(10,2), is_active BOOLEAN, updated_at TIMESTAMP(6). Set table properties: write.parquet.compression-codec = zstd, write.metadata.delete-after-commit.enabled = true. Then connect to Trino via docker exec -it trino trino --catalog iceberg and execute the SQL.

What Do These Table Properties Do?

The two table properties you are setting control how Iceberg manages data behind the scenes.

  • write.parquet.compression-codec = zstd tells Iceberg to compress data files using ZSTD. This reduces storage size significantly while keeping read performance fast.
  • write.metadata.delete-after-commit.enabled = true tells Iceberg to automatically clean up old metadata files after each commit. Without this, metadata files accumulate over time and can eventually exceed the size of your actual data.

Cursor generates the SQL file and executes it against Trino. Watch the terminal output for three CREATE TABLE confirmations. If any statement fails, Cursor reads the error and retries automatically.

The generated SQL file should look similar to this:

-- Create the e-commerce schema
CREATE SCHEMA IF NOT EXISTS iceberg.ecommerce;

-- Orders table: partitioned by month for efficient time-range queries
CREATE TABLE iceberg.ecommerce.orders (
    order_id BIGINT,
    customer_id BIGINT,
    product_id BIGINT,
    quantity INT,
    unit_price DECIMAL(10,2),
    total_amount DECIMAL(10,2),
    order_status VARCHAR,
    order_date DATE,
    updated_at TIMESTAMP(6)
)
WITH (
    partitioning = ARRAY['month(order_date)'],
    format = 'PARQUET',
    "write.parquet.compression-codec" = 'zstd',
    "write.metadata.delete-after-commit.enabled" = 'true'
);

-- Customers table: no partitioning needed for small reference data
CREATE TABLE iceberg.ecommerce.customers (
    customer_id BIGINT,
    name VARCHAR,
    email VARCHAR,
    city VARCHAR,
    state VARCHAR,
    created_at TIMESTAMP(6),
    updated_at TIMESTAMP(6)
)
WITH (
    format = 'PARQUET',
    "write.parquet.compression-codec" = 'zstd',
    "write.metadata.delete-after-commit.enabled" = 'true'
);

-- Products table: small catalog, no partitioning needed
CREATE TABLE iceberg.ecommerce.products (
    product_id BIGINT,
    product_name VARCHAR,
    category VARCHAR,
    price DECIMAL(10,2),
    is_active BOOLEAN,
    updated_at TIMESTAMP(6)
)
WITH (
    format = 'PARQUET',
    "write.parquet.compression-codec" = 'zstd',
    "write.metadata.delete-after-commit.enabled" = 'true'
);

Why Only Partition the Orders Table?

Partitioning adds overhead. Each partition creates separate data files and metadata entries. For small reference tables like customers (10 rows) and products (8 rows), partitioning would create more overhead than benefit. The orders table is the one that grows over time and gets queried by date range, making it the right candidate.

Verify Tables and Metadata in MinIO

Creating tables is only half the job. You need to confirm that Trino sees the tables and that MinIO actually contains the Iceberg metadata files. This verifies the full chain: Trino talks to Polaris (the catalog), which records that the tables exist. MinIO (the storage) holds the physical metadata files that describe each table's schema and properties.

  • Paste the following prompt into Cursor Agent Mode:
Connect to Trino and run: SHOW TABLES FROM iceberg.ecommerce, then DESCRIBE iceberg.ecommerce.orders, then check MinIO at http://localhost:9001 to confirm metadata files exist in the warehouse bucket under lakehouse/ecommerce/. Report what you find.

What Should I See?

The SHOW TABLES command should list three tables: orders, customers, and products. The DESCRIBE command should show all nine columns of the orders table with their data types.

In the MinIO console at http://localhost:9001, navigate to the warehouse bucket. You should see a folder structure like lakehouse/ecommerce/orders/metadata/ containing .json metadata files. These files are Iceberg's table metadata. They track the schema, partitioning, and properties you just defined. No data files exist yet because the tables are empty.

  • You can also verify manually by opening a browser and navigating to http://localhost:9001.
  • Log in with username admin and password password123.
  • Click on the warehouse bucket and browse into lakehouse/ecommerce/ to see the metadata folders for each table.

Tables Not Showing Up?

If SHOW TABLES returns an empty list or an error, the most common cause is a mismatch between the catalog name in your Trino config and the catalog created in Polaris. Verify that iceberg.properties points to the correct Polaris catalog URI. Ask Cursor to help debug this.

Your data model is in place. Three Iceberg tables are ready to receive data, with partitioning on the orders table for efficient time-range queries and ZSTD compression to keep file sizes small. Next up, you will load realistic e-commerce data and run analytical queries to see the lakehouse in action.

Load Data and Run Analytical Queries

Your tables are created, but an empty database is not very useful. A real e-commerce company needs realistic data to test whether queries return correct results and to establish performance baselines before making changes.

In this step, you will use parallel agents to generate sample data and analytical queries simultaneously. Then you will execute them through Trino and capture baseline performance metrics that you will compare against after compaction later.

In This Step, Get Ready To:
  • Generate sample data and analytical queries using parallel Cursor agents.
  • Load 68 rows of e-commerce data across three Iceberg tables and run five analytical queries with execution times.
  • Inspect Iceberg metadata to understand snapshots and data files.
Generate Data and Queries in Parallel

You need two SQL files: one with sample data inserts and one with analytical queries. These are independent tasks, so you can use two parallel Cursor agents to generate them at the same time.

  • Open the Agents sidebar in Cursor and start two new agent sessions, both set to Agent Mode with Composer 2.
  • In Agent 1 (Sample Data), paste the following prompt:
Create a SQL file at sql/load_data.sql that inserts sample data into the e-commerce Iceberg tables via Trino. Insert 10 customers with realistic names, emails, cities, and states. Insert 8 products across categories like Electronics, Clothing, Home, and Books with prices between 9.99 and 599.99. Insert 50 orders spread across the last 3 months with varying statuses (60% delivered, 20% shipped, 10% pending, 10% returned), random customer and product references, and realistic quantities and amounts. Use Trino SQL syntax.
  • In Agent 2 (Analytics Queries), paste the following prompt:
Create a SQL file at sql/analytics_queries.sql with five analytical queries for the e-commerce lakehouse using Trino. Include: (1) Revenue by month with order count. (2) Top 5 customers by total spend with join to customers table. (3) Product performance showing times ordered per product. (4) Order status distribution as percentages. (5) Average order value by product category. Add a comment above each query explaining what business question it answers.
  • Both agents run simultaneously. Review each agent's output in the diff view and accept when done.

Why Parallel Agents for This?

The data file and the queries file have no dependencies on each other. Each agent works in its own isolated git worktree, so they cannot interfere with each other's output. This cuts your generation time roughly in half compared to prompting one file at a time.

Execute Data Load and Run Queries

With both SQL files generated, you need to load the data into your Iceberg tables and then run each analytical query. Cursor can execute the SQL through Trino and capture the results and execution times into a baseline document.

  • In Cursor Agent Mode, paste the following prompt:
Connect to Trino via docker exec and execute the SQL file at @sql/load_data.sql. Then run each query from @sql/analytics_queries.sql one at a time, capture the results and execution times, and save them to docs/query-baseline.md as a markdown table.

What Does This Prompt Do?

Cursor connects to the Trino container using docker exec -it trino trino --catalog iceberg and runs each INSERT statement from your data file. Then it runs the five analytical queries one by one, recording how long each takes. All results go into docs/query-baseline.md so you have a performance baseline to compare against after compaction in Step 7.

Once Cursor finishes, open docs/query-baseline.md and review the results. You should see data for all five queries: revenue by month, top 5 customers, product performance, order status distribution, and average order value by category.

What Should I See?

The revenue by month query should return three rows (one per month). The top 5 customers query should join the orders and customers tables and rank by total spend. Order status distribution should show roughly 60% delivered, 20% shipped, 10% pending, and 10% returned, matching the distribution you specified in the data generation prompt.

Queries Returning Empty Results?

If any query returns zero rows, the data load may have failed silently. Check that the INSERT statements in sql/load_data.sql use the correct schema prefix iceberg.ecommerce and that column names match the DDL from Step 4. Ask Cursor to help diagnose the issue.

Inspect Iceberg Metadata with Cursor

Every time you insert data into an Iceberg table, Iceberg creates a snapshot. A snapshot is a record of the table's state at a specific point in time. Iceberg also tracks every individual data file (Parquet files stored in MinIO). Understanding these metadata tables is essential because you will use them to measure the impact of CDC operations and compaction in later steps.

  • Paste the following prompt into Cursor Agent Mode:
Connect to Trino and run these Iceberg metadata queries: (1) SELECT * FROM iceberg.ecommerce.orders$snapshots ORDER BY committed_at to show all snapshots. (2) SELECT COUNT(*) AS file_count, AVG(file_size_in_bytes) AS avg_size, SUM(file_size_in_bytes) AS total_size FROM iceberg.ecommerce.orders$files to show file statistics. Save both results to docs/metadata-baseline.md. Explain in plain English what snapshots and data files are.

What Are Iceberg Metadata Tables?

Iceberg exposes hidden metadata tables that you can query just like regular tables. The orders$snapshots table shows every version of your table, including when it was created and what operation caused it (like an append or overwrite). The orders$files table shows every Parquet data file, its size, and how many records it contains.

These metadata tables are what make Iceberg powerful. They give you full visibility into how your data is physically stored, which is critical for diagnosing performance problems and measuring the impact of maintenance operations.

Review docs/metadata-baseline.md after Cursor saves it. You should see at least one snapshot (from the data load) and file statistics showing the current number of Parquet files, their average size, and total storage used. These numbers are your baseline. In Step 6, CDC operations will create many more snapshots and small files. In Step 7, compaction will consolidate those files and you will compare the results back to this baseline.

Your lakehouse now has real data flowing through it and baseline metrics captured. Next up, you will simulate real-world data changes like order status updates, customer address changes, and product discontinuations to see how Iceberg handles change data capture.

Simulate Real-World Data Changes

Your lakehouse has clean data and baseline metrics captured. In a real e-commerce system, data never stays clean for long. Orders ship, customers move, products get discontinued, and mistakes get deleted. These ongoing changes are called change data capture (CDC), and they are the reason lakehouse maintenance exists. In this step, you will simulate a day of real production activity and watch how it affects your Iceberg metadata.

In This Step, Get Ready To:
  • Generate and execute six CDC operations (inserts, updates, and deletes) against your Iceberg tables.
  • Analyze how CDC operations impact snapshot count and file sizes using Cursor.
  • Verify data integrity after all changes are applied.
Generate and Execute CDC Operations

In production, data changes constantly. Order statuses flip from "pending" to "shipped." Customers update their addresses. Products get pulled from the catalog. Each of these changes creates a new snapshot in Iceberg, which is how Iceberg tracks every version of your data over time.

You will use Cursor to generate a SQL file with six separate CDC operations, then execute them one at a time against Trino.

  • Open Cursor Composer and switch to Agent Mode.
  • Paste the following prompt:
Create a SQL file at sql/simulate_cdc.sql that simulates real-world changes to the e-commerce tables via Trino. Include: (1) UPDATE 10 orders from 'pending' to 'shipped' with updated_at = current_timestamp. (2) UPDATE 5 orders from 'shipped' to 'delivered'. (3) INSERT 20 new orders for today with varying products and customers. (4) UPDATE 3 customers with new cities. (5) UPDATE 2 products to set is_active = false. (6) DELETE 3 orders with status 'returned' older than 2 months. Each operation must be a separate SQL statement. Then connect to Trino and execute them one at a time.

What Does This Prompt Do?

This prompt tells Cursor to generate six distinct SQL statements that mimic a typical day of e-commerce activity. Each statement targets a different type of change:

  • Two UPDATEs change order statuses (the most frequent real-world operation).
  • One INSERT adds brand-new orders.
  • Two UPDATEs modify customer and product records.
  • One DELETE removes returned orders (this is important for the Secret Mission later).

Cursor connects to Trino and runs each statement individually. Each statement creates a new Iceberg snapshot, so you will accumulate 6 or more new snapshots.

  • Review the generated sql/simulate_cdc.sql file and accept the changes.
  • Watch the Cursor terminal output for confirmation of each executed statement.

Each successful statement prints a row count showing how many rows were affected. If any statement fails, Cursor reads the error and retries automatically.

Seeing an Error During CDC Execution?

The most common issue is a reference to a customer or product ID that does not exist in the data. If Cursor encounters this, it adjusts the SQL to use valid IDs from your loaded data. If the error persists, help me debug this CDC execution error in Trino.

Analyze CDC Impact with Cursor

Every CDC operation created new data files and new snapshots. This is exactly what happens in production systems. Over time, these small files accumulate and slow down queries. This is called the small file problem, and it is the primary reason compaction exists.

You will now use Cursor to compare the current state of your metadata against the baseline you captured in Step 5.

  • Paste this analysis prompt into Cursor Agent Mode:
Connect to Trino and analyze the impact of the CDC operations: (1) Query iceberg.ecommerce.orders$snapshots to show all snapshots with their operation types. (2) Query the files metadata table to show current file count and average size. (3) Compare these numbers to the baseline in @docs/metadata-baseline.md. (4) Write an analysis to docs/cdc-impact-analysis.md explaining: how many snapshots were created, how file count changed, why this creates a 'small file problem,' and why compaction is needed. Use plain English.

What Is Cursor Doing Here?

This is Cursor acting as a data analyst. It queries the Iceberg metadata tables (orders$snapshots and orders$files), pulls the baseline numbers from your earlier document using the @docs/metadata-baseline.md reference, compares them, and writes a plain-English analysis document. This is the optimization phase of AI-assisted engineering.

  • Review the generated docs/cdc-impact-analysis.md file.

You should see that the snapshot count has grown by at least 6 (one per CDC operation). The file count should be noticeably higher than your baseline, and the average file size should be smaller. This is the small file problem in action.

Why Does Each Operation Create a New Snapshot?

Iceberg provides ACID transactions. Every INSERT, UPDATE, or DELETE is atomic. Iceberg guarantees this by writing new data files and creating a new snapshot that points to the updated set of files. The old snapshot still exists, pointing to the old files. This is what makes time-travel possible.

Verify Data Integrity

After running six different mutation operations, you need to confirm the data still makes sense. Order statuses should reflect the updates, deleted rows should be gone, and new orders should appear.

  • Paste this verification prompt into Cursor Agent Mode:
Connect to Trino and run: SELECT order_status, COUNT(*) FROM iceberg.ecommerce.orders GROUP BY 1. Verify the numbers make sense: more shipped and delivered orders than before, fewer pending, and returned orders that were deleted should be gone. If anything looks wrong, explain what happened.

What Should I See?

The query results should show a distribution that reflects your changes. The "shipped" count should be higher than your original baseline (because you moved 10 orders from pending to shipped). The "delivered" count should also be higher. The "pending" count should be lower. The "returned" count should be reduced by 3 (the rows you deleted).

The total row count should be your original 50 orders, plus 20 new inserts, minus 3 deletes, for a total of 67 orders.

Your data is now in a realistic state. You have a mix of original and new orders, updated statuses, modified customer records, discontinued products, and deleted returns. This is exactly what a production Iceberg table looks like after a day of activity. Next up, you will run compaction and expiration to clean up the small files and excess snapshots that accumulated during these changes.

Run Compaction, Expiration, and Generate Final Reports

Your CDC operations created a realistic mess. Multiple small files scattered across storage, a growing pile of snapshots, and metadata that keeps expanding. This is exactly what happens in production lakehouses when data changes frequently without maintenance. In this step, you will measure the damage, run Spark maintenance jobs to fix it, and generate documentation proving your lakehouse is production-ready.

In This Step, Get Ready To:
  • Capture pre-compaction metrics as a baseline for comparison.
  • Run compaction, snapshot expiration, and orphan file cleanup using Spark maintenance procedures.
  • Generate a validation report and industry comparison document using parallel Cursor agents.
Record Pre-Compaction Metrics

Before running any maintenance, you need a snapshot of the current state. These numbers become your "before" baseline so you can prove compaction actually improved things.

  • Paste this prompt into Cursor Agent Mode:
Connect to Trino and capture these pre-compaction metrics: (1) File count, average file size, total size from iceberg.ecommerce.orders$files. (2) Snapshot count from iceberg.ecommerce.orders$snapshots. (3) Run the revenue-by-month query from @sql/analytics_queries.sql and note execution time. Save all results to docs/pre-compaction-metrics.md.

Why Capture Metrics Before Compaction?

Without a "before" measurement, you cannot prove compaction did anything useful. Production teams track file count, average file size, and snapshot count as key health indicators for their Iceberg tables. The numbers Cursor captures here become the evidence in your validation report.

Cursor connects to Trino, runs the metadata queries, captures the revenue-by-month query timing, and saves everything to docs/pre-compaction-metrics.md. You should see a file count higher than your original baseline and multiple snapshots accumulated from the CDC operations in Step 6.

Generate and Run Maintenance Jobs with Parallel Agents

Trino handles queries and DML, but it does not support Iceberg maintenance procedures. Spark is the engine that runs compaction, snapshot expiration, and orphan file cleanup through its CALL syntax. This dual-engine pattern (Trino for queries, Spark for maintenance) is the standard production approach.

Why Spark for Maintenance Instead of Trino?

Iceberg's maintenance procedures (rewrite_data_files, expire_snapshots, remove_orphan_files) are Spark-specific stored procedures invoked via CALL syntax. Trino handles INSERT, UPDATE, DELETE, and MERGE, but it does not support these maintenance procedures. This is why production lakehouses use both engines together.

You will launch two parallel agents. One generates the maintenance notebook. The other generates an optimization guide for production best practices.

  • Open the Agents sidebar in Cursor and start two new Agent Mode sessions.

Paste this prompt into Agent 1 (Maintenance Notebook):

Create a Jupyter notebook at notebooks/maintenance.ipynb that runs Iceberg maintenance via PySpark. The notebook should: (1) Configure a SparkSession connected to the Polaris catalog at http://polaris:8181/api/catalog with Iceberg extensions, S3 endpoint http://minio:9000, credentials admin/password123, path-style access. (2) Run bin-pack compaction on iceberg.ecommerce.orders using CALL lakehouse.system.rewrite_data_files with target file size 128MB. (3) Run snapshot expiration on the same table using CALL lakehouse.system.expire_snapshots to keep only the last 3 snapshots. (4) Run remove_orphan_files on the same table. (5) Print the results of each operation. Include markdown cells explaining what each step does in plain English at a 7th-grade reading level.

What Do These Three Maintenance Procedures Do?

  • rewrite_data_files (compaction): Merges many small files into fewer large files. The bin-pack strategy preserves write order and is the fastest option. The default target file size is 512MB, but you are using 128MB because your demo data is smaller.
  • expire_snapshots: Removes old snapshots and the data files they exclusively reference. You are keeping the last 3 snapshots for time-travel capability. The default retain_last is 1 if not specified.
  • remove_orphan_files: Cleans up files on storage that are not referenced by any metadata. The default retention interval is 3 days.

Paste this prompt into Agent 2 (Optimization Guide):

Create docs/optimization-guide.md that explains Iceberg maintenance best practices for a production e-commerce lakehouse. Cover: (1) When to run compaction (weekly, or when average file size drops below 100MB). (2) When to expire snapshots (daily, keep 3-7 days for time travel). (3) When to clean orphan files (weekly, with 3-day retention). (4) How to choose target file size (256MB-512MB for production, 128MB for our demo). (5) What monitoring to set up (snapshot count, file count, average file size). Reference the actual table properties and commands used in this project. Plain English, 7th-grade reading level.
  • Review and accept the output from both agents when they finish.
  • Open Jupyter in your browser at http://localhost:8888.
  • Open the notebooks/maintenance.ipynb notebook that Agent 1 generated.
  • Run each cell in order, starting with the SparkSession configuration cell.

What Should I See?

The compaction cell prints the number of files rewritten and the new file count. The snapshot expiration cell prints how many snapshots were removed. The orphan file cleanup cell prints how many orphan files were deleted (this may be zero if no orphans exist yet). Each cell takes a few seconds to run.

Spark Session Failing to Connect?

If the SparkSession cell throws a connection error, the most common cause is the Polaris catalog not being reachable from the Jupyter container. Paste the error into Cursor Agent Mode and ask it to diagnose the issue. Help me debug this Spark connection error.

Generate the Validation Report and Industry Comparison

Now that compaction and expiration have run, you need to measure the results and prove they worked. You will launch two more parallel agents. One captures post-compaction metrics and builds a before/after comparison. The other compares your lakehouse to production benchmarks.

  • Open two new Agent Mode sessions in the Cursor Agents sidebar.

Paste this prompt into Agent 1 (Validation Report):

Connect to Trino and capture post-compaction metrics: file count, average file size, total size from orders files metadata, snapshot count, and revenue-by-month query execution time. Then create docs/validation-report.md that compares pre-compaction metrics from @docs/pre-compaction-metrics.md to post-compaction metrics. Format as a before/after comparison table. Include: file count reduction percentage, average file size increase, snapshot count reduction, query time comparison. Add a summary explaining what compaction and expiration accomplished. Plain English, 7th-grade reading level.

Paste this prompt into Agent 2 (Industry Comparison):

Create docs/industry-comparison.md that compares this project's lakehouse to production benchmarks. Use these reference points: (1) Production Iceberg file sizing best practice is 256-512MB per file for tables over 100GB. Our demo uses 128MB target for smaller data. (2) Dell Federal observed 45 million data files with 5TB of metadata exceeding actual data size when compaction was neglected. Our compaction prevents this. (3) S3 storage costs approximately $0.023/GB/month. Equivalent Snowflake or Databricks warehouse costs $2-8/TB/month for compute on top of storage. Our local setup costs $0. (4) Production lakehouses typically run compaction weekly and snapshot expiration daily. Our project implements this exact pattern. (5) The Iceberg+Trino+Spark stack (open source) delivered 18% faster query performance than Databricks on TPC-H benchmarks at 57% lower cost. Compare each metric to what we built. Assess scalability (what changes to move from laptop to production), reliability (ACID guarantees, time-travel disaster recovery), cost (open source vs proprietary), and performance (query latency targets). Identify gaps: what our demo does not cover that a production system would need (orchestration with Airflow, monitoring with Prometheus, security with RBAC, multi-node Trino cluster). Plain English, 7th-grade reading level.
  • Review and accept the output from both agents.

What Should I See in the Validation Report?

The validation report should show a clear before/after comparison table. File count should be lower after compaction. Average file size should be larger. Snapshot count should be reduced to 3 (or close to it) after expiration. Query execution time may show a slight improvement, though with small demo data the difference might be minimal.

Open docs/validation-report.md and confirm it has a before/after comparison table with percentage changes. Open docs/industry-comparison.md and confirm it covers scalability, reliability, cost, performance, and production gaps. These two documents are portfolio artifacts that demonstrate you can operate and evaluate a lakehouse, not just build one.

Your lakehouse is now fully operational and documented. You have measurable proof that maintenance jobs work, a comparison to production benchmarks, and a portfolio of documentation artifacts. Next up, you will tackle the Secret Mission and use Iceberg time-travel to recover deleted data.

Secret mission

Time-Travel to Recover Deleted Orders

In Step 6, you deleted some returned orders. In a real company, someone might accidentally delete the wrong data. Can you use Iceberg's time-travel feature to recover those deleted rows and prove your lakehouse has built-in disaster recovery?

Before you go

Clean Up Your Resources

This project runs entirely on your local machine using Docker containers. There are zero cloud costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Docker containers: MinIO, Polaris, Trino, Spark/Jupyter, plus two one-time setup containers (setup-bucket, polaris-setup).
  • Docker images: approximately 2-5 GB of disk space for pulled container images.
  • Local project files: docker-compose.yml, SQL files in ./sql/, notebooks in ./notebooks/, configs in ./trino/ and ./polaris/, and documentation in ./docs/.
  • Local data directory: ./data/ containing MinIO object storage data (Parquet files, Iceberg metadata).
  • RAM usage: approximately 8-12 GB while containers are running.

Keep everything running

No action needed. Choose this if you are still actively building or experimenting with your lakehouse.

  • Your containers will keep running and consuming approximately 8-12 GB of RAM.
  • All data persists in the ./data/ directory.
  • You can continue running queries in Trino at http://localhost:8080 and notebooks in Jupyter at http://localhost:8888.

Pause - I'll come back to this later

Shut down running containers to free up RAM, but keep all your data and config files intact.

  • Open PowerShell and navigate to your iceberg-lakehouse project folder.
  • Run docker compose stop to stop all containers without removing them.
  • Your data in ./data/ and all config files remain untouched.
  • When you are ready to resume, run docker compose start from the same folder.

Delete - I don't want to use this again

Remove all project resources and start fresh. You can rebuild everything from scratch at any time since all config files, SQL files, and notebooks are in your project folder.

  • Open PowerShell and navigate to your iceberg-lakehouse project folder.
  • Run docker compose down -v to stop and remove all containers, networks, and volumes.
  • Delete the ./data/ directory to reclaim disk space used by MinIO object storage.
  • Optionally, run docker image prune -a to remove the downloaded Docker images and free up 2-5 GB of disk space.
  • To rebuild from scratch, run docker compose up -d and re-execute the SQL files. All config files remain in your project folder.

Mission accomplished

That's a Wrap!

Nice work! You just built a production-grade Apache Iceberg lakehouse from scratch, entirely through natural language prompts in Cursor IDE. This is the same architecture that powers analytics at companies like Netflix, Apple, and Adobe.

Here's what you accomplished:

  • Built a 5-service lakehouse stack (MinIO, Polaris, Trino, Spark, Jupyter) using Cursor parallel agents, with every config file generated from natural language prompts and cross-checked for consistency.
  • Designed, loaded, and queried a real e-commerce data model with partitioned Iceberg tables, 68 rows of sample data, CDC operations (inserts, updates, deletes), and 5 analytical queries with measured execution times.
  • Ran compaction, snapshot expiration, and orphan file cleanup via Spark, with before-and-after metrics proving that maintenance reduced file count, increased average file size, and reclaimed storage.
  • Recovered "accidentally deleted" orders using Iceberg time-travel, proving your lakehouse has built-in disaster recovery.

Ready to quiz yourself?