Build an ELT Pipeline with DuckDB

Extract live weather data, load it into DuckDB, and transform it with SQL.

Introduction

30 Second Summary

Every weather app, stock dashboard, and recommendation engine relies on the same behind-the-scenes pattern: pulling raw data from somewhere, storing it, then transforming it into something useful. Data engineers call this an ELT pipeline.

In this project, you will build a complete ELT pipeline using Python and DuckDB. You'll pull live weather data for 5 world cities from a free API, load it into an analytical database, transform it with SQL, and export polished reports to Parquet files.

What You'll Build

When you run your finished pipeline, a single command fetches live weather readings for London, New York, Tokyo, Sydney, and Berlin, then prints a formatted comparison table showing which city is hottest, coldest, windiest, and most humid.

By the end of this project, you'll have:

  • A single-command pipeline you can run to watch live weather data flow from the Open-Meteo API into your database and out as clean analytical reports.
  • A queryable DuckDB database where you can answer questions like "which city had the highest temperature today?" using plain SQL.
  • Exported Parquet files you can open in any data tool (pandas, Spark, or DuckDB itself) for further analysis or hand to a teammate.
  • Secret Mission: Add data quality validation that catches bad data before it reaches your reports.

Are there any prerequisites?

This project assumes basic familiarity with Python (variables, functions, loops) and SQL (SELECT, WHERE, GROUP BY). You do not need any prior experience with DuckDB, APIs, or data pipelines.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters for your data engineering toolkit.

Set Up Your Pipeline Tools

You have committed to building a complete ELT pipeline. Now you need the right tools in place before you can write any pipeline logic.

This step sets up your project folder, your code file, and the two Python packages that power the pipeline: DuckDB for analytical queries and httpx for calling APIs.

In this step, get ready to:
  • Create a project folder and an empty pipeline.py file.
  • Install DuckDB and httpx with pip.
  • Verify both packages are working with version-check commands.
Create your project folder and pipeline file

Every project needs a home. You will create a dedicated folder on your Desktop and an empty Python file that will hold your pipeline code.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or PowerShell (Windows) and press Enter to open it.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir weather-pipeline
cd weather-pipeline
  • Create an empty pipeline file by running this command:

macOS

touch pipeline.py

Windows

New-Item pipeline.py -ItemType File
  • Confirm the file exists by running:
ls

You should see pipeline.py listed in the output.

Don't see pipeline.py?

Make sure you ran the create command inside the weather-pipeline folder. Run pwd (macOS) or cd (Windows) to check your current directory. It should end with weather-pipeline.

help me find my project folder

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type VS Code and press Enter to open it.
  • Click File in the top menu bar.
  • Click Open Folder.
  • Navigate to your Desktop and select the weather-pipeline folder, then click Open.

You should see weather-pipeline in the VS Code sidebar with pipeline.py listed underneath it.

Install DuckDB and httpx

DuckDB is a high-performance analytical database that runs inside your Python process. There is no server to install or configure. httpx is a modern HTTP client for calling APIs.

  • Install both packages by running this command in your terminal:
pip install duckdb httpx

What is DuckDB?

Most databases (like PostgreSQL or MySQL) run as a separate server process that your code connects to over a network. DuckDB is different. It runs entirely inside your Python process and stores data in a single file.

DuckDB is designed for analytical workloads: scanning large tables, computing aggregations, and running complex joins. This makes it ideal for data pipelines where you need to transform and summarize data with SQL.

You should see pip download and install both packages with a success message.

Seeing a permission error?

If pip shows a permissions error, try running pip install --user duckdb httpx instead. This installs into your user directory without needing admin access.

help me fix pip install errors

Verify your packages are installed

Before writing any pipeline code, you need to confirm both packages imported correctly and are at the expected versions.

  • Before you run the next commands, what version numbers do you expect to see for DuckDB and httpx?
  • Verify DuckDB is installed by running:
python -c "import duckdb; print(duckdb.__version__)"

You should see 1.5.5 printed.

  • Verify httpx is installed by running:
python -c "import httpx; print(httpx.__version__)"

You should see 0.28.1 printed.

Seeing ModuleNotFoundError?

If Python cannot find the module, your pip and python commands may point to different Python installations. Try running pip3 install duckdb httpx and then python3 -c "import duckdb; print(duckdb.__version__)" instead.

help me fix my Python import error

Your tools are locked in. Next up, you will write the first stage of your pipeline: extracting live weather data from an API and loading it straight into DuckDB.

Extract and Load Weather Data

Your tools are installed, but right now your pipeline file is empty. Before you can run any analysis, you need actual data in your database.

The fastest path is to pull live data from an API and load it straight into DuckDB. In this step, you will call the Open-Meteo weather API for London, load the response into a table, and see real weather readings in your terminal.

In this step, get ready to:
  • Write an extract function that fetches live weather data from a public API.
  • Write a load function that inserts API data into a DuckDB table.
  • Wire both together in a main block and run the pipeline end-to-end.
Write the extract function

The Open-Meteo API is a free weather forecast service that returns hourly readings as JSON. You will request temperature, humidity, and wind speed for 7 days (168 hours) for London.

  • In VS Code, open pipeline.py and add the following imports and extract function:
import httpx
import duckdb
import os

API_BASE_URL = "https://api.open-meteo.com/v1/forecast"


def extract_weather(city):
    """Fetch hourly weather data from Open-Meteo API for a single city."""
    params = {
        "latitude": city["latitude"],
        "longitude": city["longitude"],
        "hourly": "temperature_2m,relative_humidity_2m,wind_speed_10m",
        "timezone": "auto",
    }
    response = httpx.get(API_BASE_URL, params=params)
    response.raise_for_status()
    return response.json()

What does this code do?

API_BASE_URL stores the Open-Meteo forecast endpoint. All requests will target this URL.

extract_weather takes a city dictionary (with latitude, longitude, and name) and asks the API for three hourly variables: temperature_2m, relative_humidity_2m, and wind_speed_10m. The timezone: "auto" parameter tells the API to resolve timestamps to the city's local time zone.

raise_for_status() will raise an exception if the request fails, so you catch problems immediately instead of silently passing bad data downstream.

  • Save pipeline.py.
  • Test the extract function by running this command in your terminal:
python -c "from pipeline import extract_weather; data = extract_weather({'name': 'London', 'latitude': 51.51, 'longitude': -0.13}); print(len(data['hourly']['time']), 'hours of data')"

You should see 168 hours of data printed. That confirms the API returned 7 days of hourly readings.

Seeing a connection error?

Make sure you have an active internet connection. The Open-Meteo API is free and requires no API key, but it does require network access.

If you see a ModuleNotFoundError, confirm that httpx is installed in the same Python environment you are running the command from.

help me debug my extract_weather function

Write the load function

Now that you can pull data from the API, you need a function to insert it into DuckDB. The API response contains arrays of values. Your load function will reshape them into rows and insert them using executemany.

  • In pipeline.py, add the following function below extract_weather:
def load_weather(con, city_name, raw_data):
    """Load raw weather data into DuckDB, enriching with city name."""
    hourly = raw_data["hourly"]
    rows = []
    for i in range(len(hourly["time"])):
        rows.append((
            city_name,
            hourly["time"][i],
            hourly["temperature_2m"][i],
            hourly["relative_humidity_2m"][i],
            hourly["wind_speed_10m"][i],
        ))
    con.executemany("INSERT INTO raw_weather VALUES (?, ?, ?, ?, ?)", rows)

What does this code do?

load_weather accepts a DuckDB connection, a city name string, and the raw JSON response from the API.

It loops through the parallel arrays in the hourly object, building a list of tuples. Each tuple is one row: city name, timestamp, temperature, humidity, and wind speed.

executemany inserts all 168 rows in a single batch, which is much faster than inserting one at a time. The ? placeholders keep the SQL safe from injection.

  • Save pipeline.py.

Why include city_name in every row?

The Open-Meteo API response does not include a location identifier. It only returns arrays of numbers. If you later load data for multiple cities into the same table, every row would look identical without this enrichment. Adding the city name during load is a core ELT concept: metadata enrichment at load time.

Wire it together in main

You have an extract function and a load function. Now you need a main block that connects to a persistent DuckDB file, creates the table, and runs both functions.

  • Add the following main function and entry point at the bottom of pipeline.py:
def main():
    con = duckdb.connect("weather_pipeline.duckdb")

    con.execute("""
        CREATE OR REPLACE TABLE raw_weather (
            city VARCHAR,
            time TIMESTAMP,
            temperature_2m DOUBLE,
            relative_humidity_2m DOUBLE,
            wind_speed_10m DOUBLE
        )
    """)

    city = {"name": "London", "latitude": 51.51, "longitude": -0.13}
    print(f"  Fetching {city['name']}...")
    raw_data = extract_weather(city)
    load_weather(con, city["name"], raw_data)

    row_count = con.execute("SELECT COUNT(*) FROM raw_weather").fetchone()[0]
    print(f"Loaded {row_count} rows into raw_weather.\n")
    con.sql("SELECT * FROM raw_weather LIMIT 5").show()

    con.close()


if __name__ == "__main__":
    main()

What does this code do?

duckdb.connect("weather_pipeline.duckdb") creates a persistent database file on disk. Your data survives between runs.

CREATE OR REPLACE TABLE ensures the pipeline is idempotent. You can re-run it without hitting "table already exists" errors.

The main block calls extract_weather for London, pipes the result into load_weather, then queries the table to confirm how many rows loaded and shows a 5-row preview with con.sql(...).show().

  • Save pipeline.py.

Before you run it, what do you expect to see? Think about how many hours are in 7 days of hourly data.

  • Run your pipeline by executing this command in your terminal:
python pipeline.py

You should see output confirming 168 rows loaded, followed by a formatted 5-row preview showing timestamps, temperature, humidity, and wind speed values for London.

Seeing 0 rows or an error?

If you see a NameError, check that extract_weather and load_weather are defined above the main function in your file.

If you see a BinderException about column count, make sure your INSERT has exactly 5 ? placeholders matching the 5 columns in the table.

help me debug my pipeline

✔️ Awesome, I've got everything!

Great. Double check you have saved pipeline.py and confirm the output shows 168 rows.

ⓧ I'd like to double check the full code

Your pipeline.py should contain these three sections in order. Scroll up and confirm each one is present:

  • Imports (httpx, duckdb, os) and API_BASE_URL.
  • extract_weather function that calls the API and returns JSON.
  • load_weather function that builds rows and inserts them with executemany.
  • main function that connects to weather_pipeline.duckdb, creates the table, runs extract and load for London, prints the row count, and shows a 5-row preview.
  • if __name__ == "__main__": block calling main().

You have a working extract-and-load pipeline pulling live data into an analytical database. Next up, you will scale this to 5 cities and see why that city name enrichment becomes essential.

Scale to Multiple Cities

Your pipeline works for London. But a single city doesn't give you much to compare. To answer questions like "which city is hottest today?", you need data from multiple locations in the same database.

The Open-Meteo API doesn't include a city name in its response data. Your load_weather function already handles this by enriching each row with the city name during insertion. Now you'll scale from one city to five by adding a configuration list and looping over it.

In this step, get ready to:
  • Add a CITIES configuration list with 5 world cities and their coordinates.
  • Update the main block to loop over all cities.
  • Verify all 5 cities load correctly with 840 total rows.
Add the CITIES configuration list

A central configuration list makes it easy to add or remove cities without touching pipeline logic. Each entry stores a name plus the latitude and longitude the API needs.

  • In pipeline.py, find the line that says API_BASE_URL = "https://api.open-meteo.com/v1/forecast".
  • Add the following CITIES list directly above that line:
# --- Configuration ---
CITIES = [
    {"name": "London", "latitude": 51.51, "longitude": -0.13},
    {"name": "New York", "latitude": 40.71, "longitude": -74.01},
    {"name": "Tokyo", "latitude": 35.68, "longitude": 139.69},
    {"name": "Sydney", "latitude": -33.87, "longitude": 151.21},
    {"name": "Berlin", "latitude": 52.52, "longitude": 13.41},
]

What does this code do?

Each dictionary stores a city's display name and its geographic coordinates. The extract_weather function already accepts a dictionary with latitude and longitude keys, so these entries plug straight into the existing pipeline.

  • Save pipeline.py.
Update main to loop over all cities

Right now the main block hard-codes a single London dictionary. You'll replace that with a loop over the new CITIES list so every city gets extracted and loaded in sequence.

  • In the main() function, find these lines (below the CREATE OR REPLACE TABLE block):
    city = {"name": "London", "latitude": 51.51, "longitude": -0.13}
    print(f"  Fetching {city['name']}...")
    raw_data = extract_weather(city)
    load_weather(con, city["name"], raw_data)
  • Replace those 4 lines with the following loop:
    print("Extracting weather data...")
    for city in CITIES:
        print(f"  Fetching {city['name']}...")
        raw_data = extract_weather(city)
        load_weather(con, city["name"], raw_data)

What does this code do?

The for city in CITIES loop iterates over each dictionary in your configuration list. For each city, it calls extract_weather to fetch the API data, then load_weather to insert the rows with the city name attached.

This is metadata enrichment in action. The API response has no location identifier, but your pipeline adds one during the load phase. That's a core ELT technique.

Why enrich at load time?

In the ELT pattern, raw data is loaded as-is. But when the source doesn't include essential metadata (like which city a reading belongs to), you add it during the load phase. This keeps the raw table queryable without needing a separate lookup.

  • Save pipeline.py.

Before you run this, how many rows do you expect to see loaded? Each city returns 168 hours of data.

  • Run your pipeline to confirm the loop works:
python pipeline.py

You should see each city fetched in sequence, then a total of 840 rows loaded (168 hours x 5 cities). The preview still shows the old 5-row query for now.

Seeing fewer than 840 rows?

  • Make sure CITIES is defined at the top of your file (above API_BASE_URL), not inside a function.
  • Check that your for city in CITIES: loop is inside main() and that the extract/load calls are indented under it.
  • If you see a NameError: name 'CITIES' is not defined, the list hasn't been saved or is misspelled.

Help me debug my CITIES loop

Add a verification query for distinct cities

The old preview query shows raw rows but doesn't confirm which cities are actually in the table. A targeted SELECT DISTINCT query answers that directly.

  • In main(), find this line near the bottom (after the row_count print):
    con.sql("SELECT * FROM raw_weather LIMIT 5").show()
  • Replace it with a query that shows all distinct city names:
    con.sql("SELECT DISTINCT city FROM raw_weather").show()
  • Save pipeline.py.

What do you expect to see in the distinct cities output? Will all 5 names appear?

  • Run the pipeline one more time to verify:
python pipeline.py

You should see 840 rows loaded, followed by a table listing all 5 city names: London, New York, Tokyo, Sydney, and Berlin.

Missing a city from the output?

  • Count the entries in your CITIES list. Each dictionary must have name, latitude, and longitude keys.
  • Make sure each entry ends with a comma (including the last one before the closing ]).

Help me check my CITIES configuration

✔️ Awesome, I've got everything!

Great. Double check you've saved pipeline.py and that your terminal shows 840 rows with all 5 city names.

ⓧ I'd like to double check the full code

Scroll up and confirm your file has these sections in order:

  • Imports (httpx, duckdb, os) at the top.
  • The CITIES list with 5 entries (London, New York, Tokyo, Sydney, Berlin).
  • API_BASE_URL below the CITIES list.
  • extract_weather function (unchanged from Step 2).
  • load_weather function (unchanged from Step 2).
  • main() with a for city in CITIES: loop and a SELECT DISTINCT city verification query.

Your pipeline now pulls live weather data for 5 cities around the world and loads 840 rows into DuckDB with proper city labels. Next up, you'll transform this raw data into meaningful reports using SQL.

Transform and Export Reports

Your pipeline now extracts live data from the Open-Meteo API and loads 840 rows into DuckDB. Raw hourly readings are useful for storage, but they do not answer questions like "which city is hottest this week?" on their own.

This is where the "T" in ELT comes in. You will write SQL transformations that aggregate the raw data into daily summaries and city comparisons, then export the results to Parquet files that any analytics tool can read.

In this step, get ready to:
  • Write SQL transformations that turn hourly readings into daily summaries and city rankings.
  • Export the transformed tables to Parquet, the industry-standard columnar format.
  • Print a formatted comparison table showing which city is hottest, coldest, and windiest.
Write the transform function

SQL aggregations like AVG, MIN, and MAX collapse thousands of hourly rows into meaningful summaries. Because DuckDB is an analytical database, these operations run directly inside it, which is much faster than pulling data out and processing it in Python.

  • In pipeline.py, add the following transform function below your load_weather function:
def transform(con):
    """Run SQL transformations to produce analytical reports."""
    con.execute("""
        CREATE OR REPLACE TABLE daily_city_weather AS
        SELECT
            city,
            CAST(time AS DATE) AS date,
            ROUND(AVG(temperature_2m), 1) AS avg_temp_c,
            ROUND(MIN(temperature_2m), 1) AS min_temp_c,
            ROUND(MAX(temperature_2m), 1) AS max_temp_c,
            ROUND(AVG(relative_humidity_2m), 1) AS avg_humidity_pct,
            ROUND(AVG(wind_speed_10m), 1) AS avg_wind_kmh
        FROM raw_weather
        GROUP BY city, CAST(time AS DATE)
        ORDER BY city, date
    """)

    con.execute("""
        CREATE OR REPLACE TABLE city_comparison AS
        SELECT
            city,
            ROUND(AVG(temperature_2m), 1) AS overall_avg_temp_c,
            ROUND(MAX(temperature_2m), 1) AS highest_temp_c,
            ROUND(MIN(temperature_2m), 1) AS lowest_temp_c,
            ROUND(AVG(wind_speed_10m), 1) AS avg_wind_kmh,
            COUNT(*) AS total_readings
        FROM raw_weather
        GROUP BY city
        ORDER BY overall_avg_temp_c DESC
    """)

What does this code do?

This function creates two new tables directly from the raw data using SQL:

  • daily_city_weather groups hourly readings by city and date. For each day, it computes the average, minimum, and maximum temperature, plus average humidity and wind speed. ROUND(..., 1) keeps values to one decimal place.
  • city_comparison ranks all five cities by their overall average temperature (hottest first). It also shows the highest and lowest single reading per city and the total number of data points.
  • CREATE OR REPLACE TABLE makes the pipeline idempotent. You can re-run it without getting errors about tables already existing.

Now wire the transform function into your pipeline. In the main function, find these lines near the bottom:

    con.sql("SELECT DISTINCT city FROM raw_weather").show()

    con.close()
  • Replace those lines with the following code that calls your transform function:
    # Transform
    print("Running transformations...")
    transform(con)
    print("Created: daily_city_weather, city_comparison\n")

    con.close()
  • Save pipeline.py and run the pipeline to test the transform:
python pipeline.py

You should see the extraction messages for all 5 cities, the 840 rows loaded count, then "Running transformations..." followed by "Created: daily_city_weather, city_comparison".

Seeing a NameError for transform?

  • Make sure the transform function is defined above the main function. Python reads top to bottom, so functions must be defined before they are called.
  • Check that def transform(con): is at the module level (no indentation), not accidentally nested inside another function.

Why is my transform function not being recognized?

Write the export function

Parquet is the industry-standard columnar file format for analytical data. Tools like pandas, Apache Spark, and DuckDB itself can all read Parquet files directly.

  • In pipeline.py, add the following export_reports function below your transform function:
def export_reports(con):
    """Export transformed tables to Parquet format."""
    os.makedirs("output", exist_ok=True)
    con.execute("COPY daily_city_weather TO 'output/daily_city_weather.parquet' (FORMAT PARQUET)")
    con.execute("COPY city_comparison TO 'output/city_comparison.parquet' (FORMAT PARQUET)")

What does this code do?

  • os.makedirs("output", exist_ok=True) creates an output/ folder if it does not already exist.
  • COPY ... TO ... (FORMAT PARQUET) is DuckDB's built-in export command. It writes an entire table to a Parquet file with no extra Python libraries needed.

Now add the export call to your main function. In main, find this line:

    print("Created: daily_city_weather, city_comparison\n")
  • Add the following export block directly below it (before con.close()):
    # Export
    print("Exporting to Parquet...")
    export_reports(con)
    print("Saved: output/daily_city_weather.parquet, output/city_comparison.parquet\n")
  • Save pipeline.py and run the pipeline:
python pipeline.py
  • Confirm the Parquet files were created by running:
ls output/

You should see city_comparison.parquet and daily_city_weather.parquet listed in the output.

Don't see the output folder?

  • Make sure you still have import os at the top of your file. The os.makedirs call needs it.
  • Check that export_reports(con) is called after transform(con). The tables must exist before you can export them.

Why are my Parquet files not being created?

Add result previews to main

A good pipeline shows its results every time it runs. You will add formatted table previews so you can see the city comparison and daily summaries directly in the terminal.

  • In main, find this line:
    con.close()
  • Add the following preview code directly above con.close():
    # Preview results
    print("--- City Comparison ---")
    con.sql("SELECT * FROM city_comparison").show()

    print("\n--- Daily Weather (first 10 rows) ---")
    con.sql("SELECT * FROM daily_city_weather LIMIT 10").show()
  • Finally, add a completion message after con.close():
    con.close()
    print("\nPipeline complete!")

What does this code do?

  • con.sql("SELECT * FROM city_comparison").show() prints a formatted table of all five cities ranked by average temperature. No pandas required.
  • The LIMIT 10 on daily weather keeps the output readable. The full table has 35 rows (5 cities x 7 days).

Before you run the final pipeline, what do you think the City Comparison table will show? Which city do you expect to have the highest average temperature right now?

  • Save pipeline.py and run the full pipeline:
python pipeline.py

You should see the full pipeline output: extraction messages for 5 cities, 840 rows loaded, transformations created, Parquet files saved, then a formatted City Comparison table showing each city's average, highest, and lowest temperatures.

  • Verify the Parquet file is independently readable by running:
python -c "import duckdb; duckdb.sql(\"SELECT * FROM 'output/city_comparison.parquet'\").show()"

You should see the same City Comparison table printed again. This proves the Parquet file is a self-contained, portable export that any tool can read without your pipeline running.

Seeing a file not found error on the Parquet read?

  • Make sure you are running the command from the same directory as your output/ folder. Run ls output/ to confirm the files exist.
  • If you see a syntax error, check the escaped quotes carefully. The outer quotes use double quotes, and the inner SQL uses single quotes around the file path.

Help me debug my Parquet file read command.

✔️ Awesome, I've got everything!

Great. Double check you have saved pipeline.py and that both Parquet files exist in your output/ folder.

ⓧ I'd like to double check the full code

Your pipeline.py should contain these sections in order. Scroll up and confirm each one is present:

  • Imports: httpx, duckdb, os.
  • The CITIES list with 5 city dictionaries and API_BASE_URL.
  • The extract_weather(city) function.
  • The load_weather(con, city_name, raw_data) function.
  • The transform(con) function with two CREATE OR REPLACE TABLE statements.
  • The export_reports(con) function with os.makedirs and two COPY statements.
  • The main() function: creates table, loops extract/load, calls transform, calls export_reports, prints previews, closes connection, prints "Pipeline complete!".
  • The if __name__ == "__main__": guard at the bottom.

Your ELT pipeline is complete. You can extract live data, load it into an analytical database, transform it with SQL, and export portable Parquet files. Next up, a secret mission to add data quality validation that catches bad data before it reaches your reports.

Secret mission

Add Data Quality Validation

Production pipelines never blindly trust their data. A single null value or out-of-range reading can silently corrupt your reports. Can you add a validation gate that catches bad data before it reaches your transforms?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Project folder containing pipeline.py and weather_pipeline.duckdb.
  • Exported Parquet files in the output/ folder.
  • Python packages: duckdb and httpx.

Keep everything running

No action needed. Choose this if you are still actively building or plan to extend the pipeline with new cities, variables, or data sources.

  • Your project files stay on your Desktop.
  • The weather_pipeline.duckdb file persists your data between runs.
  • The duckdb and httpx packages remain installed in your Python environment.

Pause - I'll come back to this later

Free up disk space by removing the generated database and exports while keeping your code. You can regenerate everything by running the pipeline again.

  • Delete the weather_pipeline.duckdb file from your project folder. This frees the most disk space.
  • Delete the output/ folder containing your Parquet files.
  • Keep pipeline.py so you can re-run the pipeline later to regenerate everything.

Delete - I don't want to use this again

Remove all project files and uninstall the Python packages to start fresh.

  • Delete your project folder from the Desktop. This is the folder containing pipeline.py and weather_pipeline.duckdb.
  • On macOS/Linux, open your terminal and run:
cd ~/Desktop
rm -rf [[PROJECT_FOLDER="your project folder name"]]
  • On Windows, open PowerShell and run:
cd ~\Desktop
Remove-Item -Recurse -Force [[PROJECT_FOLDER="your project folder name"]]
  • Uninstall the Python packages by running:
pip uninstall duckdb httpx -y
  • Confirm the packages are removed by running:
python -c "import duckdb"

You should see a ModuleNotFoundError confirming the package is gone.

Nice Work!

Nice Work!

You just built a complete ELT pipeline from scratch using Python, DuckDB, and a live weather API.

You've learned how to:

  • Extract live data from the Open-Meteo API using httpx and load it into a DuckDB analytical database with metadata enrichment (city names added during the load phase).
  • Transform raw hourly readings into daily summaries and city comparison reports using SQL aggregations inside the database.
  • Export analytical results to Parquet, the industry-standard columnar format that any data tool can read.
  • Secret Mission: Added data quality validation that catches nulls, out-of-range values, duplicates, and missing cities before they corrupt your reports.

Ready to quiz yourself?