Build Your First ETL Pipeline with Python

Introduction

30 Second Summary

Every recommendation on Netflix and every forecast on your weather app depends on data that started somewhere messy. Someone had to pull it from a source, clean it up, and store it somewhere useful.

In this project, you will build your first ETL pipeline in Python. It pulls live repository data from the GitHub API, transforms messy JSON into clean records, and loads everything into a SQLite database you can query with SQL.

What You'll Build

Run a single Python script and watch it pull trending repositories from GitHub, clean up missing fields, assign popularity tiers, and load everything into a database you can query on the spot.

By the end of this project, you'll have:

• A working ETL pipeline you run with a single command to extract live data from GitHub, transform it, and load it into a real database.
• Your first SQL queries to analyze the data you collected, answering questions like "which repos have the most stars?" and "which languages dominate?".
• A pipeline report printed in your terminal showing how many records were extracted, how many had quality issues, and how many loaded successfully.
• Secret Mission: Add data quality checks and a pipeline audit report that flags bad records and tracks pipeline health metrics.

Are there any prerequisites?

This project assumes basic Python knowledge, including functions, loops, dictionaries, and lists. No SQL or database experience is needed.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

Set Up Your Project

Before you can build a pipeline, you need a clean workspace with the right tools installed. Your ETL pipeline will pull live data from the GitHub API and store it in a SQLite database.

In this step, you'll create a project folder, install the one external library you need, and confirm everything is ready to go.

In this step, get ready to:

• Create a project directory on your Desktop.
• Install the requests library for making API calls.
• Verify your environment is ready to build the pipeline.

Create your project directory

• Move to your Desktop by running this command:

cd ~/Desktop

• Create your project folder and move into it by running these commands:

mkdir etl-pipeline
cd etl-pipeline

• Confirm you're in the right place by running:

pwd

You should see a path ending in /Desktop/etl-pipeline.

Don't see etl-pipeline in the path?

• Make sure you ran cd ~/Desktop before creating the folder. If you skipped it, the folder may have been created in your home directory instead.
• Check for typos in the folder name. The folder must be called etl-pipeline exactly.
• Help me find my project directory

Install the requests library

Python's sqlite3 module is already built into Python, so you do not need to install it. However, you need one external library called requests to make HTTP calls to the GitHub API.

• Install requests by running:

pip install requests==2.34.2

What is the requests library?

The requests library makes it simple to send HTTP requests to APIs and get data back. Your pipeline will use it to fetch repository data from GitHub's REST API.

You're pinning the version to 2.34.2 so your pipeline behaves the same way every time you run it. Pinning versions is a common practice in data engineering to keep pipelines reproducible.

Seeing 'pip: command not found'?

• On some Linux systems, the command is pip3 instead of pip. Try running pip3 install requests==2.34.2 instead.
• If you see a permissions error, try adding --user to the end of the command.
• Help me install the requests library

Verify your setup

Before you move on, what do you think will happen when you try to import both requests and sqlite3 in a single Python command? Will both imports succeed?

• Test that both libraries are available by running this verification command:

python3 -c "import requests; import sqlite3; print('Setup complete! requests', requests.__version__)"

You should see this printed in your terminal:

Setup complete! requests 2.34.2

Both imports succeeded in a single command. The requests library is the one you just installed, and sqlite3 was already bundled with Python.

Seeing an ImportError?

• If the error mentions requests, the library did not install correctly. Re-run the install command from the previous substep.
• Make sure you are using the same Python version for both installing and running. If you installed with pip3, use python3 to run the verification command.
• Help me fix my Python imports

Your workspace is set up and ready to go. Next, you'll write your first Python function to pull live repository data from the GitHub API.

Extract Data from the GitHub API

Your project directory is set up and your dependencies are locked in. Now it's time to build the first stage of your ETL pipeline: extraction.

You'll write a Python function that calls the GitHub API to pull data about trending Python repositories. The data arrives as raw JSON, giving you a firsthand look at how real-world data comes in before any cleanup.

In this step, get ready to:

• Write an extract function that pulls live data from the GitHub Search API.
• Run your pipeline script and see real repository data printed to the terminal.

Write the extract function

Every data pipeline starts at a source. Yours is the GitHub Search Repositories API, which lets you search for repositories by language, star count, and more.

• Create a file called pipeline.py in your project directory and paste in this code:

import requests
import sqlite3
import time


def extract():
    """Extract trending Python repositories from the GitHub API."""
    url = "https://api.github.com/search/repositories"
    params = {
        "q": "language:python stars:>1000",
        "sort": "stars",
        "order": "desc",
        "per_page": 30
    }

    response = requests.get(url, params=params)
    response.raise_for_status()

    data = response.json()
    repos = data["items"]

    print(f"Extracted {len(repos)} repositories from GitHub API")
    return repos

What does this code do?

• The three imports bring in requests for HTTP calls, sqlite3 for database access (used in a later step), and time for tracking pipeline duration later.
• The params dictionary tells the API to search for Python repositories with more than 1,000 stars, sorted by star count in descending order, returning 30 results.
• raise_for_status() stops the script with a clear error if the API call fails (e.g. a network timeout or rate limit hit).
• response.json() parses the raw HTTP response into a Python dictionary. The actual repository list lives under the "items" key.

• Save pipeline.py.
• Confirm the file was created by running:

ls

You should see pipeline.py listed in the output.

Don't see pipeline.py?

• Make sure you saved the file in your project directory, not somewhere else on your machine.
• Check that the filename is exactly pipeline.py with no typos or extra extensions.

Help me figure out why my file isn't showing up in the project directory.

Add the main block to run extraction

The extract() function exists, but nothing calls it yet. Python scripts need a main block to kick things off when you run the file.

• In pipeline.py, add this code at the bottom of the file, below the extract() function:

if __name__ == "__main__":
    raw_data = extract()
    print(f"Total repos: {len(raw_data)}")
    print(f"First repo: {raw_data[0]['name']} with {raw_data[0]['stargazers_count']} stars")

What does this code do?

• if __name__ == "__main__": ensures this code only runs when you execute the script directly, not when another file imports it.
• It calls extract() and then prints a quick summary: the total count and the first repository's name and star count.

• Save pipeline.py.

Run the script and examine the raw data

Your script is ready to call the live GitHub API. Before you run it, think about what you expect to see. How many repos should come back? What format will the data be in?

• Run your pipeline by executing this command:

python3 pipeline.py

You should see output showing that 30 repositories were extracted, along with the first repo's name and its star count.

What should I see?

Your terminal prints three lines. The first confirms how many repositories were extracted from the API. The second shows the total count. The third shows the name and star count of the top Python repository on GitHub right now.

Take a closer look at your code. Notice that the star count field is called stargazers_count, not just stars. The owner's name is nested inside an owner object as owner.login.

Why is this data so messy?

This is what raw API data looks like in the real world. Each repository record contains dozens of fields you do not need, uses inconsistent naming like stargazers_count, and wraps related data inside nested objects.

Some fields like description and language can be None (null). Loading this into a database without cleanup would create a mess. That is exactly what transformation solves in the next step.

Seeing an error instead of results?

• If you see ModuleNotFoundError: No module named 'requests', the requests library is not installed. Go back to Step 1 and run the install command again.
• A connection error usually means your internet connection dropped. Check your network and try again.
• If you ran the script many times in quick succession, you may have hit the GitHub API rate limit (10 requests per minute for unauthenticated calls). Wait one minute and try again.

Help me debug why my pipeline.py script is failing when I run it.

✔️ Awesome, I've got everything!

Great. Double check you've saved pipeline.py and you're ready to move on.

ⓧ I'd like to double check the full code

Here is the complete pipeline.py file so far. Compare yours against it line by line.

import requests
import sqlite3
import time


def extract():
    """Extract trending Python repositories from the GitHub API."""
    url = "https://api.github.com/search/repositories"
    params = {
        "q": "language:python stars:>1000",
        "sort": "stars",
        "order": "desc",
        "per_page": 30
    }

    response = requests.get(url, params=params)
    response.raise_for_status()

    data = response.json()
    repos = data["items"]

    print(f"Extracted {len(repos)} repositories from GitHub API")
    return repos


if __name__ == "__main__":
    raw_data = extract()
    print(f"Total repos: {len(raw_data)}")
    print(f"First repo: {raw_data[0]['name']} with {raw_data[0]['stargazers_count']} stars")

You've just pulled live data from the GitHub API with a single function call. The data works, but it's messy. In the next step, you'll write a transform() function that cleans up those missing values, flattens the nested fields, and adds a brand-new column to categorize each repo by popularity.

Transform the Raw Data

In Step 2, you pulled raw data from GitHub and saw it arrive as deeply nested JSON with dozens of fields, null values, and no structure for analysis. That raw output is exactly how real-world API data looks before a pipeline cleans it up.

Transformation is the "T" in ETL, and it is where most of a data pipeline's value is created. In this step, you will write a function that selects only the fields you need, handles missing values, and adds a calculated popularity column.

In this step, get ready to:

• Write a transform() function that cleans raw API data and handles missing values.
• Add a popularity tier column based on star count.
• Run the pipeline and verify the cleaned output in a formatted table.

Write the transform function

The raw data from GitHub contains fields like stargazers_count nested inside objects, null values for descriptions and languages, and dozens of fields you do not need. Your transform function will flatten, clean, and enrich this data into simple, consistent records.

• In pipeline.py, add the following transform() function below the extract() function by copying this code:

def transform(raw_repos):
    """Clean and transform raw repository data."""
    cleaned = []

    for repo in raw_repos:
        record = {
            "name": repo["name"],
            "owner": repo["owner"]["login"],
            "stars": repo["stargazers_count"],
            "forks": repo["forks_count"],
            "language": repo["language"] if repo["language"] else "Unknown",
            "description": repo["description"] if repo["description"] else "No description provided",
            "url": repo["html_url"],
            "created_at": repo["created_at"],
        }

        if record["stars"] >= 50000:
            record["popularity"] = "viral"
        elif record["stars"] >= 10000:
            record["popularity"] = "popular"
        else:
            record["popularity"] = "emerging"

        cleaned.append(record)

What does this code do?

This is the core of your transformation logic. For each raw repo, it builds a clean record dictionary with only the fields you need.

• Nested fields get flattened: repo["owner"]["login"] becomes a simple owner string, and stargazers_count becomes stars.
• Missing values get replaced: None descriptions become "No description provided", and None languages become "Unknown".
• A new popularity column categorizes each repo as "viral" (50,000+ stars), "popular" (10,000+ stars), or "emerging". This is a derived column, one of the most common transformation patterns.

• Still inside the transform() function, add this code directly after the cleaned.append(record) line, outdented one level so it sits at the function's top level (aligned with the for loop, not inside it):

    print(f"Transformed {len(cleaned)} records")

    print(f"\n{'Name':<30} {'Owner':<20} {'Stars':<10} {'Popularity':<12}")
    print("-" * 72)
    for r in cleaned[:10]:
        print(f"{r['name']:<30} {r['owner']:<20} {r['stars']:<10} {r['popularity']:<12}")
    if len(cleaned) > 10:
        print(f"... and {len(cleaned) - 10} more records")

    return cleaned

What does the table printing do?

• The :<30 syntax is Python's string formatting for left-aligned columns. Each value gets padded to a fixed width so the output lines up neatly.
• Only the first 10 records are printed to avoid flooding the terminal. The remaining count is shown as a summary line.
• The function returns cleaned so the next pipeline stage can use the transformed data.

Update the main block to call transform

Your __main__ block currently prints the raw data. You need to replace that with a call to your new transform() function.

• Scroll to the bottom of pipeline.py and find this code:

if __name__ == "__main__":
    raw_data = extract()
    print(f"Total repos: {len(raw_data)}")
    print(f"First repo: {raw_data[0]['name']} with {raw_data[0]['stargazers_count']} stars")

• Replace the two print lines with a single call to transform(), so the block looks like this:

if __name__ == "__main__":
    raw_data = extract()
    cleaned_data = transform(raw_data)

• Save pipeline.py.

Run the pipeline and verify the output

Before you run the script, think about what you expect to see. The raw data had nested objects and null values. What do you think the output will look like now that your transform function is cleaning it?

• Run the pipeline by entering this command:

python3 pipeline.py

You should see a formatted table showing the first 10 records with columns for Name, Owner, Stars, and Popularity. Each repo has a popularity tier assigned based on its star count.

What should I see?

The output starts with "Extracted 30 repositories from GitHub API" followed by "Transformed 30 records". Below that is a table with clean, aligned columns.

Every repo has a value in each column. Repos that had no language show "Unknown" instead of a blank. The popularity column shows "viral", "popular", or "emerging" based on the star count thresholds you defined.

Seeing a NameError or IndentationError?

• If you see NameError: name 'transform' is not defined, make sure the transform() function is defined above the if __name__ block, not inside it.
• If you see an IndentationError, check that the table-printing code (the print statements and return cleaned) is aligned with the for loop, not indented inside it.
• If you see a KeyError, double check that the dictionary keys in your record match the GitHub API field names exactly (e.g. stargazers_count, not stars_count).

Still stuck? Help me debug my transform function. I'm getting an error when I run pipeline.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved pipeline.py before moving on.

ⓧ I'd like to double check the full code

Scroll up through your work and confirm these sections are present in order:

• The three imports at the top: requests, sqlite3, time.
• The extract() function from Step 2.
• The transform() function with field selection, missing value handling, popularity assignment, and the formatted table print.
• The __main__ block calling extract() then transform(), with no leftover print lines from the previous step.

Your pipeline now extracts and transforms data in one run. The messy raw JSON has been turned into clean, flat records with a new popularity tier. Next up, you will load these records into a real database so they persist after the script ends.

Load Data into SQLite

Your pipeline now extracts live data from GitHub and transforms it into clean, structured records. There's one problem: all that data exists only in memory.

When your script ends, everything vanishes. Loading your transformed data into a SQLite database makes it persistent and queryable. This is the "L" in ETL, and your first time working with a real database.

In this step, get ready to:

• Write a load() function to store transformed data in a SQLite database.
• Run the full pipeline end-to-end and verify data is persisted in pipeline.db.

Write the load() function

SQLite is a database engine built into Python. You don't need to install anything or run a separate server. You connect to a file, and Python handles the rest.

• In pipeline.py, add the first half of the load() function below the transform() function by copying this code:

def load(records):
    """Load transformed records into a SQLite database."""
    # Connect to the database file (creates it if it doesn't exist)
    conn = sqlite3.connect("pipeline.db")
    cur = conn.cursor()

    # Drop existing table for a clean reload each run
    cur.execute("DROP TABLE IF EXISTS github_repos")
    cur.execute("""
        CREATE TABLE IF NOT EXISTS github_repos (
            name TEXT,
            owner TEXT,
            stars INTEGER,
            forks INTEGER,
            language TEXT,
            description TEXT,
            url TEXT,
            created_at TEXT,
            popularity TEXT
        )
    """)

What does this code do?

• sqlite3.connect("pipeline.db") opens a connection to a database file. If the file doesn't exist yet, SQLite creates it automatically.
• DROP TABLE IF EXISTS deletes the old table before recreating it. This makes your pipeline idempotent, meaning you can re-run it without duplicating data.
• CREATE TABLE IF NOT EXISTS defines the table structure with columns matching your transformed data. TEXT and INTEGER are SQL data types for strings and whole numbers.

• Save pipeline.py.

The function isn't complete yet. The next chunk adds the code that inserts your records into the table.

• Still inside the load() function, add this code directly below the CREATE TABLE statement:

    # Insert all records using parameterized queries
    cur.executemany("""
        INSERT INTO github_repos VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
    """, [
        (
            r["name"], r["owner"], r["stars"], r["forks"],
            r["language"], r["description"], r["url"],
            r["created_at"], r["popularity"]
        )
        for r in records
    ])

    # Save changes to the database
    conn.commit()

    # Verify the load by counting rows
    count = cur.execute("SELECT COUNT(*) FROM github_repos").fetchone()[0]
    print(f"\nLoaded {count} records into github_repos table")

    conn.close()

What does this code do?

• executemany() inserts all records in a single batch operation. The ? placeholders are parameterized queries. They let SQLite handle value escaping safely, preventing SQL injection attacks.
• conn.commit() saves all the inserted rows to disk. Without it, your changes would be lost when the connection closes.
• The SELECT COUNT(*) query counts rows in the table to confirm the load worked. This is your first SQL query.

• Save pipeline.py.

The load() function is ready. The next substep connects it to your pipeline so you can run it.

Why drop and recreate the table every time?

This pattern is called a "full refresh." Every time you run the pipeline, it replaces the old data with a fresh pull from GitHub. This makes the pipeline idempotent. You can run it ten times and always get a clean result with no duplicates.

Update the main block to run the full pipeline

Your __main__ block currently calls extract() and transform(). You need to add the third stage.

• In pipeline.py, scroll down to the if __name__ == "__main__": block at the bottom of your file.
• Find this line:

    cleaned_data = transform(raw_data)

• Add the following line directly below it, so the block now looks like this:

if __name__ == "__main__":
    raw_data = extract()
    cleaned_data = transform(raw_data)
    load(cleaned_data)

• Save pipeline.py.

Run the full pipeline and verify the database

Your pipeline now has all three ETL stages wired together. Before you run it, think about what you expect to see. Will the script print a count of loaded records? Will a new file appear in your project folder?

• Run the full pipeline by executing this command:

python3 pipeline.py

You should see the familiar extraction and transformation output, followed by a new line at the bottom:

Loaded 30 records into github_repos table

What should I see?

The script extracts 30 repos, prints the transformation table, and then prints the loaded count. The count comes from the SELECT COUNT(*) query inside your load() function. It proves the data made it into the database.

• Confirm the database file was created by running:

ls

You should see pipeline.db listed alongside pipeline.py. That file is your SQLite database. Your transformed data is now stored on disk and will survive after the script ends.

Not seeing the loaded message or the database file?

• Check that load(cleaned_data) is inside the if __name__ == "__main__": block and indented at the same level as the lines above it.
• Make sure your load() function definition is at the same indentation level as extract() and transform(). It should not be nested inside another function.
• Verify the nine ? placeholders in your INSERT statement match the nine values in the tuple. A mismatch causes a binding error.

Still stuck? Ask for help.

✔️ Awesome, I've got everything!

Great. Double check you've saved pipeline.py and that pipeline.db appears in your project folder.

ⓧ I'd like to double check the full code

Your pipeline.py is now longer than 30 lines. Scroll through your file and confirm these sections are present, in order:

• The three imports at the top: import requests, import sqlite3, import time.
• The extract() function that calls the GitHub API and returns repos.
• The transform() function that cleans records and adds the popularity column.
• The load() function that connects to pipeline.db, creates the github_repos table, inserts records with executemany(), and prints the loaded count.
• The if __name__ == "__main__": block calling extract(), transform(), then load().

Your data is now safely stored in a real database. Next up, you'll query it with SQL to answer questions about the repositories you collected.

Query Your Data with SQL

Your pipeline now extracts live data from GitHub, transforms it, and loads it into a SQLite database. That database is full of structured records, but right now it just sits there. You have no way to ask it questions or pull out insights.

SQL is the language every data engineer uses daily to query databases. In this step, you will write your first SQL queries to answer real questions about the data your pipeline collected.

In this step, get ready to:

• Write SQL queries to find top repositories, popularity distribution, and language breakdown.
• Run the complete ETL pipeline end-to-end and review the query results.

Write your first SQL query to find top repositories

SQL lets you ask precise questions of a database. Instead of looping through Python lists, you write a query that describes what you want and the database figures out how to get it.

• In pipeline.py, add the following query_data() function below the load() function:

def query_data():
    """Run SQL queries to analyze the loaded data."""
    conn = sqlite3.connect("pipeline.db")
    cur = conn.cursor()

    # Query 1: Find the top 5 repos by star count
    print("\n--- Top 5 Repositories by Stars ---")
    print(f"{'Name':<30} {'Owner':<20} {'Stars':<10}")
    print("-" * 60)
    rows = cur.execute(
        "SELECT name, owner, stars FROM github_repos ORDER BY stars DESC LIMIT 5"
    ).fetchall()
    for row in rows:
        print(f"{row[0]:<30} {row[1]:<20} {row[2]:<10}")

    conn.close()

What does this SQL query do?

This function connects to pipeline.db and runs a SQL query against the data your pipeline loaded.

• SELECT name, owner, stars tells the database which columns to return.
• ORDER BY stars DESC sorts results from highest to lowest star count.
• LIMIT 5 caps the output at 5 rows so you get only the top entries.
• fetchall() pulls every matching row into a Python list of tuples.

Now you need to call query_data() from your __main__ block so it runs after the data is loaded.

• Scroll down to the if __name__ == "__main__": block at the bottom of your file.
• Find this line:

    load(cleaned_data)

• Add query_data() on a new line directly below it, so the block now looks like this:

if __name__ == "__main__":
    raw_data = extract()
    cleaned_data = transform(raw_data)
    load(cleaned_data)
    query_data()

• Save pipeline.py.
• Run your pipeline to see the first query in action:

python3 pipeline.py

• After the extraction and transformation output you have seen before, you should now see a new table titled --- Top 5 Repositories by Stars --- listing five repository names, owners, and star counts.

Not seeing the query results?

• Make sure query_data() is called inside the if __name__ == "__main__": block, not outside it. It must be indented to the same level as the other function calls.
• Check that the query_data() function is defined above the __main__ block, not inside another function.
• help me debug why my query_data function is not printing results

Add queries for popularity and language breakdown

Your first query answered "which repos have the most stars?" Now you will add two more queries that use GROUP BY to count records by category. Grouping is how you spot patterns across your entire dataset instead of looking at individual rows.

• In pipeline.py, find the conn.close() line at the end of your query_data() function.
• Add the following two queries directly above conn.close():

    # Query 2: Count repos in each popularity tier
    print("\n--- Repos by Popularity Tier ---")
    print(f"{'Popularity':<15} {'Count':<10}")
    print("-" * 25)
    rows = cur.execute(
        "SELECT popularity, COUNT(*) as count FROM github_repos GROUP BY popularity"
    ).fetchall()
    for row in rows:
        print(f"{row[0]:<15} {row[1]:<10}")

    # Query 3: Find the most common languages
    print("\n--- Top 5 Languages ---")
    print(f"{'Language':<20} {'Count':<10}")
    print("-" * 30)
    rows = cur.execute(
        "SELECT language, COUNT(*) as count FROM github_repos GROUP BY language ORDER BY count DESC LIMIT 5"
    ).fetchall()
    for row in rows:
        print(f"{row[0]:<20} {row[1]:<10}")

What do these queries do?

• Query 2 uses GROUP BY popularity to bucket all 30 repos into the tiers you created in Step 3 (viral, popular, emerging). COUNT(*) counts how many repos fall into each bucket.
• Query 3 groups by language instead of popularity, then sorts by count to show which programming languages appear most often in your dataset.

• Save pipeline.py.

Run the complete pipeline and review your results

Your pipeline now has all four stages wired together: extract, transform, load, and query. Before you run it, take a moment to predict what you will see.

You have 30 records in the database. Each has a popularity tier and a language. How many popularity tiers do you expect to see? Will every record share the same language?

• Run the full pipeline by running this command:

python3 pipeline.py

After the extraction, transformation, and loading output, you should see three query result tables printed to the terminal.

• Top 5 Repositories by Stars shows five rows with the highest-starred repos, their owners, and star counts.
• Repos by Popularity Tier shows how many repos fall into each tier (viral, popular, emerging).
• Top 5 Languages shows which programming languages appear most frequently across the 30 repos.

Why does Python dominate the language list?

Your API query filtered for language:python in Step 2, so most repos will list Python as their language. Some may show a different primary language if GitHub classifies them differently based on file contents.

Seeing an error about the github_repos table?

• Make sure you saved pipeline.py before running it. Unsaved changes will not take effect.
• Check that pipeline.db exists in your project directory. If you deleted it, the pipeline will recreate it when you run the script.
• Confirm the two new queries are indented inside the query_data() function and placed above conn.close(), not below it.
• help me debug my SQL query errors

✔️ Awesome, I've got everything!

Great. Make sure you have saved pipeline.py and that all three query tables printed in the terminal.

ⓧ I'd like to double check the full code

Your pipeline.py is now too long to show in one block. Scroll up and confirm you have each of these sections present, in this order:

• import requests, import sqlite3, and import time at the top.
• The extract() function that calls the GitHub API.
• The transform() function that cleans data and adds the popularity column.
• The load() function that inserts records into SQLite.
• The query_data() function with all three queries (top repos, popularity tier, languages) followed by conn.close().
• The __main__ block calling extract(), transform(), load(), then query_data().

Secret mission

Add Data Quality Checks and Pipeline Audit Report

In production, data pipelines fail silently. Records arrive with missing fields, duplicates sneak in, and nobody notices until a dashboard breaks. Your mission: add a validation stage and an audit report that catches bad data and tracks pipeline health metrics.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs or cloud resources to worry about. Decide whether to keep your project files for future use or delete them entirely.

Resources you used:

• Project folder on your Desktop containing pipeline.py and pipeline.db.
• requests Python library (installed via pip).

Keep everything running

No action needed. Choose this if you want to keep experimenting with your pipeline.

• Your project folder stays on your Desktop. You can re-run the pipeline anytime to pull fresh data from GitHub.
• The pipeline.db file refreshes each time you run the script, so your data stays up to date automatically.
• The requests library remains installed on your system. Other Python projects you build may use it too.

Pause - I'll come back to this later

There are no running processes or servers to stop. Your project is already paused by default.

• Close pipeline.py in your code editor if you have it open.
• Your project folder and pipeline.db will be right where you left them when you come back.

Delete - I don't want to use this again

Remove all project files from your machine. There are no cloud resources or accounts to clean up.

• Delete the project folder from your Desktop by running these commands:

cd ~/Desktop
rm -rf [[PROJECT_FOLDER="your project folder name"]]

• This removes pipeline.py, pipeline.db, and everything else in the folder.
• Optionally, uninstall the requests library by running:

pip uninstall requests

Other projects may use requests

The requests library is one of the most widely used Python packages. If you have other Python projects on your machine, they may depend on it. Skip the uninstall if you are unsure.

Nice Work!

Nice Work!

Nice work! You just built a complete ETL pipeline in Python that pulls live data from an API, cleans it up, and stores it in a real database you can query.

You've learned how to:

• Built a complete ETL pipeline that extracts live repository data from the GitHub REST API, transforms messy JSON into clean records with a calculated popularity tier, and loads everything into a persistent database.
• Loaded structured data into a SQLite database using parameterized queries and wrote your first SQL queries using SELECT, ORDER BY, GROUP BY, COUNT, and LIMIT to analyze the data you collected.
• Ran the full pipeline end-to-end with a single command, watching your script extract, transform, load, and query live data from pipeline.py.
• (Secret Mission) Added data quality validation and a pipeline audit report that tracks metrics across every stage of the pipeline.

Ready to quiz yourself?