Scrape GitHub Trending Data with Python

Scrape GitHub Trending, enrich with the REST API, and export clean JSON.

Introduction

30 Second Summary

Every developer checks what's trending on GitHub, but that page only tells part of the story. The real insights come when you pull in details the trending page doesn't show, like licenses, issue counts, and how recently a repo was updated.

In this project, you will build a Python data pipeline that scrapes GitHub's Trending page, enriches each repository with the GitHub REST API, and cleans everything into a structured JSON dataset using Pandas.

What You'll Build

A complete Python data pipeline that scrapes live repository data from GitHub Trending, enriches it with API calls, and transforms it into a clean JSON file ready for frontend consumption.

This is Part 1 of a two-part series. In Part 2, you will build an interactive frontend dashboard powered by this dataset.

By the end of this project, you'll have:

  • A working web scraper that extracts repository data from GitHub's Trending page using BeautifulSoup and enriches it with the GitHub REST API.
  • A Pandas-powered cleaning pipeline that validates, deduplicates, and transforms raw data into a structured format with derived fields like status badges and formatted star counts.
  • A production-ready data.json file with 11 typed fields per repository, pushed to GitHub and ready for Part 2.
  • A command-line filtering feature that lets users scrape trending repos by programming language and date range.

Are there any prerequisites?

You should be comfortable with basic Python concepts like variables, functions, loops, and dictionaries. Familiarity with the command line is also helpful. The project walks you through everything else, including virtual environment setup and package installation.

Set Up Your Development Environment

Every Python project benefits from an isolated environment. A virtual environment gives your project its own copy of Python and its own set of installed packages, completely separate from everything else on your machine.

In this step, you'll create the project folder structure, set up a virtual environment, and install the three libraries that power this project's scraper and data cleaner.

In this step, get ready to:
  • Create the project folder and a Python virtual environment.
  • Install requests, beautifulsoup4, and pandas.
  • Create the project files and .gitignore.
Create your project folder and virtual environment

You need a dedicated folder on your Desktop to keep the project organized. You'll also create a backend/ subfolder inside it. In Part 2 of this series, a frontend folder will sit alongside it.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type VS Code and press Enter to open it.
  • Open a new terminal by pressing Ctrl+` (backtick). You can also click Terminal in the top menu bar, then New Terminal.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and navigate into the backend directory by running these commands:
mkdir github-trending-dashboard
cd github-trending-dashboard
mkdir backend
cd backend
  • Confirm the folders were created by running:
ls

The output should be empty since the backend folder is brand new. That confirms you're in the right place.

A virtual environment is an isolated Python installation. Packages you install inside it stay separate from your system Python, preventing version conflicts between projects.

  • Create and activate a virtual environment by running these commands:

macOS

python3 -m venv venv
source venv/bin/activate

Windows

python -m venv venv
venv\Scripts\activate

What do these commands do?

  • python3 -m venv venv (or python -m venv venv on Windows) creates a new virtual environment in a folder called venv/.
  • source venv/bin/activate (or venv\Scripts\activate on Windows) switches your terminal session to use this isolated Python instead of your system Python.

Your terminal prompt should now show (venv) at the beginning. This confirms the virtual environment is active.

Don't see (venv) in your prompt?

  • On Windows, PowerShell may block the activation script due to its execution policy. Run Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned and try activating again.
  • Make sure you're inside the backend folder. The venv/ folder must be in the same directory where you run the activate command.
  • On macOS, if python3 is not found, install Python from python.org and try again.

Why can't I activate my virtual environment?

Now open the project folder in VS Code so you can see the file structure and edit files.

  • Click File in the VS Code menu bar.
  • Click Open Folder.
  • Navigate to your Desktop and select the github-trending-dashboard folder, then click Open (macOS) or Select Folder (Windows).

See a trust prompt?

VS Code may ask "Do you trust the authors of the files in this folder?" Click Yes, I trust the authors to continue.

VS Code reloads with your project folder. The Explorer sidebar on the left now shows the backend/ folder.

  • Open a new terminal by pressing Ctrl+` (backtick).
  • Navigate into the backend folder and reactivate your virtual environment by running these commands:

macOS

cd backend
source venv/bin/activate

Windows

cd backend
venv\Scripts\activate

Confirm you see (venv) in your terminal prompt before continuing.

Install dependencies

This project uses three Python libraries. Requests handles HTTP calls to fetch web pages and API data. BeautifulSoup parses HTML so you can extract specific elements. Pandas provides DataFrames for cleaning and transforming data.

A requirements.txt file lists every package your project depends on, pinned to exact versions. This lets anyone recreate your environment with a single command.

  • In the Explorer sidebar on the left, right-click the backend folder.
  • Click New File.
  • Type requirements.txt and press Enter.
  • Add the following content to the file:
requests==2.33.1
beautifulsoup4==4.14.3
pandas==3.0.2

What does pinning versions mean?

The == syntax locks each package to an exact version. This ensures that anyone who installs from this file gets the same library behavior, even months from now.

  • Save the file by pressing Cmd+S (macOS) or Ctrl+S (Windows).
  • Install all three packages by running this command in the terminal:
pip install -r requirements.txt

You should see pip download and install each package along with its dependencies. The output ends with Successfully installed followed by a list of package names.

  • Verify that all three packages are importable by running this command:

macOS

python3 -c "import requests; import bs4; import pandas; print('All packages installed successfully')"

Windows

python -c "import requests; import bs4; import pandas; print('All packages installed successfully')"

You should see All packages installed successfully printed in the terminal.

Seeing a ModuleNotFoundError?

  • Make sure your virtual environment is active. Your terminal prompt should start with (venv). If it does not, re-run the activate command from the previous substep.
  • The import name for BeautifulSoup is bs4, not beautifulsoup4. The package name and import name are different.
  • Try running pip install -r requirements.txt again and check for any error messages during installation.

Help me fix my package installation error.

Create project files and .gitignore

Your project needs two Python files: scraper.py for fetching and parsing data, and cleaner.py for transforming raw data into a clean dataset. You'll also create a .gitignore file to keep unnecessary files out of version control.

  • In the Explorer sidebar, right-click the backend folder.
  • Click New File.
  • Type scraper.py and press Enter.
  • Repeat the process to create a second file called cleaner.py.

Both files are empty for now. You'll fill them in during the next steps.

What is a .gitignore file?

A .gitignore file tells Git which files and folders to ignore when tracking changes. The virtual environment folder, compiled Python files, and intermediate data files don't belong in your repository.

  • Right-click the backend folder again and click New File.
  • Type .gitignore and press Enter.
  • Add the following content to the file:
venv/
__pycache__/
*.pyc
raw_data.json

What does each line do?

  • venv/ excludes the virtual environment folder. It's large and anyone can recreate it from requirements.txt.
  • __pycache__/ and *.pyc exclude compiled Python bytecode files that Python generates automatically.
  • raw_data.json excludes the intermediate scraped data. Only the cleaned data.json will be committed to the repository.
  • Save .gitignore.
  • Verify the project structure by running this command in the terminal:
ls

You should see cleaner.py, requirements.txt, scraper.py, and venv in the output. The .gitignore file is hidden by default. You can confirm it exists in the VS Code Explorer sidebar.

Missing a file?

  • Make sure your terminal is inside the backend folder. Run pwd (macOS) or cd (Windows) to check your current directory.
  • If you created a file in the wrong folder, drag it into the backend folder using the VS Code Explorer sidebar.

Help me check my project folder structure.

Your final project structure should look like this:

github-trending-dashboard/
  backend/
    venv/
    scraper.py
    cleaner.py
    requirements.txt
    .gitignore

Your development environment is ready. Next up, you'll open GitHub's Trending page and inspect its HTML to plan your scraping strategy.

Inspect the GitHub Trending Page

Your development environment is ready with a virtual environment, installed packages, and an empty scraper.py waiting for code.

Before writing any scraping logic, you need to understand the data source. In this step, you will open GitHub's Trending page in your browser, use Developer Tools to map out the HTML structure, and set up the foundational code in scraper.py.

In this step, get ready to:
  • Inspect the GitHub Trending page HTML with browser Developer Tools.
  • Identify which data fields require enrichment from the GitHub REST API.
  • Add imports and configuration constants to scraper.py.
Explore the page structure with browser Developer Tools

Every web scraping project starts the same way: examining the target page to find the HTML elements that contain the data you want.

  • Open your browser and navigate to https://github.com/trending.
  • Right-click on any repository name in the list.
  • Select Inspect from the context menu to open Developer Tools.

The Elements panel highlights the HTML element you clicked on. Each trending repo is wrapped in an article element with the class Box-row.

  • Click the triangle next to an article.Box-row element to expand it.
  • Explore the nested elements inside. You should see child elements for the repo name, description, programming language, stars count, and forks count.

What are the key CSS selectors?

Here are the CSS selectors that map to each data field inside an article.Box-row:

  • Repo name and URL: h2 > a tag. The href attribute gives the path like /facebook/react.
  • Description: p tag with class col-9.
  • Language: span with attribute itemprop="programmingLanguage".
  • Stars: a tag whose href contains /stargazers.
  • Forks: a tag whose href contains /network/members.
  • Stars today: span with classes d-inline-block float-sm-right.
Understand what data the trending page provides and what it does not

The trending page gives you six fields per repo: name, description, language, stars, forks, and stars today. That is a solid starting point, but your dashboard needs more.

Fields like open issues count, license type, and last updated time are not on this page at all. This is a common real-world pattern. A single data source rarely has everything you need.

Why combine scraping with API enrichment?

The solution is to enrich scraped data with the GitHub REST API. The endpoint GET /repos/{owner}/{repo} returns open_issues_count, license, and updated_at for any public repo.

The API works without authentication for up to 60 requests per hour. Since the trending page shows roughly 25 repos, that fits comfortably within the limit. No GitHub token or account setup required.

Set up scraper.py with imports and constants

Now that you know the page structure, it is time to start coding. The first thing scraper.py needs is the libraries it depends on and the configuration values it will use throughout the script.

  • Open scraper.py in VS Code by clicking on it in the Explorer sidebar.
  • Add the following imports and constants to the file:
import requests
from bs4 import BeautifulSoup
import json
import time

# Target URLs for scraping and API enrichment
TRENDING_URL = "https://github.com/trending"
API_BASE_URL = "https://api.github.com/repos"

# Browser-like headers so GitHub does not reject the request
HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
}

# GitHub's recommended header format for REST API calls
API_HEADERS = {
    "Accept": "application/vnd.github+json",
    "User-Agent": "GitHub-Trending-Scraper",
}

# --- Paste the fetch_trending_page function below this line ---
# ---------------------------------------------------------------

What does this code do?

  • requests handles HTTP requests to fetch web pages and call APIs.
  • BeautifulSoup from BeautifulSoup parses HTML into a navigable tree you can search with CSS selectors.
  • json and time are standard library modules for saving data and pausing between API calls.
  • HEADERS mimics a real browser request so GitHub does not reject the scraper.
  • API_HEADERS follows GitHub's recommended format for REST API calls.
  • Save scraper.py.
  • Verify the imports work by running this command in your terminal:
python scraper.py

The script defines imports and constants, so it exits silently with no output. If you see no error messages, everything is set up correctly.

Seeing a ModuleNotFoundError?

  • Check that your virtual environment is active. You should see (venv) at the start of your terminal prompt.
  • Make sure you installed packages inside the virtual environment. If not, reactivate it and run pip install -r requirements.txt again.
  • Confirm you are running the command from the backend folder, not the project root.

Why is my scraper.py throwing a ModuleNotFoundError when I run it?

Awesome, I've got everything!

Double check you have saved scraper.py before moving on.

I'd like to double check the full code

import requests
from bs4 import BeautifulSoup
import json
import time

TRENDING_URL = "https://github.com/trending"
API_BASE_URL = "https://api.github.com/repos"

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
}

API_HEADERS = {
    "Accept": "application/vnd.github+json",
    "User-Agent": "GitHub-Trending-Scraper",
}

Your scraper's foundation is in place. Next up, you will write the functions that fetch the trending page, parse each repository from the HTML, and enrich the data with the GitHub API.

Build the Web Scraper

In the previous step, you mapped out the HTML structure of GitHub's Trending page. You identified the exact CSS selectors for each data field and discovered that several fields your dashboard needs are missing from the page entirely.

Time to turn those selectors into working Python code. In this step, you'll build three functions in scraper.py that fetch the page, parse the HTML, and enrich each repo through the GitHub REST API.

In this step, get ready to:
  • Write fetch and parse functions that extract repository data from the trending page HTML.
  • Add an API enrichment function that fills in missing fields like issues, license, and last updated time.
  • Run the complete scraper and produce raw_data.json.
Write the fetch and parse functions

Your scraper needs two foundational functions. The first downloads the raw HTML from GitHub's Trending page. The second walks through that HTML with BeautifulSoup, pulling out each repository's data using the selectors you identified in Step 2.

  • In scraper.py, add the fetch_trending_page function below the API_HEADERS dictionary by copying this code:
def fetch_trending_page():
    """Fetch the GitHub Trending page HTML."""
    # Send a GET request with browser-like headers
    response = requests.get(TRENDING_URL, headers=HEADERS)
    # Stop execution if the request failed
    response.raise_for_status()
    return response.text

# --- Paste the parse_trending_repos function below this line ---
# ---------------------------------------------------------------

What does this code do?

  • requests.get() sends an HTTP GET request to the trending page URL, passing the browser-like headers you defined earlier.
  • raise_for_status() throws an error if the response code is anything other than 200 (success). This prevents the scraper from silently continuing with bad data.

You won't see a change yet. The next function will call this one to get the HTML it needs to parse.

  • Below fetch_trending_page, add the parse_trending_repos function by copying this code:
def parse_trending_repos(html):
    """Parse trending repositories from HTML."""
    soup = BeautifulSoup(html, "html.parser")
    repos = []

    # Loop through each trending repo on the page
    for article in soup.find_all("article", class_="Box-row"):
        # Extract the repo name and URL from the title link
        title_tag = article.find("h2").find("a")
        href = title_tag["href"].strip()
        name = href.strip("/")
        url = f"https://github.com{href}"

        # Get the description, defaulting to empty string
        desc_tag = article.find("p", class_="col-9")
        description = desc_tag.get_text(strip=True) if desc_tag else ""

        # Get the programming language
        lang_tag = article.find("span", attrs={"itemprop": "programmingLanguage"})
        language = lang_tag.get_text(strip=True) if lang_tag else ""

        # Extract total star count from the stargazers link
        stars_tag = article.find("a", href=lambda h: h and "/stargazers" in h)
        stars_text = stars_tag.get_text(strip=True) if stars_tag else "0"
        stars = int(stars_text.replace(",", ""))

        # Extract fork count from the network/members link
        forks_tag = article.find("a", href=lambda h: h and "/network/members" in h)
        forks_text = forks_tag.get_text(strip=True) if forks_tag else "0"
        forks = int(forks_text.replace(",", ""))

        # Extract "stars today" count from the daily stats span
        today_tag = article.find("span", class_="d-inline-block float-sm-right")
        stars_today = 0
        if today_tag:
            today_text = today_tag.get_text(strip=True)
            # Text looks like "7,429 stars today", grab just the number
            stars_today = int(today_text.split()[0].replace(",", ""))

        repos.append({
            "name": name,
            "description": description,
            "language": language,
            "stars": stars,
            "forks": forks,
            "stars_today": stars_today,
            "url": url,
        })

    return repos

# --- Paste the enrich_with_api function below this line ---
# ----------------------------------------------------------

What does this code do?

  • BeautifulSoup(html, "html.parser") turns the raw HTML string into a navigable tree of elements you can search through.
  • find_all("article", class_="Box-row") grabs every trending repo container on the page. Each one holds a repo's name, description, language, stars, and forks.
  • The lambda filter href=lambda h: h and "/stargazers" in h finds the first link whose URL contains "/stargazers". This is how the star count link is identified without relying on fragile class names.
  • For "stars today", the text looks like "7,429 stars today". Splitting on whitespace and taking index 0 isolates just the number.

You won't see a change yet. These two functions will be called by the main function you'll add in Substep 3.

Seeing a syntax error?

  • Check that every opening parenthesis, bracket, and quote has a matching closing one. The lambda lines and f-strings are common places for mismatches.
  • Make sure the repos.append({...}) block is indented inside the for loop, not at the same level as it.
  • Verify that return repos is at the function level (one indent), not inside the loop (two indents).

Help me fix a syntax error in my parse_trending_repos function.

Add the GitHub API enrichment function

The trending page only gives you six fields per repo. Fields like open issues count, license, and last updated time are not on the page at all.

The GitHub REST API fills this gap. The endpoint GET /repos/{owner}/{repo} returns detailed metadata for any public repository, and it works without authentication for up to 60 requests per hour.

  • Below parse_trending_repos, add the enrich_with_api function by copying this code:
def enrich_with_api(repos):
    """Enrich repository data using the GitHub REST API."""
    enriched = []
    total = len(repos)

    for i, repo in enumerate(repos, 1):
        print(f"  Enriching {i}/{total}: {repo['name']}")
        api_url = f"{API_BASE_URL}/{repo['name']}"

        try:
            response = requests.get(api_url, headers=API_HEADERS)
            if response.status_code == 200:
                data = response.json()
                # Pull fields that aren't on the trending page
                repo["issues"] = data.get("open_issues_count", 0)
                license_info = data.get("license")
                repo["license"] = license_info.get("spdx_id", "") if license_info else ""
                repo["created_at"] = data.get("created_at", "")
                repo["updated_at"] = data.get("updated_at", "")
            else:
                # Set defaults if the API call failed
                repo["issues"] = 0
                repo["license"] = ""
                repo["created_at"] = ""
                repo["updated_at"] = ""
                print(f"    API returned {response.status_code} for {repo['name']}")
        except requests.RequestException as e:
            # Handle network errors gracefully
            repo["issues"] = 0
            repo["license"] = ""
            repo["created_at"] = ""
            repo["updated_at"] = ""
            print(f"    Error: {e}")

        enriched.append(repo)
        # Polite pause between API calls to avoid rate limiting
        time.sleep(1)

    return enriched

# --- Paste the main function and entry point below this line ---
# ---------------------------------------------------------------

What does this code do?

  • For each repo, the function calls GET /repos/{owner}/{repo} and extracts four new fields: open_issues_count, license, created_at, and updated_at.
  • The try/except block handles network failures gracefully. If an API call fails, the repo still gets added with default empty values instead of crashing the entire scraper.
  • time.sleep(1) pauses for one second between API calls. With roughly 25 repos, the enrichment takes about 25 seconds and stays well within the 60 requests/hour unauthenticated limit.

You won't see a change yet. The main function in the next substep will call all three functions together.

Worried about the rate limit?

The unauthenticated GitHub API limit is 60 requests per hour. If you run the scraper multiple times in quick succession, you may hit this limit and see 403 status codes in your terminal output. Wait an hour for the limit to reset, or run the scraper only once per test.

Why am I getting 403 errors from the GitHub API?

💡 Why scraping plus API instead of just one?

The trending page gives you data that the API does not: the curated ranking order and the "stars today" count. The API gives you fields the trending page does not: open issues, license, and timestamps.

Combining both is a common real-world pattern. A single data source rarely has everything you need. Learning to merge multiple sources into one clean dataset is one of the most transferable data engineering skills.

Add the main function and test the scraper

The three core functions are ready. Now you need a main() function to orchestrate them: fetch the page, parse repos, enrich with the API, and save the results.

  • Below enrich_with_api, add the main function and the entry point guard by copying this code:
def main():
    print("Fetching GitHub Trending page...")
    html = fetch_trending_page()

    print("Parsing trending repositories...")
    repos = parse_trending_repos(html)
    print(f"Found {len(repos)} trending repositories.\n")

    print("Enriching with GitHub API data...")
    repos = enrich_with_api(repos)

    # Save the enriched data to a JSON file
    with open("raw_data.json", "w", encoding="utf-8") as f:
        json.dump(repos, f, indent=2, ensure_ascii=False)

    print(f"\nRaw data saved to raw_data.json")


if __name__ == "__main__":
    main()

What does this code do?

  • main() calls the three functions in order: fetch, parse, enrich. Progress messages print to the terminal so you can see what's happening.
  • json.dump() writes the enriched data to raw_data.json with indent=2 for human-readable formatting.
  • The if __name__ == "__main__" guard ensures main() only runs when you execute the script directly, not when another file imports it.
  • Save scraper.py.

Awesome, I've got everything!

Double check you've saved scraper.py before moving on.

I'd like to double check the full code

Here is the complete scraper.py file. Make sure your version matches this exactly.

import requests
from bs4 import BeautifulSoup
import json
import time

TRENDING_URL = "https://github.com/trending"
API_BASE_URL = "https://api.github.com/repos"

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
}

API_HEADERS = {
    "Accept": "application/vnd.github+json",
    "User-Agent": "GitHub-Trending-Scraper",
}


def fetch_trending_page():
    """Fetch the GitHub Trending page HTML."""
    response = requests.get(TRENDING_URL, headers=HEADERS)
    response.raise_for_status()
    return response.text


def parse_trending_repos(html):
    """Parse trending repositories from HTML."""
    soup = BeautifulSoup(html, "html.parser")
    repos = []

    for article in soup.find_all("article", class_="Box-row"):
        title_tag = article.find("h2").find("a")
        href = title_tag["href"].strip()
        name = href.strip("/")
        url = f"https://github.com{href}"

        desc_tag = article.find("p", class_="col-9")
        description = desc_tag.get_text(strip=True) if desc_tag else ""

        lang_tag = article.find("span", attrs={"itemprop": "programmingLanguage"})
        language = lang_tag.get_text(strip=True) if lang_tag else ""

        stars_tag = article.find("a", href=lambda h: h and "/stargazers" in h)
        stars_text = stars_tag.get_text(strip=True) if stars_tag else "0"
        stars = int(stars_text.replace(",", ""))

        forks_tag = article.find("a", href=lambda h: h and "/network/members" in h)
        forks_text = forks_tag.get_text(strip=True) if forks_tag else "0"
        forks = int(forks_text.replace(",", ""))

        today_tag = article.find("span", class_="d-inline-block float-sm-right")
        stars_today = 0
        if today_tag:
            today_text = today_tag.get_text(strip=True)
            stars_today = int(today_text.split()[0].replace(",", ""))

        repos.append({
            "name": name,
            "description": description,
            "language": language,
            "stars": stars,
            "forks": forks,
            "stars_today": stars_today,
            "url": url,
        })

    return repos


def enrich_with_api(repos):
    """Enrich repository data using the GitHub REST API."""
    enriched = []
    total = len(repos)

    for i, repo in enumerate(repos, 1):
        print(f"  Enriching {i}/{total}: {repo['name']}")
        api_url = f"{API_BASE_URL}/{repo['name']}"

        try:
            response = requests.get(api_url, headers=API_HEADERS)
            if response.status_code == 200:
                data = response.json()
                repo["issues"] = data.get("open_issues_count", 0)
                license_info = data.get("license")
                repo["license"] = license_info.get("spdx_id", "") if license_info else ""
                repo["created_at"] = data.get("created_at", "")
                repo["updated_at"] = data.get("updated_at", "")
            else:
                repo["issues"] = 0
                repo["license"] = ""
                repo["created_at"] = ""
                repo["updated_at"] = ""
                print(f"    API returned {response.status_code} for {repo['name']}")
        except requests.RequestException as e:
            repo["issues"] = 0
            repo["license"] = ""
            repo["created_at"] = ""
            repo["updated_at"] = ""
            print(f"    Error: {e}")

        enriched.append(repo)
        time.sleep(1)

    return enriched


def main():
    print("Fetching GitHub Trending page...")
    html = fetch_trending_page()

    print("Parsing trending repositories...")
    repos = parse_trending_repos(html)
    print(f"Found {len(repos)} trending repositories.\n")

    print("Enriching with GitHub API data...")
    repos = enrich_with_api(repos)

    with open("raw_data.json", "w", encoding="utf-8") as f:
        json.dump(repos, f, indent=2, ensure_ascii=False)

    print(f"\nRaw data saved to raw_data.json")


if __name__ == "__main__":
    main()

Now it's time to run the scraper and see it in action.

  • Make sure your terminal is in the backend folder and your virtual environment is active (you should see (venv) in the prompt).
  • Run the scraper by running this command:
python scraper.py

What should I see?

The terminal prints progress messages as the scraper runs. First you'll see "Fetching GitHub Trending page..." and "Parsing trending repositories...". Then the enrichment phase prints each repo as it calls the API, like Enriching 1/25: github/trending.

The enrichment takes about 25 seconds because of the polite 1-second pause between API calls. When it finishes, you'll see "Raw data saved to raw_data.json".

  • Open raw_data.json in VS Code.
  • Verify that the file contains an array of roughly 25 repository objects.
  • Check that each entry has 11 fields: name, description, language, stars, forks, stars_today, url, issues, license, created_at, and updated_at.

Scraper not working?

  • If you see ModuleNotFoundError: No module named 'requests', your virtual environment is not active. Run venv\Scripts\activate and try again.
  • If you see ConnectionError, check your internet connection. The scraper needs to reach github.com and api.github.com.
  • If the enrichment shows 403 status codes for every repo, you've hit the API rate limit. Wait an hour for it to reset, then run the scraper again.
  • If raw_data.json shows 0 repositories, GitHub may have changed their HTML structure. Double check that the selectors (like article.Box-row) still match the page by inspecting it in your browser.

Help me debug my GitHub Trending scraper.

Your scraper is pulling live data from GitHub and enriching it with the API. Next up, you'll clean and transform this raw data into a polished, structured JSON dataset using Pandas.

Clean and Transform with Pandas

Your scraper successfully pulled live data from GitHub and enriched it with the REST API. You now have a raw_data.json file with about 25 repositories and 11 fields each.

Raw data is rarely ready to use directly. Fields might be missing, duplicates can slip in, and numbers might arrive as strings. In this step, you will build a cleaning pipeline with Pandas that validates, deduplicates, and transforms this raw data into a structured dataset.

In this step, get ready to:
  • Load raw scraped data and build helper functions for formatting display values.
  • Build a cleaning pipeline that deduplicates, validates, and transforms the DataFrame.
  • Export the final dataset as a structured data.json with 11 fields per repository.
Create the data loading and formatting functions

The cleaner.py file is currently empty. You will build it up function by function, starting with imports, data loading, and two helper functions that convert raw values into human-readable display strings.

  • Open cleaner.py in VS Code and add the imports and data loading function by pasting this code:
import pandas as pd
import json
from datetime import datetime, timezone


def load_raw_data(filepath="raw_data.json"):
    """Load raw scraped data from JSON file."""
    with open(filepath, "r", encoding="utf-8") as f:
        return json.load(f)

# --- Paste the format_stars function below this line ---
# -------------------------------------------------------

What does this code do?

  • pandas is the data manipulation library you will use for cleaning and transforming.
  • datetime and timezone are needed later for converting ISO timestamps into relative time strings like "2h ago".
  • load_raw_data() reads the raw_data.json file your scraper produced and returns it as a Python list of dictionaries.

You will not see a change yet. The next chunk adds the formatting helper that uses these imports.

  • Add the star count formatting function below load_raw_data() in cleaner.py:
def format_stars(count):
    """Format star count for display (e.g., 231K, 1.2M)."""
    if count >= 1_000_000:
        return f"{count / 1_000_000:.1f}M"
    elif count >= 1_000:
        formatted = f"{count / 1_000:.1f}K"
        return formatted.replace(".0K", "K")
    return str(count)

# --- Paste the get_relative_time function below this line ---
# ------------------------------------------------------------

What does this code do?

This function converts raw star counts into readable display strings. A value like 428084 becomes "428.1K" and 1200000 becomes "1.2M". The .replace(".0K", "K") cleanup turns "12.0K" into a cleaner "12K".

  • Save cleaner.py.
  • Test the function by running this command in your terminal:
python -c "from cleaner import format_stars; print(format_stars(428084)); print(format_stars(1200000)); print(format_stars(750))"

You should see 428.1K, 1.2M, and 750 printed on separate lines.

Seeing a ModuleNotFoundError?

  • Make sure your terminal is in the backend folder where cleaner.py lives.
  • Check that your virtual environment is active. You should see (venv) at the start of your terminal prompt.

Help me fix this ModuleNotFoundError when importing from cleaner.py

  • Add the relative time function below format_stars() in cleaner.py:
def get_relative_time(updated_at):
    """Convert ISO timestamp to relative time string."""
    if not updated_at:
        return "Unknown"
    try:
        updated = datetime.fromisoformat(updated_at.replace("Z", "+00:00"))
        now = datetime.now(timezone.utc)
        diff = now - updated
        hours = int(diff.total_seconds() / 3600)
        if hours < 1:
            minutes = int(diff.total_seconds() / 60)
            return f"{minutes}m ago"
        elif hours < 24:
            return f"{hours}h ago"
        elif hours < 720:
            days = hours // 24
            return f"{days}d ago"
        else:
            months = hours // 720
            return f"{months}mo ago"
    except (ValueError, TypeError):
        return "Unknown"

# --- Paste the derive_status function below this line ---
# --------------------------------------------------------

What does this code do?

  • Takes an ISO timestamp like 2026-04-28T10:30:00Z and converts it into a human-friendly string like "2h ago" or "3d ago".
  • The .replace("Z", "+00:00") call normalizes the UTC timezone marker so Python's fromisoformat() can parse it correctly.
  • If the timestamp is empty or malformed, it safely returns "Unknown" instead of crashing.
  • Save cleaner.py.

You will not see a visible change yet. The next substep uses these helpers inside the main cleaning function.

Add the status badge and main cleaning functions

With the helper functions in place, you now need two more functions. One derives a status badge for each repo based on its metrics. The other is the main cleaning pipeline that processes the entire DataFrame.

  • Add the status badge function below get_relative_time() in cleaner.py:
def derive_status(row):
    """Derive status badge based on repository metrics."""
    if row["stars"] > 200_000:
        return "HOT"
    if row["created_at"]:
        try:
            created = datetime.fromisoformat(row["created_at"].replace("Z", "+00:00"))
            now = datetime.now(timezone.utc)
            hours_since_created = (now - created).total_seconds() / 3600
            if hours_since_created < 24:
                return "NEW"
        except (ValueError, TypeError):
            pass
    if row["stars_today"] > 500:
        return "RISING"
    return ""

# --- Paste the clean_data function below this line ---
# -----------------------------------------------------

What do the status badges mean?

  • "HOT" is for mega-popular repos with over 200,000 stars (think projects like public-apis or system-design-primer).
  • "NEW" catches repos created within the last 24 hours using the created_at timestamp from the API.
  • "RISING" flags repos gaining 500+ stars today, indicating strong daily momentum.
  • Repos that do not match any threshold get an empty string. Most trending repos will fall here.

You will not see a change yet. The next function ties everything together.

  • Add the main cleaning function below derive_status() in cleaner.py:
def clean_data(raw_data):
    """Clean and transform raw repository data."""
    df = pd.DataFrame(raw_data)

    # Remove duplicates by repository name
    df = df.drop_duplicates(subset="name", keep="first")

    # Handle missing values
    df["description"] = df["description"].fillna("No description available")
    df["language"] = df["language"].fillna("Unknown")
    df["license"] = df["license"].fillna("")
    df["license"] = df["license"].replace("NOASSERTION", "")
    df["updated_at"] = df["updated_at"].fillna("")
    df["created_at"] = df["created_at"].fillna("")

    # Ensure numeric types
    df["stars"] = pd.to_numeric(df["stars"], errors="coerce").fillna(0).astype(int)
    df["forks"] = pd.to_numeric(df["forks"], errors="coerce").fillna(0).astype(int)
    df["issues"] = pd.to_numeric(df["issues"], errors="coerce").fillna(0).astype(int)
    df["stars_today"] = pd.to_numeric(df["stars_today"], errors="coerce").fillna(0).astype(int)

    # Add derived fields
    df["stars_formatted"] = df["stars"].apply(format_stars)
    df["last_updated"] = df["updated_at"].apply(get_relative_time)
    df["status"] = df.apply(derive_status, axis=1)

    return df

# --- Paste the export_to_json function below this line ---
# ---------------------------------------------------------

What does clean_data do?

This is the core cleaning pipeline. It processes the raw data in four phases:

  • Deduplication removes any repo that appears more than once, keeping only the first occurrence.
  • Missing value handling fills in sensible defaults. Repos without a description get "No description available". The NOASSERTION license value (returned when GitHub cannot identify the license) is replaced with an empty string.
  • Type coercion ensures all numeric columns are actual integers. The errors="coerce" parameter turns unparseable values into NaN, which fillna(0) then converts to zero.
  • Derived fields add three new columns by applying the helper functions: stars_formatted, last_updated, and status.
  • Save cleaner.py.

You will not see output yet. The next substep adds the export function and the entry point so you can run the cleaner.

Seeing a syntax error when saving?

  • Check that every opening parenthesis, bracket, and quote in clean_data() has a matching closing character.
  • Make sure the clean_data function is at the same indentation level as derive_status above it. Mixing tabs and spaces causes indentation errors.

Help me fix a syntax error in my clean_data function

Add the export function and test the cleaner

The last two functions handle exporting the cleaned DataFrame to JSON and orchestrating the full pipeline. Once these are in place, you can run the cleaner and produce your final dataset.

  • Add the export function below clean_data() in cleaner.py:
def export_to_json(df, filepath="data.json"):
    """Export cleaned DataFrame to JSON."""
    output_fields = [
        "name", "description", "stars", "stars_formatted",
        "forks", "issues", "language", "license",
        "last_updated", "status", "url"
    ]
    output_df = df[output_fields]
    records = output_df.to_dict(orient="records")

    with open(filepath, "w", encoding="utf-8") as f:
        json.dump(records, f, indent=2, ensure_ascii=False)

    print(f"Exported {len(records)} repositories to {filepath}")

# --- Paste the main function and entry point below this line ---
# ---------------------------------------------------------------

What does this code do?

  • output_fields defines exactly which 11 columns to include in the final JSON. Intermediate columns like created_at and updated_at are excluded because their values have been absorbed into the derived fields.
  • to_dict(orient="records") converts the DataFrame into a list of dictionaries, which is the standard format for JSON arrays.
  • ensure_ascii=False preserves non-English characters in repo descriptions.

You will not see output yet. The next chunk adds the entry point that runs everything.

  • Add the main function below export_to_json() in cleaner.py:
def main():
    print("Loading raw data...")
    raw_data = load_raw_data()
    print(f"Loaded {len(raw_data)} repositories.\n")

    print("Cleaning and transforming data...")
    df = clean_data(raw_data)

    print(f"\nDataset Summary:")
    print(f"  Total repositories: {len(df)}")
    print(f"  Languages: {df['language'].nunique()}")
    print(f"  Avg stars: {df['stars'].mean():,.0f}")
    print(f"  Status breakdown:")
    status_counts = df["status"].value_counts()
    for status, count in status_counts.items():
        label = status if status else "(none)"
        print(f"    {label}: {count}")

    print()
    export_to_json(df)


if __name__ == "__main__":
    main()

What does this code do?

  • main() orchestrates the full pipeline: load, clean, summarize, and export.
  • The summary printout shows total repos, unique languages, average star count, and how many repos received each status badge.
  • if __name__ == "__main__" ensures the pipeline only runs when you execute the file directly, not when importing functions from it.
  • Save cleaner.py.
  • Run the cleaner by executing this command in your terminal:
python cleaner.py

You should see output showing how many repositories were loaded, a dataset summary with total repos, languages, and average stars, and a final line confirming the export to data.json.

  • Open data.json in VS Code to verify the output.

Each repository entry should have exactly 11 fields: name, description, stars (integer), stars_formatted (display string), forks, issues, language, license, last_updated, status, and url.

Seeing a FileNotFoundError?

  • The cleaner expects raw_data.json to exist in the same folder. If you deleted it or are in a different directory, run python scraper.py again to regenerate it.
  • Make sure your terminal is in the backend folder, not the parent github-trending-dashboard folder.

Help me fix this FileNotFoundError in cleaner.py

💡 Why might no repos have the NEW status?

The "NEW" badge requires a repo to have been created within the last 24 hours. Trending repos are typically at least a few days old, so it is completely normal to see zero repos with this status.

Awesome, I've got everything!

Double check that you have saved cleaner.py and that data.json exists in your backend folder.

I'd like to double check the full code

Here is the complete cleaner.py file. Compare it against yours to spot any differences.

import pandas as pd
import json
from datetime import datetime, timezone


def load_raw_data(filepath="raw_data.json"):
    """Load raw scraped data from JSON file."""
    with open(filepath, "r", encoding="utf-8") as f:
        return json.load(f)


def format_stars(count):
    """Format star count for display (e.g., 231K, 1.2M)."""
    if count >= 1_000_000:
        return f"{count / 1_000_000:.1f}M"
    elif count >= 1_000:
        formatted = f"{count / 1_000:.1f}K"
        return formatted.replace(".0K", "K")
    return str(count)


def get_relative_time(updated_at):
    """Convert ISO timestamp to relative time string."""
    if not updated_at:
        return "Unknown"
    try:
        updated = datetime.fromisoformat(updated_at.replace("Z", "+00:00"))
        now = datetime.now(timezone.utc)
        diff = now - updated
        hours = int(diff.total_seconds() / 3600)
        if hours < 1:
            minutes = int(diff.total_seconds() / 60)
            return f"{minutes}m ago"
        elif hours < 24:
            return f"{hours}h ago"
        elif hours < 720:
            days = hours // 24
            return f"{days}d ago"
        else:
            months = hours // 720
            return f"{months}mo ago"
    except (ValueError, TypeError):
        return "Unknown"


def derive_status(row):
    """Derive status badge based on repository metrics."""
    if row["stars"] > 200_000:
        return "HOT"
    if row["created_at"]:
        try:
            created = datetime.fromisoformat(row["created_at"].replace("Z", "+00:00"))
            now = datetime.now(timezone.utc)
            hours_since_created = (now - created).total_seconds() / 3600
            if hours_since_created < 24:
                return "NEW"
        except (ValueError, TypeError):
            pass
    if row["stars_today"] > 500:
        return "RISING"
    return ""


def clean_data(raw_data):
    """Clean and transform raw repository data."""
    df = pd.DataFrame(raw_data)

    # Remove duplicates by repository name
    df = df.drop_duplicates(subset="name", keep="first")

    # Handle missing values
    df["description"] = df["description"].fillna("No description available")
    df["language"] = df["language"].fillna("Unknown")
    df["license"] = df["license"].fillna("")
    df["license"] = df["license"].replace("NOASSERTION", "")
    df["updated_at"] = df["updated_at"].fillna("")
    df["created_at"] = df["created_at"].fillna("")

    # Ensure numeric types
    df["stars"] = pd.to_numeric(df["stars"], errors="coerce").fillna(0).astype(int)
    df["forks"] = pd.to_numeric(df["forks"], errors="coerce").fillna(0).astype(int)
    df["issues"] = pd.to_numeric(df["issues"], errors="coerce").fillna(0).astype(int)
    df["stars_today"] = pd.to_numeric(df["stars_today"], errors="coerce").fillna(0).astype(int)

    # Add derived fields
    df["stars_formatted"] = df["stars"].apply(format_stars)
    df["last_updated"] = df["updated_at"].apply(get_relative_time)
    df["status"] = df.apply(derive_status, axis=1)

    return df


def export_to_json(df, filepath="data.json"):
    """Export cleaned DataFrame to JSON."""
    output_fields = [
        "name", "description", "stars", "stars_formatted",
        "forks", "issues", "language", "license",
        "last_updated", "status", "url"
    ]
    output_df = df[output_fields]
    records = output_df.to_dict(orient="records")

    with open(filepath, "w", encoding="utf-8") as f:
        json.dump(records, f, indent=2, ensure_ascii=False)

    print(f"Exported {len(records)} repositories to {filepath}")


def main():
    print("Loading raw data...")
    raw_data = load_raw_data()
    print(f"Loaded {len(raw_data)} repositories.\n")

    print("Cleaning and transforming data...")
    df = clean_data(raw_data)

    print(f"\nDataset Summary:")
    print(f"  Total repositories: {len(df)}")
    print(f"  Languages: {df['language'].nunique()}")
    print(f"  Avg stars: {df['stars'].mean():,.0f}")
    print(f"  Status breakdown:")
    status_counts = df["status"].value_counts()
    for status, count in status_counts.items():
        label = status if status else "(none)"
        print(f"    {label}: {count}")

    print()
    export_to_json(df)


if __name__ == "__main__":
    main()

Your cleaning pipeline is complete. You now have a production-ready data.json with 11 typed fields per repository. Next up, you will push everything to GitHub so the dataset is versioned and ready for Part 2.

Push to GitHub

Your data pipeline is complete. You have a scraper that fetches live trending data, an enrichment layer that pulls from the GitHub REST API, and a cleaner that transforms everything into a structured JSON dataset.

The last step is to push your project to GitHub so the code and dataset are version-controlled and accessible. In Part 2 of this series, the frontend dashboard will pull directly from this repository.

In this step, get ready to:
  • Initialize a Git repository at the project root.
  • Create a GitHub repository and push your code.
  • Verify the repository structure on GitHub.
Initialize the Git repository

Your terminal is currently in the backend/ folder from the previous step. Git needs to track the entire project, so the repository should be initialized one level up in the github-trending-dashboard/ folder.

  • Navigate to the project root by running this command:
cd ..
  • Initialize a Git repository and create your first commit by running these commands:
git init
git add .
git commit -m "Part 1: GitHub Trending scraper and data cleaner"

What do these commands do?

  • git init creates a new Git repository in the current folder.
  • git add . stages all files for the first commit. Files listed in .gitignore (like venv/ and raw_data.json) are automatically skipped.
  • git commit -m "..." saves a snapshot of all staged files with a descriptive message.
  • Verify your commit by running:
git log --oneline

You should see a single commit with the message Part 1: GitHub Trending scraper and data cleaner.

Seeing an error about user identity?

Git requires your name and email before you can commit. Run these two commands with your own details, then try the commit again:

git config --global user.name "Your Name"
git config --global user.email "your@email.com"

Help me fix my Git commit error.

Create a GitHub repository and push

You need a remote repository on GitHub to store your code online. Creating an empty repository on GitHub gives you a URL to push your local code to.

  • Open your browser and go to github.com.
  • Click the + icon in the top-right corner of the page.
  • Click New repository.
  • Enter github-trending-dashboard in the Repository name field.
  • Select Public for the visibility.
  • Leave all other options unchecked (no README, no .gitignore, no license).
  • Click Create repository.

Why no README or .gitignore on GitHub?

You already have a .gitignore in your local project. If GitHub adds one too, the histories will conflict when you try to push. Starting with an empty remote keeps things clean.

After creating the repository, GitHub shows a setup page with commands you can use. You will use the commands below instead.

  • Connect your local repository to GitHub and push your code by running these commands:
git remote add origin https://github.com/[[GITHUB_USERNAME="your GitHub username"]]/github-trending-dashboard.git
git branch -M main
git push -u origin main

What do these commands do?

  • git remote add origin links your local repo to the GitHub repository you just created.
  • git branch -M main renames your default branch to main.
  • git push -u origin main uploads all committed files to GitHub and sets main as the default tracking branch.

💡 GitHub may ask you to sign in

A browser window or terminal prompt may appear asking for your GitHub credentials. Sign in with your GitHub account. If you have two-factor authentication enabled, you may need to use a personal access token instead of your password.

After the push completes, your terminal should show a success message referencing the main branch.

Push failed or authentication error?

Make sure the repository name on GitHub matches exactly: github-trending-dashboard. Also confirm you replaced the username placeholder in the remote URL with your actual GitHub username.

If you see remote: Permission denied, you may need to set up a personal access token or SSH key for GitHub authentication.

Help me fix my Git push error.

Verify the repository structure

The final check is to confirm everything uploaded correctly to GitHub.

  • Open your GitHub repository in your browser by navigating to https://github.com/ followed by your username and /github-trending-dashboard.
  • Click into the backend/ folder.

You should see exactly five files inside backend/:

  • .gitignore.
  • cleaner.py.
  • data.json.
  • requirements.txt.
  • scraper.py.

The raw_data.json file and venv/ folder should not appear anywhere in the repository.

Missing files or seeing venv/?

If data.json is missing, make sure you ran python cleaner.py in the previous step before committing. Run the cleaner, then commit and push again.

If venv/ appears in the repo, your .gitignore file may not have been saved correctly before the commit. Double check that venv/ is listed on its own line in .gitignore.

Help me fix my GitHub repository structure.

Secret mission

Add Language Filtering from the Command Line

Your scraper currently grabs all trending repos regardless of language. What if you only care about Python projects? Or want to see what's trending in Rust this week? Add command-line flags to make your scraper flexible enough to filter by language and time range, without editing a single line of source code.

Before you go

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally, so there are no ongoing costs.

Resources you used:

  • Python virtual environment (venv/) in the backend folder.
  • Project files: scraper.py, cleaner.py, requirements.txt, .gitignore.
  • Data files: raw_data.json (local only) and data.json (committed to GitHub).
  • GitHub repository: github-trending-dashboard.

Keep everything running

No action needed. Choose this if you are still actively building or plan to start Part 2 soon.

  • Your virtual environment, project files, and GitHub repository stay exactly as they are.
  • Everything is local, so nothing is running in the background or costing money.

Pause - I'll come back to this later

Deactivate the virtual environment to free up your terminal session. Your files and GitHub repo stay intact.

  • Run this command in your terminal to deactivate the virtual environment:
deactivate
  • When you come back, navigate to the backend folder and reactivate with:
venv\Scripts\activate

Delete - I don't want to use this again

Remove all project resources from your local machine and start fresh. Note that this will also remove the files needed for Part 2.

  • Deactivate the virtual environment if it is still active:
deactivate
  • Navigate to the parent directory and delete the entire project folder:
cd ..
rmdir /s /q github-trending-dashboard
  • To delete the GitHub repository, go to your repo page on GitHub.
  • Open Settings, scroll to the bottom, and select Delete this repository.
  • Confirm the deletion by typing the repository name when prompted.

Mission accomplished

You just built a real-world Python data pipeline from scratch. Starting from a blank project folder, you scraped live data from GitHub's Trending page, enriched it with the GitHub REST API, and cleaned everything into a structured JSON dataset ready for frontend consumption.

Here's what you accomplished:

  • Built a web scraper with requests and BeautifulSoup that extracts live repository data from GitHub Trending, then enriches each repo with the GitHub REST API to fill in fields the HTML page does not expose.
  • Built a Pandas cleaning pipeline in cleaner.py that validates, deduplicates, type-converts, and transforms raw data into a production-ready data.json with 11 typed fields per repository.
  • Pushed a clean, organized project to GitHub with proper .gitignore rules that separate raw intermediate files from the final dataset.
  • Added command-line --language and --date-range filtering with argparse so the scraper can target specific languages and time ranges.

This dataset is the foundation for Part 2, where you will build an interactive frontend dashboard to visualize this data.

Ready to quiz yourself?