Scrape Football Stats with Python

Build a Python scraper to extract player stats from FBref into a CSV.

Introduction

30 Second Summary

Every football fan has a prediction for who will win the Golden Boot. The difference between a guess and an informed prediction is data. Player goals, expected goals, minutes played, and assists across Europe's top leagues tell a story that gut feelings cannot.

In this project, you will build a Python web scraper that pulls player statistics from FBref.com and exports them to a clean CSV file ready for analysis.

What You'll Build

A Python script that scrapes FBref's Big 5 European Leagues player statistics page and produces a cleaned dataset of every player's scoring stats across the Premier League, La Liga, Bundesliga, Serie A, and Ligue 1.

By the end of this project, you'll have:

  • A working Python scraper that fetches player data from FBref using proper HTTP headers and rate limiting.
  • A clean CSV dataset of 2,500+ player records with goals, assists, xG, and minutes across 5 leagues.
  • Hands-on experience with HTML parsing, extracting data hidden inside HTML comments using BeautifulSoup.
  • Secret Mission: Scrape a second historical season to start building multi-year data for Golden Boot predictions.

Are there any prerequisites?

None. This guide includes setup for all required tools. You do not need prior Python or web scraping experience.

Set Up Your Python Environment

Before you can scrape any data from FBref, you need a working Python environment with the right packages installed.

In this step, you will create a dedicated project folder, set up a virtual environment to keep your packages isolated, and install the three libraries your scraper needs.

In this step, get ready to:
  • Verify Python is installed and create a project folder with a virtual environment.
  • Install the three required packages for web scraping and data processing.
  • Create the project file where your scraper code will live.
Verify Python installation and create a project folder

The packages this project uses require Python 3.9 or higher. Let's confirm your version before going further.

  • Open the integrated terminal in VS Code by pressing Cmd+` (macOS) or Ctrl+` (Windows).
  • Check your Python version by running this command:

macOS

python3 --version

Windows

python --version

✔️ I see version 3.9 or higher

You are good to go. Any version like 3.9, 3.10, 3.11, 3.12, or higher works perfectly.

ⓧ I see a version lower than 3.9

You need to upgrade Python. The pandas library this project uses requires Python 3.9 or higher.

  • Visit python.org/downloads and download the latest Python 3 installer for your operating system.
  • Run the installer. On Windows, make sure to check the "Add Python to PATH" option.
  • On macOS, the new version installs alongside the old one. After installing, run python3 --version again to confirm it shows 3.9 or higher.
  • Close and reopen VS Code's terminal after installation so it picks up the new version.

ⓧ Command not found

Python is not installed on your machine yet. Let's fix that.

  • Visit python.org/downloads and download the latest Python 3 installer for your operating system.
  • Run the installer. On Windows, check the "Add Python to PATH" checkbox before clicking Install.
  • Close and reopen VS Code's terminal after installation.
  • Run the version check command again to confirm Python is working.

Now let's create your project folder. A dedicated folder keeps your scraper code, virtual environment, and output data all in one place.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir golden-boot-scraper
cd golden-boot-scraper
  • Confirm you are in the right folder by running:
pwd

You should see a path ending in golden-boot-scraper.

Folder already exists?

If you see an error saying the directory already exists, you may have started this project before. Run cd golden-boot-scraper to move into the existing folder instead.

Help me troubleshoot my folder creation issue.

Next, create a virtual environment. This isolates your project's packages from your system Python so nothing conflicts.

  • Create and activate the virtual environment by running these commands:

macOS

python3 -m venv venv
source venv/bin/activate

Windows

python -m venv venv
venv\Scripts\activate

You should see (venv) appear at the start of your terminal prompt. This confirms the virtual environment is active.

What is a virtual environment?

A virtual environment is a self-contained folder that holds its own copy of Python and any packages you install. It prevents version conflicts between different projects on your machine.

When you see (venv) in your prompt, every pip install command installs packages into this isolated folder instead of your system Python.

Don't see (venv) in your prompt?

Make sure you ran the activate command (not just the python3 -m venv venv part). On Windows, check that you are using PowerShell, not Command Prompt.

Help me activate my Python virtual environment.

Install the three required packages

Your scraper needs three packages: requests for fetching web pages, beautifulsoup4 for parsing HTML, and pandas for organizing and exporting data.

  • Install all three packages by running this command:
pip install requests==2.32.3 beautifulsoup4==4.14.3 pandas==2.3.3

Why pin specific versions?

The == syntax locks each package to an exact version. This ensures your code works the same way as this tutorial regardless of when you run it.

  • Verify all three packages installed correctly by running:
python3 -c "import requests, bs4, pandas; print('All packages installed!')"

You should see All packages installed! printed in your terminal.

Seeing an import error?

Make sure your virtual environment is still active. You should see (venv) in your terminal prompt. If not, run the activate command again from inside the golden-boot-scraper folder.

If you see a network error during installation, check your internet connection and re-run the pip install command.

Help me fix my package installation issue.

Create the project file

Now you need a Python file where your scraper code will live.

  • In VS Code, look at the Explorer sidebar on the left. If it is not visible, press Cmd+Shift+E (macOS) or Ctrl+Shift+E (Windows) to open it.
  • Click the New File icon at the top of the Explorer panel (it looks like a page with a plus sign).
  • Type scrape_fbref.py as the file name and press Enter.
  • Confirm the file was created by running this in your terminal:
ls

You should see scrape_fbref.py and venv listed in the output.

Why this file name?

FBref is the football statistics website you will scrape. Naming the file scrape_fbref.py makes it immediately clear what the script does when you come back to it later.

Your environment is ready. Next up, you will write the code to fetch the FBref page and see real football data come back.

Fetch the FBref Page with Proper Headers

Your Python environment is ready with all three packages installed. Now it's time to write the code that pulls live football data from the web.

FBref.com hosts detailed player statistics for every major European league. In this step, you'll send an HTTP request to fetch the Big 5 leagues page while respecting the site's rate limits.

In this step, get ready to:
  • Send an HTTP request to FBref with a proper User-Agent header.
  • Handle common HTTP errors like rate limiting and forbidden responses.
  • Verify your script successfully fetches the page.
Write the HTTP request with a User-Agent header

Before you can parse any data, your script needs to download the raw HTML from FBref's server. Websites expect visitors to identify themselves, so you'll include a User-Agent header that tells FBref your request is coming from a standard browser.

  • Add the imports and configuration to your script by copying this code into scrape_fbref.py:
import requests
from bs4 import BeautifulSoup, Comment
import pandas as pd
import time

# --- Configuration ---
URL = "https://fbref.com/en/comps/Big5/stats/players/Big-5-European-Leagues-Stats"

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
}

What does this code do?

The first four lines import the libraries you'll use throughout the project. requests handles HTTP calls, BeautifulSoup parses HTML, pandas works with tabular data, and time lets you add delays between requests.

URL points to FBref's combined player stats page covering the Premier League, La Liga, Bundesliga, Serie A, and Ligue 1. HEADERS includes a User-Agent string that makes your script look like a regular Chrome browser to FBref's server.

  • Save scrape_fbref.py.
  • You won't see a change yet. The next chunk adds the fetch logic that uses this configuration.

Why a User-Agent header?

When you visit a website in Chrome, your browser automatically sends a User-Agent string identifying itself. Without one, your Python script sends a generic identifier that many websites block as bot traffic.

FBref specifically checks for this. Including a realistic User-Agent string is the difference between getting the page and getting a 403 Forbidden response.

Handle errors and understand rate limiting

FBref blocks users who send more than 10 requests per minute. Your script needs to wait before making a request and handle common error responses gracefully.

  • Add the fetch logic below the HEADERS dictionary by copying this code at the bottom of scrape_fbref.py:
# --- Step 1: Fetch the page ---
print("Fetching FBref Big 5 European Leagues stats page...")
print("Waiting 6 seconds to respect rate limits...")
time.sleep(6)

response = requests.get(URL, headers=HEADERS)

if response.status_code != 200:
    print(f"Error: Received status code {response.status_code}")
    if response.status_code == 429:
        print("Rate limited! Wait a minute and try again.")
    elif response.status_code == 403:
        print("Access forbidden. Check your User-Agent header.")
    exit()

print(f"Success! Status: {response.status_code}, Content length: {len(response.text)} characters")

What does this code do?

time.sleep(6) pauses your script for 6 seconds before making the request. FBref enforces a limit of 10 requests per minute, which works out to one request every 6 seconds.

requests.get(URL, headers=HEADERS) sends the actual HTTP GET request to FBref's server, passing your User-Agent header along with it.

The if block catches two specific errors. A 429 status means you're sending requests too fast. A 403 status means the server rejected your request entirely (usually a bad or missing User-Agent). If everything succeeds, the final print confirms status 200 and shows how many characters of HTML were downloaded.

  • Save scrape_fbref.py.

What is rate limiting?

Rate limiting is when a server restricts how many requests you can send within a time window. It protects the server from being overloaded by automated scripts.

FBref's policy blocks users who exceed 10 requests per minute. Violations can result in your session being blocked for up to one day. The 6-second sleep keeps you safely within the limit.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before running it in the next substep.

ⓧ I'd like to double check the full code

Here's what your complete scrape_fbref.py should look like so far:

import requests
from bs4 import BeautifulSoup, Comment
import pandas as pd
import time

# --- Configuration ---
URL = "https://fbref.com/en/comps/Big5/stats/players/Big-5-European-Leagues-Stats"

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
}

# --- Step 1: Fetch the page ---
print("Fetching FBref Big 5 European Leagues stats page...")
print("Waiting 6 seconds to respect rate limits...")
time.sleep(6)

response = requests.get(URL, headers=HEADERS)

if response.status_code != 200:
    print(f"Error: Received status code {response.status_code}")
    if response.status_code == 429:
        print("Rate limited! Wait a minute and try again.")
    elif response.status_code == 403:
        print("Access forbidden. Check your User-Agent header.")
    exit()

print(f"Success! Status: {response.status_code}, Content length: {len(response.text)} characters")
Run the script and verify you get a successful response
  • Run your script to test the connection by running this command in your terminal:
python3 scrape_fbref.py
  • After a 6-second pause, you should see output like this:
Fetching FBref Big 5 European Leagues stats page...
Waiting 6 seconds to respect rate limits...
Success! Status: 200, Content length: 876543 characters

What should I see?

Your exact character count will differ from the example above, but it should be a large number (several hundred thousand characters). This confirms you successfully downloaded the entire HTML page containing player data for 2,500+ players across all 5 leagues.

Seeing an error instead?

  • If you see status code 403: make sure the HEADERS dictionary in your script includes the full User-Agent string exactly as shown above.
  • If you see status code 429: you've hit the rate limit. Wait at least one minute before running the script again.
  • If you see ModuleNotFoundError: make sure your virtual environment is activated. You should see (venv) at the start of your terminal prompt.

Help me debug my FBref scraper connection error

Your script can now reliably fetch the FBref page. Next up, you'll parse the hidden stats table buried inside the HTML to extract the actual player data.

Parse the Hidden HTML Table

Your script now fetches the full FBref page successfully. The next challenge is finding the actual player data inside all that HTML.

FBref hides their stats tables inside HTML comments to make scraping harder. In this step, you'll dig through the page structure, extract the hidden table, and convert it into a pandas DataFrame you can work with.

In this step, get ready to:
  • Parse the fetched HTML and locate the hidden stats div.
  • Extract the table from inside an HTML comment.
  • Convert the table into a pandas DataFrame ready for cleaning.
Understand FBref's hidden table structure

Most websites put data directly in HTML table elements. FBref takes an extra step to deter scrapers by wrapping their stats tables inside HTML comments.

Your browser ignores comments when rendering the page, but JavaScript on FBref's frontend un-comments them after the page loads. Your script needs to find these comments and extract the table manually.

What is an HTML comment?

HTML comments are text wrapped in <!-- and --> tags. Browsers skip over them entirely. FBref puts the full stats table inside a comment so that a simple scraper using soup.find("table") won't find it.

  • In scrape_fbref.py, find the last line of your script:
print(f"Success! Status: {response.status_code}, Content length: {len(response.text)} characters")
  • Add the following code below that line to parse the HTML and find the hidden stats div:
# --- Step 2: Parse the hidden table ---
print("\nParsing HTML to find the hidden stats table...")

soup = BeautifulSoup(response.text, "html.parser")

# Find the div that contains the standard stats table
div = soup.find("div", id="all_stats_standard")

if div is None:
    print("Error: Could not find the stats table div. The page structure may have changed.")
    exit()

What does this code do?

BeautifulSoup takes the raw HTML text from your response and turns it into a searchable tree structure. Think of it like giving you a map of the page.

  • BeautifulSoup(response.text, "html.parser") creates the searchable tree from the page's HTML.
  • soup.find("div", id="all_stats_standard") searches for the specific div that FBref uses to hold the player stats table.
  • The if div is None check exits early if FBref changed their page structure, saving you from a confusing error later.
  • Save scrape_fbref.py.
  • You won't run the script yet. The next chunk will extract the actual table from within this div.
Extract the commented table and parse it

You've found the div that contains the stats data. Now you need to crack it open. Inside this div, the real table is wrapped in an HTML comment node.

  • Add the following code below the exit() line you just wrote to extract the table from the comment:
# The table is hidden inside an HTML comment within this div
comment = div.find(string=lambda text: isinstance(text, Comment))

if comment is None:
    print("Error: Could not find the commented table. Trying direct table search...")
    table = div.find("table", id="stats_standard")
else:
    # Parse the comment as HTML to extract the table
    comment_soup = BeautifulSoup(comment, "html.parser")
    table = comment_soup.find("table", id="stats_standard")

if table is None:
    print("Error: Could not find table with id 'stats_standard'.")
    exit()

print("Found the Player Standard Stats table!")

What does this code do?

This is where you outsmart FBref's anti-scraping technique. The Comment class from BeautifulSoup lets you find HTML comment nodes specifically.

  • div.find(string=lambda text: isinstance(text, Comment)) searches inside the div for a comment node (text between <!-- and -->).
  • Once found, BeautifulSoup(comment, "html.parser") re-parses that comment's text as fresh HTML, making the table searchable again.
  • The fallback (if comment is None) tries finding the table directly in case FBref ever removes the comment wrapper.
  • Save scrape_fbref.py.
  • You won't see a change yet. The next chunk converts this table into a DataFrame you can work with.
Convert the table to a pandas DataFrame

You have the raw HTML table extracted. Now you need to turn it into structured data you can filter, sort, and export.

  • Add the following code at the bottom of scrape_fbref.py to convert the table to a DataFrame:
# --- Step 3: Convert to DataFrame ---
print("\nConverting table to DataFrame...")

df = pd.read_html(str(table), header=1)[0]

print(f"Raw data shape: {df.shape[0]} rows x {df.shape[1]} columns")

What does this code do?

pd.read_html() is a pandas function that reads HTML tables directly into DataFrames. It handles all the row and column parsing for you.

  • str(table) converts the BeautifulSoup table object back into an HTML string that pandas can read.
  • header=1 tells pandas to use the second row as column headers. FBref uses a multi-level header where the first row is a category grouping. Skipping it gives you clean column names like Gls, Ast, and xG.
  • [0] grabs the first (and only) table from the list that read_html() returns.
  • Save scrape_fbref.py.
  • Run the script in your terminal by running:
python3 scrape_fbref.py
  • After the 6-second wait, you should see output ending with something like Raw data shape: 2800 rows x 35 columns. Your exact numbers will vary depending on the current season's data.

Seeing an error about the stats div or table?

  • If you see "Could not find the stats table div", your fetch may have returned an error page instead of the real content. Check that the previous output shows status 200 and a content length of at least 100,000 characters.
  • If you see "Could not find table with id 'stats_standard'", the page structure may have changed. Double check that the div id all_stats_standard matches what you see in your browser's DevTools on the FBref page.
  • If you get a NameError for BeautifulSoup or Comment, check that your imports at the top of the file say from bs4 import BeautifulSoup, Comment.

Help me debug my FBref table parsing code

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

Here's what your full scrape_fbref.py should look like at this point:

import requests
from bs4 import BeautifulSoup, Comment
import pandas as pd
import time

# --- Configuration ---
URL = "https://fbref.com/en/comps/Big5/stats/players/Big-5-European-Leagues-Stats"

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
}

# --- Step 1: Fetch the page ---
print("Fetching FBref Big 5 European Leagues stats page...")
print("Waiting 6 seconds to respect rate limits...")
time.sleep(6)

response = requests.get(URL, headers=HEADERS)

if response.status_code != 200:
    print(f"Error: Received status code {response.status_code}")
    if response.status_code == 429:
        print("Rate limited! Wait a minute and try again.")
    elif response.status_code == 403:
        print("Access forbidden. Check your User-Agent header.")
    exit()

print(f"Success! Status: {response.status_code}, Content length: {len(response.text)} characters")

# --- Step 2: Parse the hidden table ---
print("\nParsing HTML to find the hidden stats table...")

soup = BeautifulSoup(response.text, "html.parser")

# Find the div that contains the standard stats table
div = soup.find("div", id="all_stats_standard")

if div is None:
    print("Error: Could not find the stats table div. The page structure may have changed.")
    exit()

# The table is hidden inside an HTML comment within this div
comment = div.find(string=lambda text: isinstance(text, Comment))

if comment is None:
    print("Error: Could not find the commented table. Trying direct table search...")
    table = div.find("table", id="stats_standard")
else:
    # Parse the comment as HTML to extract the table
    comment_soup = BeautifulSoup(comment, "html.parser")
    table = comment_soup.find("table", id="stats_standard")

if table is None:
    print("Error: Could not find table with id 'stats_standard'.")
    exit()

print("Found the Player Standard Stats table!")

# --- Step 3: Convert to DataFrame ---
print("\nConverting table to DataFrame...")

df = pd.read_html(str(table), header=1)[0]

print(f"Raw data shape: {df.shape[0]} rows x {df.shape[1]} columns")

You've cracked open FBref's hidden table and loaded thousands of player records into a DataFrame. Next up, you'll clean this raw data and export it as a CSV ready for analysis.

Clean the Data and Export to CSV

Your script can now fetch the FBref page and extract the hidden stats table into a DataFrame. The raw data has a problem though. FBref inserts repeated header rows throughout the table, and all the values are stored as strings instead of numbers.

In this step, you will clean the raw DataFrame and export it to a CSV file. You will end up with a tidy dataset of 2,500+ players sorted by goals scored.

In this step, get ready to:
  • Remove junk rows and select only the columns you need.
  • Convert string data to proper numeric types.
  • Export to CSV and display the top 10 goal scorers.
Remove separator rows and select relevant columns

FBref inserts duplicate header rows into the table every 25 rows. These rows have the literal text "Player" in the Player column instead of an actual name. You also have far more columns than you need for Golden Boot prediction.

  • In scrape_fbref.py, find the line that says print(f"Raw data shape: {df.shape[0]} rows x {df.shape[1]} columns").
  • Add the following code below it to filter out junk rows and select only the columns relevant to scoring stats:
# --- Step 4: Clean the data ---
print("\nCleaning data...")

# Remove repeated header rows (FBref inserts these every 25 rows)
df = df[df["Player"] != "Player"].reset_index(drop=True)

# Select columns relevant to Golden Boot prediction
columns_to_keep = [
    "Player", "Nation", "Pos", "Squad", "Comp", "Age",
    "MP", "Starts", "Min", "90s",
    "Gls", "Ast", "G+A", "G-PK", "PK", "PKatt",
    "xG", "npxG", "xAG"
]

df = df[columns_to_keep]

What does this code do?

df[df["Player"] != "Player"] keeps only rows where the Player column contains an actual player name, not the repeated header text "Player". reset_index(drop=True) renumbers the rows cleanly after filtering.

The columns_to_keep list narrows the DataFrame from 30+ columns down to 19 that matter for goal-scoring analysis: player info, match appearances, goals, assists, expected goals (xG), and penalties.

  • Save scrape_fbref.py.
  • You won't see a change yet. The next chunk converts these columns to proper numeric types.
Convert data types and handle missing values

When pandas reads an HTML table, every value comes in as a string. You cannot sort by goals or calculate averages until the numeric columns are actual numbers.

  • Add the following code below df = df[columns_to_keep] to convert string values to numbers and sort by goals:
# Convert numeric columns to proper types
numeric_columns = ["MP", "Starts", "Min", "90s", "Gls", "Ast", "G+A",
                   "G-PK", "PK", "PKatt", "xG", "npxG", "xAG"]

for col in numeric_columns:
    df[col] = pd.to_numeric(df[col], errors="coerce").fillna(0)

# Sort by goals scored (descending)
df = df.sort_values("Gls", ascending=False).reset_index(drop=True)

What does this code do?

pd.to_numeric(df[col], errors="coerce") converts each column from strings to numbers. The errors="coerce" argument turns any unconvertible values (like empty cells) into NaN instead of crashing the script.

.fillna(0) replaces those NaN values with 0, which makes sense for stats like goals or assists where a missing value means "none recorded". Finally, sort_values("Gls", ascending=False) puts the top scorers at the top of the DataFrame.

  • Save scrape_fbref.py.
  • You won't see a change yet. The next chunk exports this cleaned data to a CSV file and prints the results.
Save to CSV and display the top 10 scorers
  • Add the following code at the bottom of scrape_fbref.py to export the DataFrame and print a summary:
# --- Step 5: Export and display results ---
output_file = "big5_player_stats.csv"
df.to_csv(output_file, index=False)

print(f"\nData saved to {output_file}")
print(f"Total players: {len(df)}")
print(f"Leagues covered: {df['Comp'].nunique()}")
print(f"\nTop 10 Goal Scorers - Big 5 European Leagues:")
print("-" * 60)

top_scorers = df.head(10)[["Player", "Squad", "Comp", "Gls", "xG", "Min"]]
print(top_scorers.to_string(index=False))

print("\nDone! Your Golden Boot data is ready.")

What does this code do?

df.to_csv(output_file, index=False) writes the entire cleaned DataFrame to a CSV file. The index=False argument prevents pandas from adding an extra row-number column to the output.

The rest prints a human-readable summary: total players, number of leagues, and a formatted table of the top 10 goal scorers showing their name, squad, league, goals, xG, and minutes played.

  • Save scrape_fbref.py.
  • Run the complete script by entering this command in your Terminal:
python3 scrape_fbref.py

After the 6-second rate-limit pause, you should see output showing status 200, the table being found, cleaning messages, and finally the top 10 goal scorers printed as a formatted table. The last line should say "Done! Your Golden Boot data is ready."

Seeing a KeyError or column not found?

If you see an error about a missing column, double check that your columns_to_keep list matches exactly what is in the code above. Column names like G+A and G-PK are case-sensitive and include special characters.

If you get a 429 status code, FBref is rate limiting you. Wait at least one minute and try again.

Help me debug my scrape_fbref.py script

  • Confirm the CSV file was created by running:
ls

You should see big5_player_stats.csv listed alongside scrape_fbref.py and the venv folder.

✔️ Awesome, I've got everything!

Great. Double check you have saved your file before moving on.

ⓧ I'd like to double check the full code

Here is the complete scrape_fbref.py file at this point:

import requests
from bs4 import BeautifulSoup, Comment
import pandas as pd
import time

# --- Configuration ---
URL = "https://fbref.com/en/comps/Big5/stats/players/Big-5-European-Leagues-Stats"

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
}

# --- Step 1: Fetch the page ---
print("Fetching FBref Big 5 European Leagues stats page...")
print("Waiting 6 seconds to respect rate limits...")
time.sleep(6)

response = requests.get(URL, headers=HEADERS)

if response.status_code != 200:
    print(f"Error: Received status code {response.status_code}")
    if response.status_code == 429:
        print("Rate limited! Wait a minute and try again.")
    elif response.status_code == 403:
        print("Access forbidden. Check your User-Agent header.")
    exit()

print(f"Success! Status: {response.status_code}, Content length: {len(response.text)} characters")

# --- Step 2: Parse the hidden table ---
print("\nParsing HTML to find the hidden stats table...")

soup = BeautifulSoup(response.text, "html.parser")

# Find the div that contains the standard stats table
div = soup.find("div", id="all_stats_standard")

if div is None:
    print("Error: Could not find the stats table div. The page structure may have changed.")
    exit()

# The table is hidden inside an HTML comment within this div
comment = div.find(string=lambda text: isinstance(text, Comment))

if comment is None:
    print("Error: Could not find the commented table. Trying direct table search...")
    table = div.find("table", id="stats_standard")
else:
    # Parse the comment as HTML to extract the table
    comment_soup = BeautifulSoup(comment, "html.parser")
    table = comment_soup.find("table", id="stats_standard")

if table is None:
    print("Error: Could not find table with id 'stats_standard'.")
    exit()

print("Found the Player Standard Stats table!")

# --- Step 3: Convert to DataFrame ---
print("\nConverting table to DataFrame...")

df = pd.read_html(str(table), header=1)[0]

print(f"Raw data shape: {df.shape[0]} rows x {df.shape[1]} columns")

# --- Step 4: Clean the data ---
print("\nCleaning data...")

# Remove repeated header rows (FBref inserts these every 25 rows)
df = df[df["Player"] != "Player"].reset_index(drop=True)

# Select columns relevant to Golden Boot prediction
columns_to_keep = [
    "Player", "Nation", "Pos", "Squad", "Comp", "Age",
    "MP", "Starts", "Min", "90s",
    "Gls", "Ast", "G+A", "G-PK", "PK", "PKatt",
    "xG", "npxG", "xAG"
]

df = df[columns_to_keep]

# Convert numeric columns to proper types
numeric_columns = ["MP", "Starts", "Min", "90s", "Gls", "Ast", "G+A",
                   "G-PK", "PK", "PKatt", "xG", "npxG", "xAG"]

for col in numeric_columns:
    df[col] = pd.to_numeric(df[col], errors="coerce").fillna(0)

# Sort by goals scored (descending)
df = df.sort_values("Gls", ascending=False).reset_index(drop=True)

# --- Step 5: Export and display results ---
output_file = "big5_player_stats.csv"
df.to_csv(output_file, index=False)

print(f"\nData saved to {output_file}")
print(f"Total players: {len(df)}")
print(f"Leagues covered: {df['Comp'].nunique()}")
print(f"\nTop 10 Goal Scorers - Big 5 European Leagues:")
print("-" * 60)

top_scorers = df.head(10)[["Player", "Squad", "Comp", "Gls", "xG", "Min"]]
print(top_scorers.to_string(index=False))

print("\nDone! Your Golden Boot data is ready.")

You now have a clean CSV file with 2,500+ player records from Europe's top 5 leagues. This is real, structured data that you scraped, cleaned, and exported entirely with Python. Next up, you will take on a secret mission to scrape historical data across multiple seasons.

Secret mission

Scrape Historical Data for Multi-Season Analysis

Your single-season scraper works, but a Golden Boot predictor needs trends over time. Can you extend your script to scrape the 2023-2024 season as well, combining both into one multi-year dataset?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your files for future use or delete them to free up disk space.

Resources you used:

  • Project folder golden-boot-scraper (contains your script, CSV files, and virtual environment).

Keep everything running

No action needed. Choose this if you plan to keep building on your scraper or want to reference the data later.

  • Your scrape_fbref.py script and CSV files stay on your Desktop indefinitely at no cost.
  • You can rerun the script any time to get updated stats (the current season's data changes as matches are played).
  • To reactivate your virtual environment later, open Terminal, navigate to your project folder, and run source venv/bin/activate (macOS/Linux) or venv\Scripts\activate (Windows).

Pause - I'll come back to this later

Deactivate your virtual environment to free up your terminal session. Your files remain on disk for when you return.

  • Deactivate your virtual environment by running:
deactivate
  • Your project folder, script, and CSV files remain on your Desktop. No data is lost.

Delete - I don't want to use this again

Remove the entire project folder including your script, CSV files, and virtual environment.

  • Deactivate your virtual environment first (if it is still active):
deactivate
  • Navigate to your Desktop and delete the project folder by running:

macOS

cd ~/Desktop
rm -rf golden-boot-scraper

Windows

cd ~\Desktop
rmdir /s /q golden-boot-scraper
  • This removes scrape_fbref.py, big5_player_stats.csv, big5_player_stats_multi_season.csv, and the venv virtual environment. Nothing else on your machine is affected.

That's a Wrap!

That's a Wrap!

Nice work! You just built a Python web scraper that pulls real player statistics from FBref and exports them into a clean, analysis-ready CSV dataset.

You've learned how to:

  • Send HTTP requests with proper User-Agent headers and rate limiting to scrape websites ethically.
  • Extract data hidden inside HTML comments using BeautifulSoup and parse it into structured tables.
  • Clean and export 2,500+ player records to a reusable CSV using pandas DataFrames.
  • Secret Mission: Scraped multiple historical seasons and combined them into a multi-year dataset for Golden Boot predictions.

Ready to quiz yourself?