Data-Driven Player Recruitment Engine

Build a scouting tool that finds statistically similar footballers.

Introduction

30 Second Summary

Football clubs spend millions on transfers that flop because scouts rely on gut feeling and highlight reels instead of data. What if you could find hidden gems by comparing players' actual performance numbers across leagues?

In this project, you will build a player recruitment tool that finds statistically similar midfielders from cheaper European leagues. You will create an interactive web app with similarity scores and radar chart visualizations.

What You'll Build

You type in a famous midfielder's name like Rodri, and the app instantly returns the top 5 most similar young players from affordable leagues, with visual proof of how their playing styles compare.

By the end of this project, you'll have:

  • A working data pipeline that scrapes real season stats from FBref for 7 European leagues, cleans them, and normalizes them into a format your similarity engine can use.
  • A cosine similarity engine that finds player "twins" by comparing normalized per-90 metrics, filtered by age and league affordability.
  • An interactive Streamlit web app where you search any midfielder's name, see a ranked shortlist of affordable alternatives, and compare their profiles side-by-side with radar charts.
  • Secret Mission: Add PCA dimensionality reduction to plot all midfielders on a 2D scatter map, showing where your target player sits relative to their twins.

Are there any prerequisites?

You need Python 3.10 or higher installed on your machine, along with pip and a code editor like VS Code. Basic familiarity with running Python scripts from a terminal is helpful.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters. This is your moment to commit to the goal before any hands-on work begins.

Set Up Your Environment

To build a player recruitment engine, you need three categories of tools: a data scraping library to pull real football stats, a math library to compute similarity, and an app framework to present results visually.

In this step, you will install all required libraries and configure the soccerdata package to access leagues beyond the built-in top 5. By the end, your environment will be ready to pull data from 7 European leagues.

In this step, get ready to:
  • Create a project folder and virtual environment with all dependencies installed.
  • Configure soccerdata to access the Eredivisie, Belgian Pro League, and Primeira Liga.
  • Verify the configuration recognizes all custom leagues.
Create a project folder and virtual environment

Every Python project starts with an isolated environment. A virtual environment keeps your project's packages separate from your system Python so nothing conflicts.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir player-scout
cd player-scout
  • Confirm the folder exists by running:
pwd

You should see a path ending in player-scout printed in your terminal.

  • Create a virtual environment by running:
python -m venv venv
  • Activate the virtual environment by running the command for your operating system:

macOS

source venv/bin/activate

Windows

venv\Scripts\activate

Your terminal prompt now shows (venv) at the beginning. This confirms you are working inside the virtual environment.

Don't see (venv) in your prompt?

Make sure you ran the activate command from inside the player-scout folder. If you are on Windows and get a permissions error, run Set-ExecutionPolicy -Scope CurrentUser RemoteSigned in PowerShell first, then try activating again.

help me activate my Python virtual environment

  • Install all project dependencies by running:
pip install soccerdata pandas scikit-learn streamlit matplotlib

What does this install?

  • soccerdata scrapes football statistics from FBref and other sources.
  • pandas handles DataFrames for cleaning and merging player stats.
  • scikit-learn provides the cosine similarity and normalization algorithms.
  • streamlit turns your Python script into an interactive web app.
  • matplotlib draws radar charts to visually compare player profiles.

The install takes a minute or two. You will see packages downloading and a success message at the end.

  • Verify all packages imported correctly by running:
python -c "import soccerdata, sklearn, streamlit, pandas, matplotlib; print('All packages imported')"

You should see All packages imported printed with no errors.

Seeing an ImportError?

Make sure your virtual environment is active (you see (venv) in the prompt). If not, activate it again and re-run the pip install command. The most common cause is running python outside the venv so it cannot find the newly installed packages.

help me fix my package import errors in my player-scout project

Configure soccerdata for custom leagues

The soccerdata library only includes the Big 5 European leagues by default. To access the Eredivisie, Belgian Pro League, and Primeira Liga, you need to create a configuration file that maps each league to its FBref identifier.

  • Create the soccerdata config directory by running the command for your operating system:

macOS

mkdir -p ~/soccerdata/config

Windows

New-Item -ItemType Directory -Force -Path "$HOME\soccerdata\config"
  • Create the league configuration file by running the command for your operating system:

macOS

cat > ~/soccerdata/config/league_dict.json << 'EOF'
{
  "NED-Eredivisie": {"FBref": "Dutch Eredivisie"},
  "BEL-Belgian Pro League": {"FBref": "Belgian First Division A"},
  "POR-Primeira Liga": {"FBref": "Portuguese Primeira Liga"}
}
EOF

Windows

@'
{
  "NED-Eredivisie": {"FBref": "Dutch Eredivisie"},
  "BEL-Belgian Pro League": {"FBref": "Belgian First Division A"},
  "POR-Primeira Liga": {"FBref": "Portuguese Primeira Liga"}
}
'@ | Set-Content "$HOME\soccerdata\config\league_dict.json"

What does this config file do?

The league_dict.json file tells soccerdata how to find each league on FBref. Each key (like NED-Eredivisie) becomes the league ID you use in your code. The FBref value maps it to the competition name that FBref uses internally.

These three leagues represent affordable transfer markets where clubs from Europe's biggest leagues often find undervalued talent.

  • Confirm the file was created correctly by running:

macOS

cat ~/soccerdata/config/league_dict.json

Windows

Get-Content "$HOME\soccerdata\config\league_dict.json"

You should see the JSON you just wrote, showing all three league mappings.

✔️ Awesome, I've got everything!

Great. Make sure you have saved the file and move on to the verification step below.

ⓧ I'd like to double check the full code

Your ~/soccerdata/config/league_dict.json should contain exactly this:

{
  "NED-Eredivisie": {"FBref": "Dutch Eredivisie"},
  "BEL-Belgian Pro League": {"FBref": "Belgian First Division A"},
  "POR-Primeira Liga": {"FBref": "Portuguese Primeira Liga"}
}
Verify the configuration works

The real test is whether soccerdata picks up your custom leagues. When you import the library, it reads league_dict.json and merges your entries with the built-in defaults.

  • Before you run the next command, predict: will you see just 3 leagues, or more than 3? Think about why.
  • Check the available leagues by running:
python -c "import soccerdata as sd; print(sd.FBref.available_leagues())"

You should see a list that includes the original built-in leagues AND your three new entries: NED-Eredivisie, BEL-Belgian Pro League, and POR-Primeira Liga. The library merges your config with its defaults, so you see everything together.

Custom leagues not appearing in the list?

If you only see the default leagues without your three custom ones, check these common causes:

  • Verify the file is at the correct path. Run ls ~/soccerdata/config/league_dict.json (macOS) or Test-Path "$HOME\soccerdata\config\league_dict.json" (Windows) to confirm it exists.
  • Check for JSON syntax errors by running python -m json.tool ~/soccerdata/config/league_dict.json to validate the file.
  • Make sure you are running python from within the activated virtual environment.

help me get my custom leagues to appear in soccerdata

Your environment is locked and loaded. Next up, you will use this setup to pull real player statistics from FBref across all 7 leagues.

Pull and Explore the Raw Data

Your libraries are installed and your league configuration is ready. Before you can find similar players, you need actual season stats for midfielders across multiple leagues.

In this step, you'll write a script that pulls raw data from FBref via the soccerdata library. You'll quickly discover that FBref returns data with messy multi-level column headers that need flattening before you can work with them.

In this step, get ready to:
  • Pull standard, passing, and defense season stats for all 7 leagues from FBref.
  • Flatten multi-level columns and filter the data to midfielders only.
  • Merge the three stat tables into one DataFrame and save as raw_midfielders.csv.
Fetch stats for all 7 leagues

FBref is one of the most comprehensive free sources of football statistics. The soccerdata library wraps FBref's pages into pandas DataFrames you can work with in Python.

  • Create a new file called scrape_data.py in your player-scout folder and paste the following code:
import soccerdata as sd
import pandas as pd
import time

# Big 5 leagues (built-in)
big5_fbref = sd.FBref(leagues="Big 5 European Leagues Combined", seasons="2526")

print("Fetching Big 5 standard stats...")
big5_standard = big5_fbref.read_player_season_stats(stat_type="standard")
time.sleep(8)

print("Fetching Big 5 passing stats...")
big5_passing = big5_fbref.read_player_season_stats(stat_type="passing")
time.sleep(8)

print("Fetching Big 5 defense stats...")
big5_defense = big5_fbref.read_player_season_stats(stat_type="defense")
time.sleep(8)

What does this code do?

  • sd.FBref(leagues=..., seasons="2526") creates a scraper targeting the 2025-26 season for the Big 5 European leagues (Premier League, La Liga, Bundesliga, Serie A, Ligue 1).
  • read_player_season_stats(stat_type="standard") pulls each player's season totals. You also fetch "passing" and "defense" stats for a complete midfielder profile.
  • time.sleep(8) pauses 8 seconds between requests to avoid FBref's rate limit (which blocks you with a 429 error if you exceed 10 requests per minute).
  • Below the Big 5 fetches, add the custom leagues loop:
# Custom leagues (configured in league_dict.json)
custom_leagues = ["NED-Eredivisie", "BEL-Belgian Pro League", "POR-Primeira Liga"]

custom_standard_frames = []
custom_passing_frames = []
custom_defense_frames = []

for league in custom_leagues:
    print(f"Fetching {league} standard stats...")
    fbref = sd.FBref(leagues=league, seasons="2526")
    custom_standard_frames.append(fbref.read_player_season_stats(stat_type="standard"))
    time.sleep(8)

    print(f"Fetching {league} passing stats...")
    custom_passing_frames.append(fbref.read_player_season_stats(stat_type="passing"))
    time.sleep(8)

    print(f"Fetching {league} defense stats...")
    custom_defense_frames.append(fbref.read_player_season_stats(stat_type="defense"))
    time.sleep(8)

What does this loop do?

This iterates through the three custom leagues you configured in league_dict.json during Step 1. For each league, it creates a fresh FBref scraper, fetches all three stat types, and appends each DataFrame to a list.

The 8-second sleep between every request keeps you well under FBref's rate limit across 12 total fetches.

  • Save scrape_data.py.
  • Confirm the file exists by running:
ls scrape_data.py

You should see scrape_data.py listed in the output.

Don't see the file?

  • Make sure you saved the file inside the player-scout folder, not a parent directory.
  • Confirm your terminal is in the player-scout directory by running pwd.
  • Help me find my scrape_data.py file
Combine, flatten, and filter to midfielders

FBref organizes its stats with grouped column headers like "Playing Time", "Performance", and "Per 90 Minutes". When soccerdata reads these, it creates a multi-level column index that makes simple column access impossible without flattening first.

  • Add this processing code below the custom leagues loop in scrape_data.py:
# Combine all leagues
all_standard = pd.concat([big5_standard] + custom_standard_frames)
all_passing = pd.concat([big5_passing] + custom_passing_frames)
all_defense = pd.concat([big5_defense] + custom_defense_frames)

# Reset index to get league, season, team, player as columns
all_standard = all_standard.reset_index()
all_passing = all_passing.reset_index()
all_defense = all_defense.reset_index()

# Flatten multi-level columns
all_standard.columns = ["_".join(col).strip("_") if isinstance(col, tuple) else col for col in all_standard.columns]
all_passing.columns = ["_".join(col).strip("_") if isinstance(col, tuple) else col for col in all_passing.columns]
all_defense.columns = ["_".join(col).strip("_") if isinstance(col, tuple) else col for col in all_defense.columns]

# Filter to midfielders only
midfielders_standard = all_standard[all_standard["pos"].str.contains("MF", na=False)].copy()
midfielders_passing = all_passing[all_passing["pos"].str.contains("MF", na=False)].copy()
midfielders_defense = all_defense[all_defense["pos"].str.contains("MF", na=False)].copy()

What does the flattening do?

FBref returns columns as tuples like ("Per 90 Minutes", "Gls"). The join line turns those into flat strings like Per 90 Minutes_Gls so you can access them normally.

The str.contains("MF") filter catches all midfielder variants: pure midfielders ("MF"), attacking midfielders ("MF,FW"), and defensive midfielders ("DF,MF").

Merge passing and defense stats into one dataset

Right now you have three separate DataFrames for midfielders: standard, passing, and defense. You need to join them into a single row per player with all the stats you'll use for similarity calculations.

  • Add the merge and save logic at the bottom of scrape_data.py:
# Merge on common keys
merge_keys = ["league", "season", "team", "player"]
merged = midfielders_standard.merge(
    midfielders_passing[merge_keys + ["Total_Cmp%", "KP", "PrgP", "Att"]],
    on=merge_keys,
    how="left",
    suffixes=("", "_passing")
)
merged = merged.merge(
    midfielders_defense[merge_keys + ["Tkl", "Int", "Blocks"]],
    on=merge_keys,
    how="left",
    suffixes=("", "_defense")
)

print(f"\nDataset shape: {merged.shape}")
print(f"Unique players: {merged['player'].nunique()}")
print(f"Leagues: {merged['league'].unique().tolist()}")
print(merged.head())

merged.to_csv("raw_midfielders.csv", index=False)
print("\nSaved raw_midfielders.csv")

What does the merge do?

  • The merge uses four keys (league, season, team, player) to match the correct row across DataFrames.
  • From passing stats, you pull Total_Cmp% (pass completion), KP (key passes), PrgP (progressive passes), and Att (passes attempted).
  • From defense stats, you pull Tkl (tackles), Int (interceptions), and Blocks.
  • Save scrape_data.py.

Before you run this, what do you expect the output to show? Think about how many midfielders might be playing across 7 European leagues in a season.

  • Run the scraping script:
python scrape_data.py

This takes 2-3 minutes

The script makes 12 total requests to FBref with 8-second pauses between each one. You'll see progress messages as each league and stat type is fetched. Let it run until you see the final "Saved raw_midfielders.csv" message.

You should see output showing a dataset with 500+ rows, columns including player, team, league, and per-90 stats. All 7 leagues should appear in the leagues list.

  • Confirm the CSV was saved by running:
ls raw_midfielders.csv

You should see raw_midfielders.csv in the output.

Seeing a 429 Too Many Requests error?

  • FBref blocks IPs that make too many requests too quickly. Wait 5-10 minutes before re-running the script.
  • Make sure you have the time.sleep(8) calls between every fetch. The 8-second delay keeps you under FBref's rate limit of 10 requests per minute.
  • If you still get blocked after waiting, try running the script from a different network or wait an hour for the block to expire.
  • Help me fix the 429 rate limit error from FBref

✔️ Awesome, I've got everything!

Great. Double check you've saved scrape_data.py and that raw_midfielders.csv exists in your player-scout folder.

ⓧ I'd like to double check the full code

Your scrape_data.py should contain these sections in order:

  • Imports (soccerdata, pandas, time) and Big 5 league fetches for standard, passing, and defense stats with time.sleep(8) between each.
  • Custom leagues loop fetching all three stat types for NED-Eredivisie, BEL-Belgian Pro League, and POR-Primeira Liga.
  • Combine with pd.concat, reset index, flatten multi-level columns with "_".join(col), and filter to rows where pos contains "MF".
  • Merge passing columns (Total_Cmp%, KP, PrgP, Att) and defense columns (Tkl, Int, Blocks) on merge_keys, then print summary and save to raw_midfielders.csv.

Scroll up through each code chunk and confirm they're all present in your file.

You now have a real dataset of midfielder stats from across Europe's top leagues. Next up, you'll clean and normalize this raw data so that every stat is on the same 0-1 scale for fair similarity comparisons.

Clean and Normalize the Feature Matrix

You have raw stats for 500+ midfielders, but there is a problem. Pass completion percentage ranges from 0 to 100, while tackles per 90 ranges from 0 to 5.

If you ran a similarity calculation on these raw numbers, pass completion would dominate the result purely because its numbers are larger. You need to normalize everything to a common 0-1 scale so each stat contributes equally to the comparison.

In this step, get ready to:
  • Select 7 per-90 features that define a midfielder's statistical profile.
  • Apply Min-Max normalization to compress every feature into a 0-1 range.
  • Save a clean feature matrix ready for cosine similarity.
Select features and filter by minutes played

Some columns in your CSV are already per-90 values (goals, assists), but others (key passes, tackles) are season totals. You need to convert those totals into per-90 rates before comparing players fairly.

  • Create a new file called build_features.py in your player-scout folder and add this code:
import pandas as pd
from sklearn.preprocessing import MinMaxScaler

# Load raw data
df = pd.read_csv("raw_midfielders.csv")

# Filter: minimum 900 minutes played
df = df[df["Playing Time_Min"] >= 900].copy()

# Calculate per-90 values where not already provided
df["90s"] = df["Playing Time_Min"] / 90

# Select features for similarity comparison
# These capture a midfielder's complete profile
feature_cols = [
    "Per 90 Minutes_Gls",    # Goals per 90
    "Per 90 Minutes_Ast",    # Assists per 90
    "Total_Cmp%",            # Pass completion %
    "KP",                    # Key passes (total, will normalize)
    "PrgP",                  # Progressive passes (total, will normalize)
    "Tkl",                   # Tackles (total, will normalize)
    "Int",                   # Interceptions (total, will normalize)
]

# Convert totals to per-90 where needed
for col in ["KP", "PrgP", "Tkl", "Int"]:
    df[f"{col}_per90"] = df[col] / df["90s"]

What does this code do?

The 900-minute filter removes players with tiny sample sizes. A player with 200 minutes might have extreme per-90 stats from just a few appearances, which would skew similarity results.

The for loop at the bottom divides each player's total key passes, progressive passes, tackles, and interceptions by their 90s value (minutes divided by 90). This converts raw totals into rate stats so a player with 3000 minutes is compared fairly against one with 1000 minutes.

  • Save build_features.py.
  • Confirm the partial script runs without errors by running:
python build_features.py

The script exits silently with no output. That confirms the data loaded and filtered without crashing.

Seeing a FileNotFoundError?

Make sure raw_midfielders.csv exists in the same folder where you are running the command. Run ls to check it is listed.

If you see a ModuleNotFoundError for sklearn, make sure your virtual environment is activated.

help me debug a FileNotFoundError or ModuleNotFoundError in my build_features.py script

Apply Min-Max normalization

Now you need to define the final feature list (using the new per-90 columns) and squash every value into a 0-1 range. MinMaxScaler from scikit-learn does this by mapping each column's minimum to 0 and its maximum to 1.

  • In build_features.py, add this code below the per-90 conversion loop:
# Final feature list (all per-90 or percentage)
features = [
    "Per 90 Minutes_Gls",
    "Per 90 Minutes_Ast",
    "Total_Cmp%",
    "KP_per90",
    "PrgP_per90",
    "Tkl_per90",
    "Int_per90",
]

# Drop rows with any NaN in features
df = df.dropna(subset=features)

# Metadata to keep alongside features
metadata_cols = ["player", "team", "league", "season", "born", "Playing Time_Min"]
metadata = df[metadata_cols].reset_index(drop=True)

What does this code do?

The features list swaps in the new per-90 columns (KP_per90, PrgP_per90, etc.) instead of the raw totals. Every column in this list is now either a per-90 rate or a percentage.

The dropna call removes any player missing data in the selected features. The metadata extraction keeps player names, teams, and leagues separate from the numbers so normalization only touches the stats.

  • Still in build_features.py, add the normalization and save logic below the metadata line:
# Normalize features to 0-1 range
scaler = MinMaxScaler()
normalized = pd.DataFrame(
    scaler.fit_transform(df[features]),
    columns=features
)

# Combine metadata with normalized features
feature_matrix = pd.concat([metadata, normalized], axis=1)

print(f"Feature matrix shape: {feature_matrix.shape}")
print(f"\nFeature ranges (should be 0-1):")
print(feature_matrix[features].describe().loc[["min", "max"]])
print(f"\nSample player:")
print(feature_matrix.iloc[0])

feature_matrix.to_csv("feature_matrix.csv", index=False)
print("\nSaved feature_matrix.csv")

What does this code do?

MinMaxScaler's fit_transform learns the min and max of each column, then maps every value into the 0-1 range. A player with the highest tackles per 90 in the dataset gets a 1.0. A player with the lowest gets a 0.0.

The final pd.concat stitches the player names back onto the normalized numbers, giving you one clean CSV where every feature column is on the same scale.

  • Save build_features.py.

Why Min-Max instead of Z-score normalization?

Min-Max keeps values bounded between 0 and 1, which makes radar charts readable later. Z-score normalization can produce negative values and unbounded ranges, which complicates visualization.

Run the script and verify the output

Before you run the script, think about what you expect to see. Every feature column should have a minimum of exactly 0.0 and a maximum of exactly 1.0. Do you think the player count will be higher or lower than your original 500+ rows?

  • Run your feature builder by running:
python build_features.py

You should see output showing the matrix shape, feature ranges confirming 0.0 min and 1.0 max for every column, and a sample player row with all values between 0 and 1.

  • Confirm that feature_matrix.csv was created by running:
ls

You should see feature_matrix.csv alongside raw_midfielders.csv and build_features.py in the listing.

Seeing NaN values or a KeyError?

A KeyError on a column name means the column in your raw_midfielders.csv does not match what the script expects. Open the CSV and check the exact column headers match (they are case-sensitive).

If the feature matrix shape shows 0 rows, the 900-minute filter or the dropna removed all players. Check that your raw CSV has a Playing Time_Min column with numeric values.

help me debug my build_features.py output

✔️ Awesome, I've got everything!

Great. Double check you have saved build_features.py and that feature_matrix.csv exists in your project folder.

ⓧ I'd like to double check the full code

Here is the complete build_features.py file. Scroll up and confirm each section is present in order: imports, load/filter, feature_cols list, per-90 loop, features list, dropna, metadata, normalization, print/save.

import pandas as pd
from sklearn.preprocessing import MinMaxScaler

# Load raw data
df = pd.read_csv("raw_midfielders.csv")

# Filter: minimum 900 minutes played
df = df[df["Playing Time_Min"] >= 900].copy()

# Calculate per-90 values where not already provided
df["90s"] = df["Playing Time_Min"] / 90

# Select features for similarity comparison
# These capture a midfielder's complete profile
feature_cols = [
    "Per 90 Minutes_Gls",    # Goals per 90
    "Per 90 Minutes_Ast",    # Assists per 90
    "Total_Cmp%",            # Pass completion %
    "KP",                    # Key passes (total, will normalize)
    "PrgP",                  # Progressive passes (total, will normalize)
    "Tkl",                   # Tackles (total, will normalize)
    "Int",                   # Interceptions (total, will normalize)
]
# Convert totals to per-90 where needed
for col in ["KP", "PrgP", "Tkl", "Int"]:
    df[f"{col}_per90"] = df[col] / df["90s"]

# Final feature list (all per-90 or percentage)
features = [
    "Per 90 Minutes_Gls",
    "Per 90 Minutes_Ast",
    "Total_Cmp%",
    "KP_per90",
    "PrgP_per90",
    "Tkl_per90",
    "Int_per90",
]

# Drop rows with any NaN in features
df = df.dropna(subset=features)

# Metadata to keep alongside features
metadata_cols = ["player", "team", "league", "season", "born", "Playing Time_Min"]
metadata = df[metadata_cols].reset_index(drop=True)
# Normalize features to 0-1 range
scaler = MinMaxScaler()
normalized = pd.DataFrame(
    scaler.fit_transform(df[features]),
    columns=features
)

# Combine metadata with normalized features
feature_matrix = pd.concat([metadata, normalized], axis=1)

print(f"Feature matrix shape: {feature_matrix.shape}")
print(f"\nFeature ranges (should be 0-1):")
print(feature_matrix[features].describe().loc[["min", "max"]])
print(f"\nSample player:")
print(feature_matrix.iloc[0])

feature_matrix.to_csv("feature_matrix.csv", index=False)
print("\nSaved feature_matrix.csv")

Your feature matrix is ready. Every midfielder now lives in a 7-dimensional space where each axis is on the same scale. Next up, you will use cosine similarity to measure how close any two players are in that space.

Build the Cosine Similarity Engine

You have a clean feature matrix with every midfielder's playing style reduced to 7 numbers between 0 and 1. The next question is: how do you find which players are most "alike"?

This is where cosine similarity comes in. It measures how alike two players' statistical profiles are regardless of volume, focusing purely on the shape of their style. A score of 1.0 means identical profiles. A score of 0.0 means completely different.

In this step, get ready to:
  • Write a script that loads the feature matrix and looks up any target player.
  • Compute cosine similarity and filter results to young, affordable-league players.
  • Print a ranked table of the top 5 statistical twins.
Write the core similarity script

The script needs to accept a player name, find their row in the feature matrix, and then compare their 7-dimensional profile against every other player in the dataset.

  • Create a new file called find_similar.py in your player-scout folder and add the following code:
import sys
import pandas as pd
from sklearn.metrics.pairwise import cosine_similarity
from datetime import date

def find_similar_players(player_name, max_age=23):
    # Load feature matrix
    df = pd.read_csv("feature_matrix.csv")

    feature_cols = [
        "Per 90 Minutes_Gls",
        "Per 90 Minutes_Ast",
        "Total_Cmp%",
        "KP_per90",
        "PrgP_per90",
        "Tkl_per90",
        "Int_per90",
    ]

    # Find the target player
    mask = df["player"].str.contains(player_name, case=False, na=False)
    if mask.sum() == 0:
        print(f"Player '{player_name}' not found in dataset.")
        return None

    target_idx = mask.idxmax()
    target = df.loc[target_idx]
    print(f"\nTarget: {target['player']} ({target['team']}, {target['league']})")

What does this code do?

This sets up the function skeleton and handles the first job: finding the target player in your dataset.

  • The feature_cols list defines the same 7 features you normalized in the previous step. Both the target and comparison players are compared on these dimensions.
  • The str.contains() lookup with case=False means you can type "rodri" or "Rodri" and it still finds the player.
  • If no match is found, the script exits early with a helpful message instead of crashing.
  • Save find_similar.py.
Compute similarity and apply recruitment filters

Now you need the actual comparison engine. The cosine_similarity function takes the target player's feature vector and compares it against every other player at once, returning a score for each.

  • In find_similar.py, add the following code directly below the print(f"\nTarget: ...") line, still inside the function:
    # Compute cosine similarity
    target_vector = df.loc[[target_idx], feature_cols].values
    all_vectors = df[feature_cols].values
    similarities = cosine_similarity(target_vector, all_vectors)[0]
    df["similarity"] = similarities

    # Calculate age from birth year
    current_year = date.today().year
    df["age"] = current_year - df["born"]

    # Apply filters: under max_age, from affordable leagues, not the target player
    affordable_leagues = ["NED-Eredivisie", "BEL-Belgian Pro League", "POR-Primeira Liga"]
    filtered = df[
        (df["age"] <= max_age) &
        (df["league"].isin(affordable_leagues)) &
        (df.index != target_idx)
    ]

    # Sort by similarity and get top 5
    top5 = filtered.nlargest(5, "similarity")

What does this code do?

This is the core intelligence of your scouting engine.

  • The cosine_similarity call takes the target's (1, 7) feature vector and compares it against all players' (n, 7) matrix. The result is a similarity score between 0 and 1 for every player.
  • Age is calculated by subtracting the birth year from the current year. This keeps the script accurate no matter when you run it.
  • The three filters simulate a real recruitment brief: young (under 23), from leagues where transfer fees are lower, and excluding the target player from their own results.

Why cosine similarity instead of Euclidean distance?

Euclidean distance measures absolute distance between points. If two players have the same style but one plays slightly more minutes, their Euclidean distance grows.

Cosine similarity ignores magnitude and focuses on direction. Two players whose stats form the same "shape" across all 7 dimensions get a high score, even if the raw numbers differ slightly. This is exactly what a scout wants: players who play the same way.

  • Save find_similar.py.
Add formatted output and run the engine

The final piece is the output table and the script entry point that reads the player name from the command line.

  • In find_similar.py, add the following code directly below the top5 = filtered.nlargest(5, "similarity") line:
    print(f"\nTop 5 similar midfielders (U{max_age}, affordable leagues):")
    print("-" * 70)
    for rank, (_, row) in enumerate(top5.iterrows(), 1):
        print(f"{rank}. {row['player']:25s} | {row['team']:15s} | {row['league']:20s} | Age: {int(row['age'])} | Sim: {row['similarity']:.3f}")

    return top5

if __name__ == "__main__":
    player = sys.argv[1] if len(sys.argv) > 1 else "Rodri"
    find_similar_players(player)

What does this code do?

  • The print loop formats each result with the player's name, team, league, age, and similarity score in a clean table.
  • The if __name__ == "__main__" block reads a player name from the command line. If you don't provide one, it defaults to "Rodri".
  • The function also returns the top 5 DataFrame, which you will reuse in the Streamlit app later.
  • Save find_similar.py.

Before you run this, what do you think will happen? Will the script find players similar to Rodri in the Eredivisie, Belgian Pro League, and Primeira Liga?

  • Run the similarity engine by running this command:
python find_similar.py "Rodri"

You should see the target player confirmed, followed by a table of 5 players with similarity scores between 0 and 1. Each result is a young midfielder from one of the three affordable leagues whose statistical profile most closely matches Rodri's.

Not seeing results?

  • If you see "Player 'Rodri' not found in dataset", check that feature_matrix.csv exists in the same folder as find_similar.py. Run ls to confirm.
  • If you see an indentation error, make sure all lines after def find_similar_players are indented with 4 spaces.
  • If the results table is empty, your dataset may not have enough young players from the affordable leagues. Try increasing the age filter by changing max_age=23 to max_age=25 temporarily to confirm the logic works.

Why isn't my find_similar.py producing results?

✔️ Awesome, I've got everything!

Great. Double check you have saved find_similar.py and confirmed the output shows 5 ranked players.

ⓧ I'd like to double check the full code

Here is the complete find_similar.py. Confirm your file matches this exactly:

import sys
import pandas as pd
from sklearn.metrics.pairwise import cosine_similarity
from datetime import date

def find_similar_players(player_name, max_age=23):
    # Load feature matrix
    df = pd.read_csv("feature_matrix.csv")

    feature_cols = [
        "Per 90 Minutes_Gls",
        "Per 90 Minutes_Ast",
        "Total_Cmp%",
        "KP_per90",
        "PrgP_per90",
        "Tkl_per90",
        "Int_per90",
    ]
    # Find the target player
    mask = df["player"].str.contains(player_name, case=False, na=False)
    if mask.sum() == 0:
        print(f"Player '{player_name}' not found in dataset.")
        return None

    target_idx = mask.idxmax()
    target = df.loc[target_idx]
    print(f"\nTarget: {target['player']} ({target['team']}, {target['league']})")

    # Compute cosine similarity
    target_vector = df.loc[[target_idx], feature_cols].values
    all_vectors = df[feature_cols].values
    similarities = cosine_similarity(target_vector, all_vectors)[0]
    df["similarity"] = similarities
    # Calculate age from birth year
    current_year = date.today().year
    df["age"] = current_year - df["born"]

    # Apply filters: under max_age, from affordable leagues, not the target player
    affordable_leagues = ["NED-Eredivisie", "BEL-Belgian Pro League", "POR-Primeira Liga"]
    filtered = df[
        (df["age"] <= max_age) &
        (df["league"].isin(affordable_leagues)) &
        (df.index != target_idx)
    ]

    # Sort by similarity and get top 5
    top5 = filtered.nlargest(5, "similarity")

    print(f"\nTop 5 similar midfielders (U{max_age}, affordable leagues):")
    print("-" * 70)
    for rank, (_, row) in enumerate(top5.iterrows(), 1):
        print(f"{rank}. {row['player']:25s} | {row['team']:15s} | {row['league']:20s} | Age: {int(row['age'])} | Sim: {row['similarity']:.3f}")

    return top5

if __name__ == "__main__":
    player = sys.argv[1] if len(sys.argv) > 1 else "Rodri"
    find_similar_players(player)

Your similarity engine works. You can now type any midfielder's name and instantly get back a shortlist of affordable statistical twins. Next up, you will turn this command-line tool into a visual web app with search, filters, and radar charts.

Build the Streamlit App with Radar Charts

Your similarity engine works in the terminal, but a sporting director needs something they can interact with visually.

In this step, you will turn your find_similar.py logic into a full web app using Streamlit. The app will have a search box, sidebar filters, a results table, and radar charts comparing player profiles.

In this step, get ready to:
  • Create the Streamlit app layout with player search and sidebar filters.
  • Wire up cosine similarity with age, minutes, and league filters.
  • Add radar chart visualizations comparing player profiles.
Create the app layout and data loader

Streamlit apps are just Python scripts that run top-to-bottom. Every widget you add (a text input, a slider, a table) appears in the browser automatically.

  • Create a new file called app.py in your player-scout folder and add the following code:
import streamlit as st
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.metrics.pairwise import cosine_similarity
from datetime import date
from math import pi

st.set_page_config(page_title="Player Recruitment Engine", layout="wide")
st.title("Player Recruitment Engine")
st.markdown("Find affordable statistical twins for any midfielder across Europe's top leagues.")

# Load data
@st.cache_data
def load_data():
    return pd.read_csv("feature_matrix.csv")

df = load_data()

What does this code do?

  • The imports bring in Streamlit for the web app, pandas for data, matplotlib for charts, cosine_similarity for the matching logic, and pi for the radar chart angles.
  • st.set_page_config sets a wide layout so radar charts have room to breathe.
  • @st.cache_data caches the CSV load so the app does not re-read the file on every interaction.
  • Save app.py.
  • Launch the app by running:
streamlit run app.py

Your browser opens to localhost:8501 and you should see the title "Player Recruitment Engine" with the subtitle below it.

What about the "Always rerun" button?

Streamlit detects file changes and shows an "Always rerun" button in the top-right corner. Click it so the app refreshes automatically whenever you save.

Page is blank or shows an error?

  • Make sure feature_matrix.csv is in the same folder as app.py. The load_data() function reads from the current directory.
  • Confirm your virtual environment is active. Streamlit must be installed in the same environment you are running from.

Help me debug why my Streamlit app is not loading.

Now add the feature definitions, sidebar filters, and search input below your existing code in app.py:

  • Add the following code below the df = load_data() line:
feature_cols = [
    "Per 90 Minutes_Gls",
    "Per 90 Minutes_Ast",
    "Total_Cmp%",
    "KP_per90",
    "PrgP_per90",
    "Tkl_per90",
    "Int_per90",
]

feature_labels = [
    "Goals/90",
    "Assists/90",
    "Pass Cmp%",
    "Key Passes/90",
    "Prog Passes/90",
    "Tackles/90",
    "Interceptions/90",
]

# Sidebar filters
st.sidebar.header("Filters")
max_age = st.sidebar.slider("Maximum age", 18, 30, 23)
min_minutes = st.sidebar.slider("Minimum minutes played", 500, 2500, 900, step=100)

# Player search
player_name = st.text_input("Enter a midfielder's name (e.g., Rodri, Pedri, De Bruyne):")

What does this code do?

  • feature_cols lists the normalized columns used for similarity math.
  • feature_labels provides human-readable names for radar chart axes.
  • The sidebar sliders let a sporting director adjust filters without touching code. The defaults (age 23, 900 minutes) match the filters you used in find_similar.py.
  • Save app.py and check your browser.

You should now see a sidebar with two sliders (Maximum age and Minimum minutes played) and a text input field in the main area.

Sidebar not appearing?

  • Make sure your code uses st.sidebar.slider (not st.slider). Without the sidebar prefix, widgets appear in the main column.
  • If you see a traceback mentioning feature_matrix.csv columns, confirm the column names in your CSV match the strings in feature_cols exactly.

Why is my Streamlit sidebar not showing up?

Wire up the similarity engine

The search box is live but does nothing yet. You need to hook it up to the same cosine similarity logic from your CLI script.

  • Add the following code below the player_name = st.text_input(...) line in app.py:
if player_name:
    mask = df["player"].str.contains(player_name, case=False, na=False)

    if mask.sum() == 0:
        st.error(f"Player '{player_name}' not found. Try a different spelling.")
    else:
        target_idx = mask.idxmax()
        target = df.loc[target_idx]
        st.success(f"Target: **{target['player']}** ({target['team']}, {target['league']})")

        # Compute similarities
        target_vector = df.loc[[target_idx], feature_cols].values
        all_vectors = df[feature_cols].values
        similarities = cosine_similarity(target_vector, all_vectors)[0]
        df_results = df.copy()
        df_results["similarity"] = similarities

        # Calculate age
        current_year = date.today().year
        df_results["age"] = current_year - df_results["born"]

What does this code do?

  • The if player_name: block only fires when the user types something in the search box.
  • str.contains does a case-insensitive partial match, so typing "rodri" finds "Rodri" and "Rodrigo".
  • st.success shows a green banner confirming which player was matched.
  • The similarity computation is identical to what find_similar.py does. You compare the target's 7-feature vector against every other player.
  • Save app.py and type Rodri in the search box.

You should see a green success banner showing the target player's name, team, and league.

Now add the filtering and results table. Add this code directly below the age calculation you just wrote:

  • Add the following code below the df_results["age"] = current_year - df_results["born"] line, keeping the same indentation level (inside the else block):
        # Apply filters
        affordable_leagues = ["NED-Eredivisie", "BEL-Belgian Pro League", "POR-Primeira Liga"]
        filtered = df_results[
            (df_results["age"] <= max_age) &
            (df_results["Playing Time_Min"] >= min_minutes) &
            (df_results["league"].isin(affordable_leagues)) &
            (df_results.index != target_idx)
        ]

        top5 = filtered.nlargest(5, "similarity")

        if top5.empty:
            st.warning("No matching players found. Try increasing the age limit.")
        else:
            # Results table
            st.subheader("Top 5 Statistical Twins")
            display_df = top5[["player", "team", "league", "age", "similarity"]].copy()
            display_df.columns = ["Player", "Team", "League", "Age", "Similarity"]
            display_df["Similarity"] = display_df["Similarity"].apply(lambda x: f"{x:.3f}")
            display_df = display_df.reset_index(drop=True)
            display_df.index = display_df.index + 1
            st.table(display_df)

What does this code do?

  • The filter restricts results to under-max_age players from the three "affordable" leagues who have played at least min_minutes minutes.
  • nlargest(5, "similarity") picks the top 5 most similar players after filtering.
  • The display DataFrame reformats column names and truncates similarity scores to 3 decimal places for readability.
  • Save app.py and type Rodri in the search box again.

You should see the green success banner followed by a "Top 5 Statistical Twins" table showing 5 players with their team, league, age, and similarity score.

Seeing "No matching players found"?

  • Try increasing the Maximum age slider in the sidebar to 25 or 28. If your dataset has fewer young midfielders in the three affordable leagues, the default filter of 23 may be too strict.
  • Confirm your feature_matrix.csv includes players from NED-Eredivisie, BEL-Belgian Pro League, and POR-Primeira Liga. Open the CSV and check the league column.

Help me figure out why my player similarity results are empty.

Add radar chart visualizations

The table tells you who the twins are, but a radar chart shows exactly where the profiles overlap and diverge. Each axis represents one of the 7 normalized features.

  • Add the following code inside the else block, directly below the st.table(display_df) line (keeping the same indentation):
            # Radar charts
            st.subheader("Profile Comparison (Radar Charts)")

            def draw_radar(target_vals, comp_vals, target_name, comp_name, similarity):
                categories = feature_labels
                n = len(categories)
                angles = [x / float(n) * 2 * pi for x in range(n)]
                angles += angles[:1]

                target_vals_plot = list(target_vals) + [target_vals[0]]
                comp_vals_plot = list(comp_vals) + [comp_vals[0]]

                fig, ax = plt.subplots(figsize=(4, 4), subplot_kw=dict(polar=True))
                ax.plot(angles, target_vals_plot, "o-", linewidth=1.5, label=target_name, color="#e74c3c")
                ax.fill(angles, target_vals_plot, alpha=0.1, color="#e74c3c")
                ax.plot(angles, comp_vals_plot, "o-", linewidth=1.5, label=comp_name, color="#3498db")
                ax.fill(angles, comp_vals_plot, alpha=0.1, color="#3498db")

                ax.set_xticks(angles[:-1])
                ax.set_xticklabels(categories, size=7)
                ax.set_ylim(0, 1)
                ax.legend(loc="upper right", bbox_to_anchor=(1.3, 1.1), fontsize=7)
                ax.set_title(f"Similarity: {similarity:.3f}", size=9, pad=15)

                return fig

How does the radar chart work?

  • A radar chart plots values on spokes radiating from a center point. Each spoke is one feature (goals, tackles, interceptions, etc.).
  • subplot_kw=dict(polar=True) tells matplotlib to use polar coordinates, which creates the circular layout.
  • The target player is drawn in red (#e74c3c) and the comparison player in blue (#3498db). Where the two shapes overlap, the players have similar strengths.
  • angles += angles[:1] closes the polygon by wrapping back to the first point.
  • Now add the loop that renders one chart per twin. Add this below the draw_radar function, at the same indentation level:
            target_vals = df.loc[target_idx, feature_cols].values

            cols = st.columns(min(len(top5), 3))
            for i, (_, row) in enumerate(top5.iterrows()):
                col_idx = i % 3
                comp_vals = row[feature_cols].values
                fig = draw_radar(
                    target_vals, comp_vals,
                    target["player"], row["player"],
                    row["similarity"]
                )
                with cols[col_idx]:
                    st.pyplot(fig)
                    plt.close(fig)

What does this loop do?

  • st.columns(min(len(top5), 3)) creates a 3-column grid so charts sit side by side.
  • col_idx = i % 3 wraps charts into rows of 3. If there are 5 results, you get 3 on the first row and 2 on the second.
  • plt.close(fig) frees memory after each chart is rendered. Without it, matplotlib holds all figures open.
  • Save app.py.

Before you test, what do you expect to see below the results table when you search for a player?

  • Type Rodri in the search box (or re-submit if it is still there).

You should see 5 radar charts below the table. Each chart overlays the target player (red) and one twin (blue) on 7 axes. The more the shapes overlap, the more similar the two players are.

Charts not appearing or showing errors?

  • Check that the draw_radar function and the loop are both inside the else block (indented 12 spaces from the left). Python is strict about indentation.
  • If you see "list index out of range", confirm your feature_labels list has exactly 7 items matching the 7 items in feature_cols.
  • If charts appear but are blank circles, check that feature_matrix.csv has values between 0 and 1 in the feature columns (not all zeros).

Help me debug why my radar charts are not rendering in Streamlit.

✔️ Awesome, I've got everything!

Great. Double check you have saved app.py and your Streamlit app is still running at localhost:8501.

ⓧ I'd like to double check the full code

Your app.py is over 30 lines, so confirm you have each of these sections in order by scrolling through the file:

  • Imports (streamlit, pandas, numpy, matplotlib, cosine_similarity, date, pi).
  • Page config, title, and markdown description.
  • @st.cache_data load_data() function reading feature_matrix.csv.
  • feature_cols (7 items) and feature_labels (7 human-readable names).
  • Sidebar sliders for max_age and min_minutes.
  • player_name text input.
  • if player_name: block with player lookup, error handling, and st.success banner.
  • Cosine similarity computation and age calculation.
  • Filter block (affordable_leagues, max_age, min_minutes) and top5 selection.
  • st.table(display_df) showing the results.
  • draw_radar() function with polar axes, red target, blue comparison.
  • Loop over top5 rendering charts in st.columns(min(len(top5), 3)).

You now have a fully interactive recruitment tool. A sporting director can type any midfielder's name, tweak the age and minutes filters, and instantly see who the closest statistical twins are. The radar charts give visual proof of similarity at a glance.

Secret mission

Add a PCA Scatter Map

Your radar charts compare players one at a time. But what if you could see ALL midfielders on a single map, with your target and their twins clustered together? PCA dimensionality reduction compresses your 7-feature space into 2D, revealing hidden groupings in playing style.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally, so there are no ongoing costs.

Resources you used:

  • The player-scout project folder on your Desktop (contains your virtual environment, Python scripts, and CSV data files).
  • The ~/soccerdata/config/league_dict.json configuration file for custom league mappings.

Keep everything running

No action needed. Choose this if you're still actively building or want to run the Streamlit app again later.

  • Your player-scout folder stays on your Desktop with all scripts, data, and your virtual environment intact.
  • The soccerdata config at ~/soccerdata/config/league_dict.json remains in place so you can scrape custom leagues anytime.
  • To relaunch the app later, activate your virtual environment and run streamlit run app.py from inside the project folder.

Pause - I'll come back to this later

Shut down any running processes to free up memory. Your files stay on disk for when you return.

  • If Streamlit is still running in your terminal, press Ctrl+C to stop it.
  • Deactivate your virtual environment by running:
deactivate

Everything else stays in place. When you come back, navigate to player-scout, activate the virtual environment, and pick up where you left off.

Delete - I don't want to use this again

Remove all project resources and start fresh. This deletes your scripts, data files, virtual environment, and custom league configuration permanently.

  • If Streamlit is still running, press Ctrl+C to stop it.
  • Deactivate your virtual environment if it is active:
deactivate
  • Delete the project folder by running:
cd ~/Desktop
rm -rf player-scout
  • Delete the soccerdata configuration directory by running:
rm -rf ~/soccerdata
  • Confirm everything is gone by running:
ls ~/Desktop | grep player-scout
ls ~/soccerdata

Both commands should return nothing or show "No such file or directory". Your machine is back to its original state.

Nice Work!

Nice Work!

You just built a full data-driven player recruitment engine from scratch. Type any midfielder's name into your app and get back a ranked shortlist of affordable statistical twins, complete with radar charts proving the match.

You learned how to:

  • Built a data pipeline that scrapes FBref stats for 7 European leagues and normalizes per-90 midfielder metrics into a clean feature matrix.
  • Implemented cosine similarity to find statistically similar players, filtered by age and league affordability.
  • Created an interactive Streamlit web app with player search, results table, and radar chart visualizations comparing player profiles.
  • Secret Mission: Added PCA dimensionality reduction to visualize player similarity clusters on a 2D scatter map.

Ready to quiz yourself?