Supervised Learning: Football Predictions

Train 6 classifiers on imbalanced football data and learn why feature engineering beats model tuning.

Introduction

30 Second Summary

When you watch a football match, you instinctively weigh factors like home advantage, recent form, and team quality to predict the outcome. Machine learning algorithms do the same thing, but they can process hundreds of features at once and learn patterns invisible to the human eye.

In this project, you will train six different classification algorithms on real international football match data, pit them against each other, and visualize exactly how and why each one performs differently.

What You'll Build

A complete supervised learning benchmarking system that predicts international football match outcomes using historical data and compares six classification algorithms head-to-head.

By the end of this project, you'll have:

  • A trained pipeline of 6 supervised learning classifiers (Logistic Regression, KNN, Decision Tree, Random Forest, SVM, XGBoost) benchmarked against each other on a real-world sports dataset.
  • A visual comparison dashboard showing accuracy, F1 scores, and confusion matrices for each algorithm.
  • A feature importance analysis revealing how different algorithms prioritize different input features.
  • Secret Mission: Squeeze more performance out of your top models with hyperparameter tuning using GridSearchCV.

Are there any prerequisites?

This project assumes basic Python proficiency and some familiarity with pandas and matplotlib. You do not need prior machine learning experience.

Before We Start

Predicting football match outcomes is a classic challenge in machine learning. No single algorithm dominates every dataset, so the only way to know which model fits best is to train several and compare them head-to-head.

In this step, you will set up a clean Python environment with all the libraries needed for the project. By the end, you will have a dedicated virtual environment, every required package installed, and an empty script file ready for code.

In this step, get ready to:
  • Create a Python 3.13 virtual environment for the project.
  • Install all required ML and data packages.
  • Create the project file and open it in VS Code.
Create a virtual environment with Python 3.13

A virtual environment isolates your project's packages from the rest of your system. This prevents version conflicts between projects.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal and press Enter to open it.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir football-ml
cd football-ml
  • Create the virtual environment by running this command:
python3.13 -m venv venv

What does this command do?

python3.13 -m venv venv tells Python 3.13 specifically to create a virtual environment in a folder called venv. Using python3.13 ensures the environment uses the correct Python version, even if other versions are installed on your machine.

  • Activate the virtual environment:

macOS

  • Run this command to activate:
source venv/bin/activate

Windows

  • Run this command to activate:
venv\Scripts\activate

You should see (venv) appear at the beginning of your terminal prompt. This confirms the virtual environment is active.

Don't see (venv) in your prompt?

Make sure you are inside the football-ml folder before running the activate command. Run ls and confirm you see a venv folder listed.

On Windows PowerShell, you may need to allow script execution first by running Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser before retrying the activate command.

help me activate my Python virtual environment

Install the required packages

This project uses several libraries across data handling, machine learning, and visualization. You will install them all in one command.

  • Install all project dependencies by running this command:
pip install scikit-learn xgboost seaborn huggingface_hub pandas matplotlib numpy

What are these packages?

  • scikit-learn provides the classification algorithms (Logistic Regression, KNN, Decision Tree, Random Forest, SVM) and evaluation tools.
  • xgboost is a powerful gradient boosting library for the sixth model in our comparison.
  • seaborn creates statistical visualizations on top of matplotlib.
  • huggingface_hub lets you download datasets directly from Hugging Face.
  • pandas and numpy handle data loading and numeric operations.
  • matplotlib is the base plotting library used by seaborn and for custom charts.

The installation may take a minute or two. Wait until you see the terminal return to the (venv) prompt with no errors.

Seeing an installation error?

If xgboost fails to install, confirm your virtual environment is using Python 3.12 or higher. Run python --version inside the active venv. xgboost requires Python 3.12 or above.

help me fix a pip install error in my Python virtual environment

Create the project file and open VS Code
  • Create an empty Python file for your project by running this command:
touch match_predictor.py
  • Confirm the file exists by running:
ls

You should see match_predictor.py and the venv folder listed in the output.

  • Open the project folder in VS Code by running this command:
code .
  • In VS Code, click on match_predictor.py in the Explorer sidebar to open the file.

Why start with an empty file?

You will build match_predictor.py incrementally across the remaining steps. Starting with an empty file lets you add each section (data loading, preprocessing, training, evaluation) one piece at a time and verify as you go.

VS Code did not open?

The code command needs to be installed in your system PATH. In VS Code, press Cmd+Shift+P (macOS) or Ctrl+Shift+P (Windows), type Shell Command: Install 'code' command in PATH, and select it. Then retry code . in your terminal.

help me open VS Code from the terminal

Your environment is locked and loaded. Next up, you will explore the football dataset and prepare it for the six machine learning models.

Download and Load the Dataset

Your environment is set up with all the libraries you need. Now it's time to bring in some real data.

You'll download a dataset of approximately 49,000 international football matches from Hugging Face. This dataset contains 191 engineered features covering ELO ratings, FIFA rankings, team form, squad values, and head-to-head records.

In this step, get ready to:
  • Import all required libraries for the full project.
  • Download the football predictions dataset from Hugging Face.
  • Load the data into a pandas DataFrame and verify its structure.
Add the project imports

Machine learning projects typically use many libraries working together. You will import everything upfront so the script has access to all the tools it needs throughout the remaining steps.

  • In match_predictor.py, add the following import block at the top of the file:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from huggingface_hub import hf_hub_download
from sklearn.model_selection import train_test_split, cross_val_score, StratifiedKFold
from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.metrics import (
    classification_report,
    confusion_matrix,
    ConfusionMatrixDisplay,
    accuracy_score,
    f1_score,
)
from xgboost import XGBClassifier

What do these imports do?

These imports pull in the full toolkit you will use across all steps of this project:

  • numpy and pandas handle numerical arrays and tabular data.
  • matplotlib and seaborn produce charts and visualizations.
  • huggingface_hub downloads the dataset directly from Hugging Face's servers.
  • sklearn modules provide the classification algorithms, preprocessing tools, and evaluation metrics.
  • xgboost adds the XGBoost gradient boosting classifier.
  • Save match_predictor.py.

You won't see any output yet. The next chunk uses these imports to download and load the dataset.

Download the dataset from Hugging Face

The hf_hub_download function downloads a single file from a Hugging Face repository and caches it on your disk. You will point it at the football predictions dataset and load the resulting CSV into a pandas DataFrame.

  • Below your imports, add the following code to download and load the dataset:


# ============================================================
# STEP 1: Download and Load Dataset
# ============================================================
print("Downloading dataset from Hugging Face...")
file_path = hf_hub_download(
    repo_id="adibmed/football-predictions-dataset",
    filename="output/01_matches_all.csv",
    repo_type="dataset",
)
df = pd.read_csv(file_path, low_memory=False)
print(f"Dataset shape: {df.shape}")
print(f"Columns: {df.shape[1]}")
print(df.head())

What does this code do?

hf_hub_download reaches out to Hugging Face and pulls the CSV file output/01_matches_all.csv from the adibmed/football-predictions-dataset repository. Setting repo_type="dataset" tells the library this is a dataset repo, not a model repo.

pd.read_csv loads the downloaded file into a DataFrame called df. The low_memory=False flag ensures pandas reads the entire file in one pass, which avoids mixed-type column warnings on large datasets.

  • Save match_predictor.py.
  • Run the script in your terminal by running:
python match_predictor.py

You should see output showing the dataset shape (approximately 49,000 rows and 191 columns), the column count, and the first five rows of data.

Seeing a connection error?

  • Make sure you have an active internet connection. The first run downloads approximately 113 MB from Hugging Face.
  • Check that your virtual environment is activated. You should see (venv) or similar at the start of your terminal prompt.
  • If you see a ModuleNotFoundError, re-run your pip install command from the previous step to make sure all packages installed correctly.

Still stuck? Help me debug why my script can't download the dataset from Hugging Face.

✔️ Awesome, I've got everything!

Great. Double check you have saved your file before moving on.

ⓧ I'd like to double check the full code

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from huggingface_hub import hf_hub_download
from sklearn.model_selection import train_test_split, cross_val_score, StratifiedKFold
from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.metrics import (
    classification_report,
    confusion_matrix,
    ConfusionMatrixDisplay,
    accuracy_score,
    f1_score,
)
from xgboost import XGBClassifier


# ============================================================
# STEP 1: Download and Load Dataset
# ============================================================
print("Downloading dataset from Hugging Face...")
file_path = hf_hub_download(
    repo_id="adibmed/football-predictions-dataset",
    filename="output/01_matches_all.csv",
    repo_type="dataset",
)
df = pd.read_csv(file_path, low_memory=False)
print(f"Dataset shape: {df.shape}")
print(f"Columns: {df.shape[1]}")
print(df.head())

Your data is loaded and ready for exploration. Next up, you will dig into the dataset structure, clean up missing values, and prepare the features for training.

Explore and Preprocess the Data

You have a raw dataset loaded with nearly 49,000 international football matches and 191 columns. Before any machine learning model can learn from this data, you need to understand its structure and clean it up.

In this step, you will explore what the dataset contains, derive a prediction target (home win, draw, or away win), remove incomplete data, and prepare clean feature matrices that your models can train on.

In this step, get ready to:
  • Explore the dataset and visualize the target class distribution.
  • Select numeric features, drop high-null columns, and encode the target variable.
  • Split into training and test sets, then scale features for distance-based algorithms.
Derive the match result target column

Every classification problem needs a target variable. This dataset may not have an explicit "result" column, so you need to derive one from the match scores.

  • In match_predictor.py, scroll to the bottom of your file (below the Step 1 code) and add this section header and target derivation logic:


# ============================================================
# STEP 2: EDA and Preprocessing
# ============================================================

# Check target column - derive from scores if no 'result' column exists
if "result" not in df.columns:
    home_score_col = [c for c in df.columns if "home" in c.lower() and "score" in c.lower() or "home" in c.lower() and "goal" in c.lower() and "avg" not in c.lower()]
    away_score_col = [c for c in df.columns if "away" in c.lower() and "score" in c.lower() or "away" in c.lower() and "goal" in c.lower() and "avg" not in c.lower()]

    if home_score_col and away_score_col:
        df["result"] = np.where(
            df[home_score_col[0]] > df[away_score_col[0]], "home_win",
            np.where(df[home_score_col[0]] < df[away_score_col[0]], "away_win", "draw")
        )
    else:
        raise ValueError("Cannot find score columns to derive match result.")

What does this code do?

This code searches the DataFrame for columns containing home/away score or goal data. If the home score is higher, the result is home_win. If the away score is higher, it is away_win. Otherwise, it is a draw.

The np.where() function works like an if/else applied to every row at once. This gives you a 3-class target for your classification models.

You will not see any output yet. The next chunk prints the distribution to confirm this column was created correctly.

Visualize the target class distribution

Before training models, you need to check whether your three classes are balanced. An imbalanced dataset (e.g. far more home wins than draws) can cause models to favour the majority class.

  • Add this code directly below the target derivation block to print and visualize the class distribution:
# Visualize target distribution
print("\nTarget distribution:")
print(df["result"].value_counts())
print(df["result"].value_counts(normalize=True).round(3))

plt.figure(figsize=(8, 5))
sns.countplot(x="result", data=df, order=["home_win", "draw", "away_win"])
plt.title("Match Outcome Distribution")
plt.xlabel("Result")
plt.ylabel("Count")
plt.tight_layout()
plt.savefig("target_distribution.png", dpi=150)
plt.close()

What does this code do?

First, it prints both the raw counts and the proportions (normalized) for each class. This tells you at a glance how balanced your data is.

Then it uses seaborn's countplot to create a bar chart. The chart is saved as target_distribution.png so you can reference it later.

  • Save match_predictor.py and run it in your terminal:
python match_predictor.py
  • You should see the target distribution printed in your terminal, showing counts for home_win, draw, and away_win.
  • Check your project folder. A file called target_distribution.png should now exist.

What should I expect from the distribution?

Football matches historically show roughly 45% home wins, 25% draws, and 30% away wins. Draws are the minority class, which means your models will likely struggle most with predicting draws accurately.

Seeing a KeyError or empty chart?

Make sure the target derivation code (the if "result" not in df.columns block) appears before the visualization code. Python runs top to bottom, so the result column must exist before you try to plot it.

Help me debug my target distribution code.

Select numeric features and drop incomplete columns

The dataset has 191 columns, but many are non-numeric (team names, dates, tournament identifiers) or have large gaps of missing data. Machine learning models need clean, numeric input.

  • Add this code below the visualization block to filter down to usable numeric features:
# Select numeric features only
numeric_df = df.select_dtypes(include=[np.number])
print(f"\nNumeric columns available: {numeric_df.shape[1]}")

# Drop columns with >30% missing values
threshold = 0.3
null_pct = numeric_df.isnull().mean()
cols_to_keep = null_pct[null_pct <= threshold].index.tolist()
numeric_df = numeric_df[cols_to_keep]
print(f"Columns after dropping high-null: {numeric_df.shape[1]}")

What does this code do?

First, select_dtypes(include=[np.number]) keeps only numeric columns, automatically excluding text like team names and dates.

Then it calculates what percentage of each column is null. Any column with more than 30% missing data gets dropped. Columns with too many gaps would introduce noise or force you to impute values that could mislead the model.

  • Save and run the script. You should see two printed lines showing how many numeric columns were available and how many survived the 30% threshold.

Seeing 0 columns after dropping?

Check that you typed the threshold as 0.3 (30%), not 30. The isnull().mean() method returns a fraction between 0 and 1, not a percentage.

Help me debug my feature selection code.

Prepare the feature matrix and encode the target

With clean columns selected, you need to drop any remaining rows with missing values and convert the text labels (home_win, draw, away_win) into numbers that scikit-learn can work with.

  • Add this code below the column filtering block to finalize your feature matrix and encode the target:
# Drop any remaining rows with missing values
numeric_df = numeric_df.dropna()
df_clean = df.loc[numeric_df.index].copy()
df_clean[cols_to_keep] = numeric_df

# Prepare features and target
X = df_clean[cols_to_keep].values
le = LabelEncoder()
y = le.fit_transform(df_clean["result"])
class_names = le.classes_
print(f"\nClasses: {class_names}")
print(f"Final dataset: {X.shape[0]} samples, {X.shape[1]} features")

What does this code do?

The dropna() call removes any rows that still have missing values after column filtering. Then df_clean keeps only the rows that survived, preserving alignment between features and labels.

LabelEncoder converts text labels into integers (0, 1, 2). The class_names array stores the mapping so you can convert predictions back to human-readable labels later.

  • Save and run the script. You should see the three class names printed (alphabetically: away_win, draw, home_win) along with the final sample and feature counts.

Seeing a KeyError on "result"?

The df_clean DataFrame must still have the result column from the derivation step. Make sure the df_clean = df.loc[numeric_df.index].copy() line uses df (the full DataFrame), not numeric_df.

Help me debug my LabelEncoder setup.

Split into training and test sets, then scale features

You need a held-out test set that the models never see during training. This gives you an honest measure of how well each algorithm generalises to new matches.

  • Add this code below the feature preparation block to split your data and scale it:
# Train/test split with stratification
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
print(f"Training set: {X_train.shape[0]} samples")
print(f"Test set: {X_test.shape[0]} samples")

# Scale features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

Why stratify and scale?

The stratify=y parameter ensures both training and test sets have the same class proportions as the full dataset. Without this, you could end up with almost no draws in the test set, making evaluation unreliable.

StandardScaler transforms features to have zero mean and unit variance. Distance-based algorithms like KNN and SVM are sensitive to feature magnitudes. A feature ranging from 0 to 1000 would dominate one ranging from 0 to 1. Tree-based models do not need scaling, so you will keep both scaled and unscaled versions.

Why fit on training data only?

Notice that fit_transform is called on X_train but only transform is called on X_test. If you fit the scaler on test data too, you would leak future information into your preprocessing. This is called data leakage and it inflates your accuracy estimates.

  • Save match_predictor.py and run the script. You should see the training and test set sizes printed (an 80/20 split of your final sample count).

Seeing a ValueError about class counts?

If stratification fails, it usually means one class has fewer samples than the number of splits. Check that your data cleaning did not accidentally drop all rows for one class. Re-run and check the class distribution printed earlier.

Help me fix my train/test split error.

✔️ Awesome, I've got everything!

Great. Double check you have saved your file before moving on.

ⓧ I'd like to double check the full code

Here is the complete match_predictor.py file up to this point:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from huggingface_hub import hf_hub_download
from sklearn.model_selection import train_test_split, cross_val_score, StratifiedKFold
from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.metrics import (
    classification_report,
    confusion_matrix,
    ConfusionMatrixDisplay,
    accuracy_score,
    f1_score,
)
from xgboost import XGBClassifier


# ============================================================
# STEP 1: Download and Load Dataset
# ============================================================
print("Downloading dataset from Hugging Face...")
file_path = hf_hub_download(
    repo_id="adibmed/football-predictions-dataset",
    filename="output/01_matches_all.csv",
    repo_type="dataset",
)
df = pd.read_csv(file_path, low_memory=False)
print(f"Dataset shape: {df.shape}")
print(f"Columns: {df.shape[1]}")
print(df.head())


# ============================================================
# STEP 2: EDA and Preprocessing
# ============================================================

# Check target column - derive from scores if no 'result' column exists
if "result" not in df.columns:
    home_score_col = [c for c in df.columns if "home" in c.lower() and "score" in c.lower() or "home" in c.lower() and "goal" in c.lower() and "avg" not in c.lower()]
    away_score_col = [c for c in df.columns if "away" in c.lower() and "score" in c.lower() or "away" in c.lower() and "goal" in c.lower() and "avg" not in c.lower()]

    if home_score_col and away_score_col:
        df["result"] = np.where(
            df[home_score_col[0]] > df[away_score_col[0]], "home_win",
            np.where(df[home_score_col[0]] < df[away_score_col[0]], "away_win", "draw")
        )
    else:
        raise ValueError("Cannot find score columns to derive match result.")

# Visualize target distribution
print("\nTarget distribution:")
print(df["result"].value_counts())
print(df["result"].value_counts(normalize=True).round(3))

plt.figure(figsize=(8, 5))
sns.countplot(x="result", data=df, order=["home_win", "draw", "away_win"])
plt.title("Match Outcome Distribution")
plt.xlabel("Result")
plt.ylabel("Count")
plt.tight_layout()
plt.savefig("target_distribution.png", dpi=150)
plt.close()

# Select numeric features only
numeric_df = df.select_dtypes(include=[np.number])
print(f"\nNumeric columns available: {numeric_df.shape[1]}")

# Drop columns with >30% missing values
threshold = 0.3
null_pct = numeric_df.isnull().mean()
cols_to_keep = null_pct[null_pct <= threshold].index.tolist()
numeric_df = numeric_df[cols_to_keep]
print(f"Columns after dropping high-null: {numeric_df.shape[1]}")

# Drop any remaining rows with missing values
numeric_df = numeric_df.dropna()
df_clean = df.loc[numeric_df.index].copy()
df_clean[cols_to_keep] = numeric_df

# Prepare features and target
X = df_clean[cols_to_keep].values
le = LabelEncoder()
y = le.fit_transform(df_clean["result"])
class_names = le.classes_
print(f"\nClasses: {class_names}")
print(f"Final dataset: {X.shape[0]} samples, {X.shape[1]} features")

# Train/test split with stratification
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
print(f"Training set: {X_train.shape[0]} samples")
print(f"Test set: {X_test.shape[0]} samples")

# Scale features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

Your data is clean, encoded, split, and scaled. Next up, you will train all six classification algorithms on this prepared dataset and compare how they perform.

Train All Six Models

Your data is clean, split, and scaled. Now comes the core of any machine learning comparison project: training multiple algorithms on the same dataset to see which one performs best.

In this step, you will define six different classification algorithms, evaluate each one using cross-validation, and collect their predictions on the held-out test set. By training all six on the same data, you get a fair head-to-head comparison.

In this step, get ready to:
  • Define all 6 classifiers with sensible default hyperparameters.
  • Train each model using 5-fold stratified cross-validation.
  • Fit each model on the full training set and generate test predictions.
Define all six classifiers

Each algorithm approaches classification differently. Logistic Regression draws linear decision boundaries. KNN looks at nearby data points. Decision Trees split on feature thresholds. Random Forest and XGBoost combine many trees for robustness. SVM finds the optimal separating hyperplane.

You will store all six in a dictionary so the training loop can iterate over them uniformly.

  • In match_predictor.py, add the following code at the bottom of the file (after the X_test_scaled = scaler.transform(X_test) line):
# ============================================================
# STEP 3: Train All Six Models
# ============================================================
print("\n" + "=" * 60)
print("TRAINING ALL MODELS")
print("=" * 60)

# Define all 6 classifiers in a dictionary for uniform training
models = {
    "Logistic Regression": LogisticRegression(max_iter=1000, random_state=42),
    "KNN": KNeighborsClassifier(n_neighbors=5),
    "Decision Tree": DecisionTreeClassifier(random_state=42),
    "Random Forest": RandomForestClassifier(n_estimators=100, random_state=42),
    "SVM": SVC(probability=True, random_state=42),
    "XGBoost": XGBClassifier(
        n_estimators=100, eval_metric="mlogloss", random_state=42
    ),
}

# Models that need scaled data (distance or gradient based)
scaled_models = {"Logistic Regression", "KNN", "SVM"}

# Set up 5-fold stratified cross-validation
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
results = {}

What does this code do?

The models dictionary maps a human-readable name to each classifier instance. This lets you loop through all six with a single for loop later.

The scaled_models set identifies which algorithms need the scaled version of the data. Logistic Regression, KNN, and SVM are sensitive to feature magnitudes because they use distance or gradient calculations. Tree-based models are scale-invariant, so they use the raw data.

The StratifiedKFold object ensures each fold preserves the class distribution (home win, draw, away win) during cross-validation. This prevents any single fold from being unrepresentative.

Why these specific hyperparameters?

Each model uses sensible defaults that work well out of the box. max_iter=1000 gives Logistic Regression enough iterations to converge. n_estimators=100 gives Random Forest and XGBoost enough trees without excessive training time.

Setting probability=True on SVM enables probability estimates (needed for ROC curves later). The random_state=42 ensures reproducible results every time you run the script.

Train and evaluate each model with cross-validation

Cross-validation trains each model multiple times on different slices of the training data. This gives you a more reliable performance estimate than a single train/test split alone.

  • Add the training loop below the code you just wrote:
for name, model in models.items():
    print(f"\nTraining {name}...")
    # Use scaled data for distance/gradient models, raw for trees
    X_tr = X_train_scaled if name in scaled_models else X_train

    # Cross-validation scores
    cv_acc = cross_val_score(model, X_tr, y_train, cv=cv, scoring="accuracy")
    cv_f1 = cross_val_score(model, X_tr, y_train, cv=cv, scoring="f1_macro")

    # Fit on full training set
    model.fit(X_tr, y_train)

    # Predict on test set
    X_te = X_test_scaled if name in scaled_models else X_test
    y_pred = model.predict(X_te)

    # Store all metrics for comparison
    results[name] = {
        "cv_accuracy_mean": cv_acc.mean(),
        "cv_accuracy_std": cv_acc.std(),
        "cv_f1_mean": cv_f1.mean(),
        "cv_f1_std": cv_f1.std(),
        "test_accuracy": accuracy_score(y_test, y_pred),
        "test_f1": f1_score(y_test, y_pred, average="macro"),
        "y_pred": y_pred,
    }

    print(f"  CV Accuracy: {cv_acc.mean():.4f} (+/- {cv_acc.std():.4f})")
    print(f"  CV F1 Macro: {cv_f1.mean():.4f} (+/- {cv_f1.std():.4f})")
    print(f"  Test Accuracy: {results[name]['test_accuracy']:.4f}")
    print(f"  Test F1 Macro: {results[name]['test_f1']:.4f}")

What does this code do?

The loop iterates through all six models and performs three things for each one:

  • Cross-validation: cross_val_score splits the training set into 5 folds, trains on 4 and tests on 1, and repeats for all fold combinations. It collects accuracy and macro F1 scores.
  • Full fit: model.fit(X_tr, y_train) trains the model on the entire training set so it has seen all available training data before making final predictions.
  • Test prediction: model.predict(X_te) generates predictions on the held-out test set that the model has never seen.

The results dictionary stores CV means, standard deviations, test accuracy, test F1, and raw predictions for every model. This structured storage makes comparison easy in the next step.

Why macro F1 instead of just accuracy?

Accuracy alone can be misleading with imbalanced classes. If 45% of matches are home wins, a model that always predicts "home win" gets 45% accuracy without learning anything useful.

Macro F1 computes the F1 score for each class independently, then averages them. This penalizes models that ignore the minority class (draws). A model needs to predict all three outcomes well to score high.

  • Save match_predictor.py.
  • Run the script in your terminal by running:
python match_predictor.py

You should see each model print its CV Accuracy, CV F1 Macro, Test Accuracy, and Test F1 Macro. The full run may take 1-3 minutes depending on your machine (SVM is the slowest due to the large dataset).

Script running very slowly?

SVM has O(n squared) to O(n cubed) time complexity, so it takes significantly longer than the other models on a dataset of this size. Let it run. If it takes more than 5 minutes, your machine may be resource-constrained.

If you see a ConvergenceWarning from Logistic Regression, the model still produces predictions. The max_iter=1000 setting should prevent this in most cases.

Help me debug why my model training is failing or running too slowly.

✔️ Awesome, I've got everything!

Great. Double check you have saved your file before moving on.

ⓧ I'd like to double check the full code

Here is the complete match_predictor.py up to this point:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from huggingface_hub import hf_hub_download
from sklearn.model_selection import train_test_split, cross_val_score, StratifiedKFold
from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.metrics import (
    classification_report,
    confusion_matrix,
    ConfusionMatrixDisplay,
    accuracy_score,
    f1_score,
)
from xgboost import XGBClassifier


# ============================================================
# STEP 1: Download and Load Dataset
# ============================================================
print("Downloading dataset from Hugging Face...")
file_path = hf_hub_download(
    repo_id="adibmed/football-predictions-dataset",
    filename="output/01_matches_all.csv",
    repo_type="dataset",
)
df = pd.read_csv(file_path, low_memory=False)
print(f"Dataset shape: {df.shape}")
print(f"Columns: {df.shape[1]}")
print(df.head())


# ============================================================
# STEP 2: EDA and Preprocessing
# ============================================================

if "result" not in df.columns:
    home_score_col = [c for c in df.columns if "home" in c.lower() and "score" in c.lower() or "home" in c.lower() and "goal" in c.lower() and "avg" not in c.lower()]
    away_score_col = [c for c in df.columns if "away" in c.lower() and "score" in c.lower() or "away" in c.lower() and "goal" in c.lower() and "avg" not in c.lower()]

    if home_score_col and away_score_col:
        df["result"] = np.where(
            df[home_score_col[0]] > df[away_score_col[0]], "home_win",
            np.where(df[home_score_col[0]] < df[away_score_col[0]], "away_win", "draw")
        )
    else:
        raise ValueError("Cannot find score columns to derive match result.")

print("\nTarget distribution:")
print(df["result"].value_counts())
print(df["result"].value_counts(normalize=True).round(3))

plt.figure(figsize=(8, 5))
sns.countplot(x="result", data=df, order=["home_win", "draw", "away_win"])
plt.title("Match Outcome Distribution")
plt.xlabel("Result")
plt.ylabel("Count")
plt.tight_layout()
plt.savefig("target_distribution.png", dpi=150)
plt.close()

numeric_df = df.select_dtypes(include=[np.number])
print(f"\nNumeric columns available: {numeric_df.shape[1]}")

threshold = 0.3
null_pct = numeric_df.isnull().mean()
cols_to_keep = null_pct[null_pct <= threshold].index.tolist()
numeric_df = numeric_df[cols_to_keep]
print(f"Columns after dropping high-null: {numeric_df.shape[1]}")

numeric_df = numeric_df.dropna()
df_clean = df.loc[numeric_df.index].copy()
df_clean[cols_to_keep] = numeric_df

X = df_clean[cols_to_keep].values
le = LabelEncoder()
y = le.fit_transform(df_clean["result"])
class_names = le.classes_
print(f"\nClasses: {class_names}")
print(f"Final dataset: {X.shape[0]} samples, {X.shape[1]} features")

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
print(f"Training set: {X_train.shape[0]} samples")
print(f"Test set: {X_test.shape[0]} samples")

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)


# ============================================================
# STEP 3: Train All Six Models
# ============================================================
print("\n" + "=" * 60)
print("TRAINING ALL MODELS")
print("=" * 60)

models = {
    "Logistic Regression": LogisticRegression(max_iter=1000, random_state=42),
    "KNN": KNeighborsClassifier(n_neighbors=5),
    "Decision Tree": DecisionTreeClassifier(random_state=42),
    "Random Forest": RandomForestClassifier(n_estimators=100, random_state=42),
    "SVM": SVC(probability=True, random_state=42),
    "XGBoost": XGBClassifier(
        n_estimators=100, eval_metric="mlogloss", random_state=42
    ),
}

scaled_models = {"Logistic Regression", "KNN", "SVM"}

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
results = {}

for name, model in models.items():
    print(f"\nTraining {name}...")
    X_tr = X_train_scaled if name in scaled_models else X_train

    cv_acc = cross_val_score(model, X_tr, y_train, cv=cv, scoring="accuracy")
    cv_f1 = cross_val_score(model, X_tr, y_train, cv=cv, scoring="f1_macro")

    model.fit(X_tr, y_train)

    X_te = X_test_scaled if name in scaled_models else X_test
    y_pred = model.predict(X_te)

    results[name] = {
        "cv_accuracy_mean": cv_acc.mean(),
        "cv_accuracy_std": cv_acc.std(),
        "cv_f1_mean": cv_f1.mean(),
        "cv_f1_std": cv_f1.std(),
        "test_accuracy": accuracy_score(y_test, y_pred),
        "test_f1": f1_score(y_test, y_pred, average="macro"),
        "y_pred": y_pred,
    }

    print(f"  CV Accuracy: {cv_acc.mean():.4f} (+/- {cv_acc.std():.4f})")
    print(f"  CV F1 Macro: {cv_f1.mean():.4f} (+/- {cv_f1.std():.4f})")
    print(f"  Test Accuracy: {results[name]['test_accuracy']:.4f}")
    print(f"  Test F1 Macro: {results[name]['test_f1']:.4f}")

All six models are trained and their metrics are stored. Next up, you will visualize these results with comparison charts and confusion matrices to see exactly where each algorithm shines and struggles.

Evaluate and Visualize Model Performance

You have six trained models and a dictionary full of scores. Numbers alone make it hard to spot which algorithm genuinely outperforms the others.

In this step you will turn those raw metrics into publication-quality charts and a ranked summary table. Visual comparisons reveal patterns that tables of decimals hide.

In this step, get ready to:
  • Create a side-by-side bar chart comparing cross-validation accuracy and F1 scores.
  • Generate a 2x3 grid of confusion matrices showing each model's mistakes.
  • Print a ranked summary table and classification reports for the top performers.
Create a comparison bar chart

A bar chart with error bars is the quickest way to see which models score highest and which have the most variance across folds. You will plot cross-validation accuracy on the left and macro F1 on the right.

  • In match_predictor.py, scroll to the bottom of your Step 3 code (after the training loop).
  • Add the following code to extract results and build the comparison chart:
# ============================================================
# STEP 4: Evaluate and Visualize
# ============================================================
print("\n" + "=" * 60)
print("EVALUATION AND VISUALIZATION")
print("=" * 60)

# Comparison bar chart
model_names = list(results.keys())
cv_accs = [results[m]["cv_accuracy_mean"] for m in model_names]
cv_accs_std = [results[m]["cv_accuracy_std"] for m in model_names]
cv_f1s = [results[m]["cv_f1_mean"] for m in model_names]
cv_f1s_std = [results[m]["cv_f1_std"] for m in model_names]

fig, axes = plt.subplots(1, 2, figsize=(14, 6))

x = np.arange(len(model_names))
width = 0.6

axes[0].bar(x, cv_accs, width, yerr=cv_accs_std, capsize=5, color="steelblue")
axes[0].set_xlabel("Model")
axes[0].set_ylabel("Accuracy")
axes[0].set_title("Cross-Validation Accuracy")
axes[0].set_xticks(x)
axes[0].set_xticklabels(model_names, rotation=45, ha="right")
axes[0].set_ylim(0, 1)

axes[1].bar(x, cv_f1s, width, yerr=cv_f1s_std, capsize=5, color="coral")
axes[1].set_xlabel("Model")
axes[1].set_ylabel("F1 Score (Macro)")
axes[1].set_title("Cross-Validation F1 Macro")
axes[1].set_xticks(x)
axes[1].set_xticklabels(model_names, rotation=45, ha="right")
axes[1].set_ylim(0, 1)

plt.tight_layout()
plt.savefig("model_comparison.png", dpi=150)
plt.close()
print("Saved: model_comparison.png")

What does this code do?

The code extracts the mean and standard deviation of each metric from the results dictionary into flat lists that matplotlib can plot directly.

The yerr parameter adds error bars showing the standard deviation across the 5 folds. Larger error bars mean a model's performance fluctuates more depending on which data it trains on.

  • Save match_predictor.py.
  • Run the script in your terminal. After training completes, you should see Saved: model_comparison.png printed in the console.
  • Open model_comparison.png from your project folder. You should see two bar charts side by side with all 6 model names on the x-axis.

Chart not saving or looks blank?

  • Make sure plt.close() comes after plt.savefig(). Closing the figure before saving produces an empty file.
  • If the bars look squished or labels are cut off, confirm that plt.tight_layout() is present before the save call.
  • help me debug my model comparison chart
Generate confusion matrices for all models

A confusion matrix shows exactly where a classifier gets confused. Each cell reveals how many times the model predicted one class when the true label was another.

  • Add the following code below the bar chart section to build a 2x3 grid of confusion matrices:
# Confusion matrices grid
fig, axes = plt.subplots(2, 3, figsize=(16, 10))
axes_flat = axes.flatten()

for idx, name in enumerate(model_names):
    cm = confusion_matrix(y_test, results[name]["y_pred"])
    disp = ConfusionMatrixDisplay(cm, display_labels=class_names)
    disp.plot(ax=axes_flat[idx], cmap="Blues", colorbar=False)
    axes_flat[idx].set_title(name, fontsize=11)

plt.suptitle("Confusion Matrices - All Models", fontsize=14, y=1.02)
plt.tight_layout()
plt.savefig("confusion_matrices.png", dpi=150, bbox_inches="tight")
plt.close()
print("Saved: confusion_matrices.png")

What does this code do?

  • plt.subplots(2, 3) creates a grid of 6 subplots (one per model). axes.flatten() converts the 2D array of axes into a flat list so you can loop through them.
  • ConfusionMatrixDisplay from scikit-learn renders the matrix as a heatmap with numeric annotations. The Blues colormap makes higher counts darker, so you can instantly spot where each model concentrates its predictions.
  • bbox_inches="tight" ensures the suptitle at the top is not clipped when saving.
  • Save match_predictor.py and run the script again.
  • Open confusion_matrices.png from your project folder. You should see a 2x3 grid where each cell is a labeled heatmap for one of the 6 models.

How do I read a confusion matrix?

Rows represent the true label. Columns represent what the model predicted. Diagonal cells (top-left to bottom-right) are correct predictions. Off-diagonal cells are mistakes.

For football predictions, look at the "draw" row. Most models struggle to predict draws correctly and tend to classify them as home wins instead. This is a known difficulty in sports prediction.

Confusion matrix labels overlapping?

  • If the class labels overlap on the axes, confirm the figsize is set to (16, 10). A smaller figure squeezes the labels together.
  • If you see an error about class_names, make sure your Step 2 code defines it with class_names = le.classes_.
  • help me fix my confusion matrix grid
Print a ranked summary table

Charts are great for quick visual comparison. A ranked table gives you the precise numbers you need to declare a winner and understand how close the competition is.

  • Add the following code below the confusion matrices section to print a summary table and detailed classification reports:
# Summary table
print("\n" + "-" * 80)
print(f"{'Model':<22} {'Test Acc':>10} {'Test F1':>10} {'CV Acc':>12} {'CV F1':>12}")
print("-" * 80)

sorted_models = sorted(model_names, key=lambda m: results[m]["test_f1"], reverse=True)
for name in sorted_models:
    r = results[name]
    print(
        f"{name:<22} {r['test_accuracy']:>10.4f} {r['test_f1']:>10.4f} "
        f"{r['cv_accuracy_mean']:>8.4f}+/-{r['cv_accuracy_std']:.3f} "
        f"{r['cv_f1_mean']:>5.4f}+/-{r['cv_f1_std']:.3f}"
    )
print("-" * 80)

# Per-model classification reports
for name in sorted_models[:3]:
    print(f"\n--- {name} Classification Report ---")
    print(classification_report(y_test, results[name]["y_pred"], target_names=class_names))

What does this code do?

  • sorted_models ranks all models by their test F1 score from highest to lowest. This puts the best performer at the top of the table.
  • The table prints test accuracy, test F1, and the cross-validation scores with their standard deviations so you can compare generalisation performance at a glance.
  • classification_report gives per-class precision, recall, and F1 for the top 3 models. This reveals whether a model is strong across all three outcomes or just inflating its score on the majority class.
  • Save match_predictor.py and run the script one more time.
  • In your terminal output, you should see a formatted table with all 6 models ranked by F1 score, followed by detailed classification reports for the top 3.

Table columns misaligned?

  • Column alignment depends on the f-string width specifiers like <22 and >10. Make sure you copied these format codes exactly.
  • If you see a KeyError, confirm that your results dictionary keys match the exact model names from Step 3.
  • help me debug my summary table output

✔️ Awesome, I've got everything!

Great. Double check you have saved your file and all three outputs are present: model_comparison.png, confusion_matrices.png, and the summary table in your terminal.

ⓧ I'd like to double check the full code

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from huggingface_hub import hf_hub_download
from sklearn.model_selection import train_test_split, cross_val_score, StratifiedKFold
from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.metrics import (
    classification_report,
    confusion_matrix,
    ConfusionMatrixDisplay,
    accuracy_score,
    f1_score,
)
from xgboost import XGBClassifier


# ============================================================
# STEP 1: Download and Load Dataset
# ============================================================
print("Downloading dataset from Hugging Face...")
file_path = hf_hub_download(
    repo_id="adibmed/football-predictions-dataset",
    filename="output/01_matches_all.csv",
    repo_type="dataset",
)
df = pd.read_csv(file_path, low_memory=False)
print(f"Dataset shape: {df.shape}")
print(f"Columns: {df.shape[1]}")
print(df.head())


# ============================================================
# STEP 2: EDA and Preprocessing
# ============================================================

if "result" not in df.columns:
    home_score_col = [c for c in df.columns if "home" in c.lower() and "score" in c.lower() or "home" in c.lower() and "goal" in c.lower() and "avg" not in c.lower()]
    away_score_col = [c for c in df.columns if "away" in c.lower() and "score" in c.lower() or "away" in c.lower() and "goal" in c.lower() and "avg" not in c.lower()]

    if home_score_col and away_score_col:
        df["result"] = np.where(
            df[home_score_col[0]] > df[away_score_col[0]], "home_win",
            np.where(df[home_score_col[0]] < df[away_score_col[0]], "away_win", "draw")
        )
    else:
        raise ValueError("Cannot find score columns to derive match result.")

print("\nTarget distribution:")
print(df["result"].value_counts())
print(df["result"].value_counts(normalize=True).round(3))

plt.figure(figsize=(8, 5))
sns.countplot(x="result", data=df, order=["home_win", "draw", "away_win"])
plt.title("Match Outcome Distribution")
plt.xlabel("Result")
plt.ylabel("Count")
plt.tight_layout()
plt.savefig("target_distribution.png", dpi=150)
plt.close()

numeric_df = df.select_dtypes(include=[np.number])
print(f"\nNumeric columns available: {numeric_df.shape[1]}")

threshold = 0.3
null_pct = numeric_df.isnull().mean()
cols_to_keep = null_pct[null_pct <= threshold].index.tolist()
numeric_df = numeric_df[cols_to_keep]
print(f"Columns after dropping high-null: {numeric_df.shape[1]}")

numeric_df = numeric_df.dropna()
df_clean = df.loc[numeric_df.index].copy()
df_clean[cols_to_keep] = numeric_df

X = df_clean[cols_to_keep].values
le = LabelEncoder()
y = le.fit_transform(df_clean["result"])
class_names = le.classes_
print(f"\nClasses: {class_names}")
print(f"Final dataset: {X.shape[0]} samples, {X.shape[1]} features")

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
print(f"Training set: {X_train.shape[0]} samples")
print(f"Test set: {X_test.shape[0]} samples")

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)


# ============================================================
# STEP 3: Train All Six Models
# ============================================================
print("\n" + "=" * 60)
print("TRAINING ALL MODELS")
print("=" * 60)

models = {
    "Logistic Regression": LogisticRegression(max_iter=1000, random_state=42),
    "KNN": KNeighborsClassifier(n_neighbors=5),
    "Decision Tree": DecisionTreeClassifier(random_state=42),
    "Random Forest": RandomForestClassifier(n_estimators=100, random_state=42),
    "SVM": SVC(probability=True, random_state=42),
    "XGBoost": XGBClassifier(
        n_estimators=100, eval_metric="mlogloss", random_state=42
    ),
}

scaled_models = {"Logistic Regression", "KNN", "SVM"}

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
results = {}

for name, model in models.items():
    print(f"\nTraining {name}...")
    X_tr = X_train_scaled if name in scaled_models else X_train

    cv_acc = cross_val_score(model, X_tr, y_train, cv=cv, scoring="accuracy")
    cv_f1 = cross_val_score(model, X_tr, y_train, cv=cv, scoring="f1_macro")

    model.fit(X_tr, y_train)

    X_te = X_test_scaled if name in scaled_models else X_test
    y_pred = model.predict(X_te)

    results[name] = {
        "cv_accuracy_mean": cv_acc.mean(),
        "cv_accuracy_std": cv_acc.std(),
        "cv_f1_mean": cv_f1.mean(),
        "cv_f1_std": cv_f1.std(),
        "test_accuracy": accuracy_score(y_test, y_pred),
        "test_f1": f1_score(y_test, y_pred, average="macro"),
        "y_pred": y_pred,
    }

    print(f"  CV Accuracy: {cv_acc.mean():.4f} (+/- {cv_acc.std():.4f})")
    print(f"  CV F1 Macro: {cv_f1.mean():.4f} (+/- {cv_f1.std():.4f})")
    print(f"  Test Accuracy: {results[name]['test_accuracy']:.4f}")
    print(f"  Test F1 Macro: {results[name]['test_f1']:.4f}")


# ============================================================
# STEP 4: Evaluate and Visualize
# ============================================================
print("\n" + "=" * 60)
print("EVALUATION AND VISUALIZATION")
print("=" * 60)

model_names = list(results.keys())
cv_accs = [results[m]["cv_accuracy_mean"] for m in model_names]
cv_accs_std = [results[m]["cv_accuracy_std"] for m in model_names]
cv_f1s = [results[m]["cv_f1_mean"] for m in model_names]
cv_f1s_std = [results[m]["cv_f1_std"] for m in model_names]

fig, axes = plt.subplots(1, 2, figsize=(14, 6))

x = np.arange(len(model_names))
width = 0.6

axes[0].bar(x, cv_accs, width, yerr=cv_accs_std, capsize=5, color="steelblue")
axes[0].set_xlabel("Model")
axes[0].set_ylabel("Accuracy")
axes[0].set_title("Cross-Validation Accuracy")
axes[0].set_xticks(x)
axes[0].set_xticklabels(model_names, rotation=45, ha="right")
axes[0].set_ylim(0, 1)

axes[1].bar(x, cv_f1s, width, yerr=cv_f1s_std, capsize=5, color="coral")
axes[1].set_xlabel("Model")
axes[1].set_ylabel("F1 Score (Macro)")
axes[1].set_title("Cross-Validation F1 Macro")
axes[1].set_xticks(x)
axes[1].set_xticklabels(model_names, rotation=45, ha="right")
axes[1].set_ylim(0, 1)

plt.tight_layout()
plt.savefig("model_comparison.png", dpi=150)
plt.close()
print("Saved: model_comparison.png")

fig, axes = plt.subplots(2, 3, figsize=(16, 10))
axes_flat = axes.flatten()

for idx, name in enumerate(model_names):
    cm = confusion_matrix(y_test, results[name]["y_pred"])
    disp = ConfusionMatrixDisplay(cm, display_labels=class_names)
    disp.plot(ax=axes_flat[idx], cmap="Blues", colorbar=False)
    axes_flat[idx].set_title(name, fontsize=11)

plt.suptitle("Confusion Matrices - All Models", fontsize=14, y=1.02)
plt.tight_layout()
plt.savefig("confusion_matrices.png", dpi=150, bbox_inches="tight")
plt.close()
print("Saved: confusion_matrices.png")

print("\n" + "-" * 80)
print(f"{'Model':<22} {'Test Acc':>10} {'Test F1':>10} {'CV Acc':>12} {'CV F1':>12}")
print("-" * 80)

sorted_models = sorted(model_names, key=lambda m: results[m]["test_f1"], reverse=True)
for name in sorted_models:
    r = results[name]
    print(
        f"{name:<22} {r['test_accuracy']:>10.4f} {r['test_f1']:>10.4f} "
        f"{r['cv_accuracy_mean']:>8.4f}+/-{r['cv_accuracy_std']:.3f} "
        f"{r['cv_f1_mean']:>5.4f}+/-{r['cv_f1_std']:.3f}"
    )
print("-" * 80)

for name in sorted_models[:3]:
    print(f"\n--- {name} Classification Report ---")
    print(classification_report(y_test, results[name]["y_pred"], target_names=class_names))

You now have a complete visual evaluation of all 6 algorithms. Next up, you will dig into feature importance to understand which input signals drive these predictions.

Analyze Feature Importance

You now have a full performance comparison showing which models score highest. But accuracy alone doesn't explain why a model makes the predictions it does.

Tree-based models expose a property called feature importance that tells you how much each input feature contributes to the model's decisions. In this step, you'll extract and visualize those importances to understand which data points actually drive match outcome predictions.

In this step, get ready to:
  • Plot the top 15 most important features for each tree-based model as horizontal bar charts.
  • Compare how Decision Tree, Random Forest, and XGBoost rank features differently.
Extract and plot feature importances from tree-based models

Only tree-based algorithms (Decision Tree, Random Forest, XGBoost) provide a built-in feature_importances_ attribute. Linear models and KNN don't split data along feature thresholds, so they don't produce this metric natively.

  • In match_predictor.py, scroll to the bottom of your file (below the classification reports loop).
  • Add the Step 5 header and feature name setup by pasting this code:


# ============================================================
# STEP 5: Feature Importance Analysis
# ============================================================
print("\n" + "=" * 60)
print("FEATURE IMPORTANCE ANALYSIS")
print("=" * 60)

# Map column indices back to readable names
feature_names = [f"Feature_{i}" for i in range(X.shape[1])]
if len(cols_to_keep) == X.shape[1]:
    feature_names = cols_to_keep

# Select only the tree-based models that expose feature importances
tree_models = {
    "Decision Tree": models["Decision Tree"],
    "Random Forest": models["Random Forest"],
    "XGBoost": models["XGBoost"],
}

What does this code do?

This block prepares the data you need for feature importance analysis.

  • feature_names maps column indices back to their original names from cols_to_keep. Without this, your chart labels would just show numbers.
  • tree_models pulls the three already-fitted tree-based classifiers from the models dict. These are the only models that have a feature_importances_ attribute.
  • Now add the visualization code directly below to create the 1x3 horizontal bar chart grid:
# Create a 1x3 grid of horizontal bar charts
fig, axes = plt.subplots(1, 3, figsize=(18, 8))

for idx, (name, model) in enumerate(tree_models.items()):
    importances = model.feature_importances_
    # Get indices of top 15 features sorted by importance
    top_indices = np.argsort(importances)[-15:]
    top_features = [feature_names[i] for i in top_indices]
    top_importances = importances[top_indices]

    axes[idx].barh(range(15), top_importances, color="teal")
    axes[idx].set_yticks(range(15))
    axes[idx].set_yticklabels(top_features, fontsize=8)
    axes[idx].set_xlabel("Importance")
    axes[idx].set_title(f"{name}\nTop 15 Features")

plt.suptitle("Feature Importance Comparison", fontsize=14)
plt.tight_layout()
plt.savefig("feature_importance.png", dpi=150, bbox_inches="tight")
plt.close()
print("Saved: feature_importance.png")

What does this code do?

  • feature_importances_ returns an array where each value represents how much a feature contributed to the model's splits.
  • np.argsort(importances)[-15:] grabs the indices of the 15 highest-importance features, sorted from lowest to highest (so the most important feature appears at the top of each horizontal bar).
  • Each subplot gets its own chart so you can visually compare which features matter most to each algorithm.
  • Save match_predictor.py.

Why horizontal bar charts?

Feature names can be long (like "home_elo_rating_delta_5"). Horizontal bars let the full name display as a y-axis label without overlapping. Vertical bars would require rotating labels 90 degrees, making them unreadable.

Compare feature rankings and print a summary

A chart is great for spotting patterns at a glance, but a printed summary gives you exact feature names and importance scores to reference.

  • Add this code below the plotting section to print the top 5 features for each model and a final completion summary:
# Print top 5 features per model for quick comparison
for name, model in tree_models.items():
    importances = model.feature_importances_
    top_indices = np.argsort(importances)[-5:][::-1]
    print(f"\n{name} - Top 5 Features:")
    for rank, i in enumerate(top_indices, 1):
        print(f"  {rank}. {feature_names[i]} ({importances[i]:.4f})")

print("\n" + "=" * 60)
print("PROJECT COMPLETE")
print("=" * 60)
print("Generated outputs:")
print("  - target_distribution.png")
print("  - model_comparison.png")
print("  - confusion_matrices.png")
print("  - feature_importance.png")

What does this code do?

  • np.argsort(importances)[-5:][::-1] gets the top 5 feature indices in descending order (most important first).
  • The loop prints a ranked list for each tree model so you can compare which feature groups (ELO ratings, form metrics, rankings) dominate each algorithm's decisions.
  • The completion summary lists all four PNG files the script generated across every step.
  • Save match_predictor.py.
  • Run the full script in your terminal by running:
python match_predictor.py
  • Near the end of the terminal output, you should see a "FEATURE IMPORTANCE ANALYSIS" header followed by the top 5 features for each of the three tree-based models.
  • The final lines should read "PROJECT COMPLETE" and list all four output PNG files.
  • Check your project folder. You should see feature_importance.png alongside the three PNGs from earlier steps.

Script failing or no feature_importance.png?

  • If you see an AttributeError mentioning feature_importances_, make sure tree_models references the models from the models dict (which are fitted). Unfitted models don't have this attribute.
  • If the chart labels show "Feature_0", "Feature_1", etc. instead of real column names, check that cols_to_keep hasn't been overwritten between Step 2 and Step 5.
  • If the script runs for a long time, that's expected. SVM training on ~49,000 samples is slow. Give it a few minutes to finish.

Help me debug my feature importance analysis

Why do models rank features differently?

A Decision Tree picks the single best split at each node, so it may over-rely on one dominant feature. Random Forest averages importance across 100 trees with random feature subsets, spreading importance more evenly. XGBoost uses gradient boosting to iteratively correct errors, often surfacing subtle features that other models miss.

Comparing all three gives you a more complete picture of which features genuinely matter versus which ones happen to look good in a single tree.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file and that all four PNG files are in your project folder.

ⓧ I'd like to double check the full code

Here's the complete match_predictor.py file with all five steps:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from huggingface_hub import hf_hub_download
from sklearn.model_selection import train_test_split, cross_val_score, StratifiedKFold
from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.metrics import (
    classification_report,
    confusion_matrix,
    ConfusionMatrixDisplay,
    accuracy_score,
    f1_score,
)
from xgboost import XGBClassifier


# ============================================================
# STEP 1: Download and Load Dataset
# ============================================================
print("Downloading dataset from Hugging Face...")
file_path = hf_hub_download(
    repo_id="adibmed/football-predictions-dataset",
    filename="output/01_matches_all.csv",
    repo_type="dataset",
)
df = pd.read_csv(file_path, low_memory=False)
print(f"Dataset shape: {df.shape}")
print(f"Columns: {df.shape[1]}")
print(df.head())


# ============================================================
# STEP 2: EDA and Preprocessing
# ============================================================

if "result" not in df.columns:
    home_score_col = [c for c in df.columns if "home" in c.lower() and "score" in c.lower() or "home" in c.lower() and "goal" in c.lower() and "avg" not in c.lower()]
    away_score_col = [c for c in df.columns if "away" in c.lower() and "score" in c.lower() or "away" in c.lower() and "goal" in c.lower() and "avg" not in c.lower()]

    if home_score_col and away_score_col:
        df["result"] = np.where(
            df[home_score_col[0]] > df[away_score_col[0]], "home_win",
            np.where(df[home_score_col[0]] < df[away_score_col[0]], "away_win", "draw")
        )
    else:
        raise ValueError("Cannot find score columns to derive match result.")

print("\nTarget distribution:")
print(df["result"].value_counts())
print(df["result"].value_counts(normalize=True).round(3))

plt.figure(figsize=(8, 5))
sns.countplot(x="result", data=df, order=["home_win", "draw", "away_win"])
plt.title("Match Outcome Distribution")
plt.xlabel("Result")
plt.ylabel("Count")
plt.tight_layout()
plt.savefig("target_distribution.png", dpi=150)
plt.close()

numeric_df = df.select_dtypes(include=[np.number])
print(f"\nNumeric columns available: {numeric_df.shape[1]}")

threshold = 0.3
null_pct = numeric_df.isnull().mean()
cols_to_keep = null_pct[null_pct <= threshold].index.tolist()
numeric_df = numeric_df[cols_to_keep]
print(f"Columns after dropping high-null: {numeric_df.shape[1]}")

numeric_df = numeric_df.dropna()
df_clean = df.loc[numeric_df.index].copy()
df_clean[cols_to_keep] = numeric_df

X = df_clean[cols_to_keep].values
le = LabelEncoder()
y = le.fit_transform(df_clean["result"])
class_names = le.classes_
print(f"\nClasses: {class_names}")
print(f"Final dataset: {X.shape[0]} samples, {X.shape[1]} features")

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
print(f"Training set: {X_train.shape[0]} samples")
print(f"Test set: {X_test.shape[0]} samples")

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)


# ============================================================
# STEP 3: Train All Six Models
# ============================================================
print("\n" + "=" * 60)
print("TRAINING ALL MODELS")
print("=" * 60)

models = {
    "Logistic Regression": LogisticRegression(max_iter=1000, random_state=42),
    "KNN": KNeighborsClassifier(n_neighbors=5),
    "Decision Tree": DecisionTreeClassifier(random_state=42),
    "Random Forest": RandomForestClassifier(n_estimators=100, random_state=42),
    "SVM": SVC(probability=True, random_state=42),
    "XGBoost": XGBClassifier(
        n_estimators=100, eval_metric="mlogloss", random_state=42
    ),
}

scaled_models = {"Logistic Regression", "KNN", "SVM"}

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
results = {}

for name, model in models.items():
    print(f"\nTraining {name}...")
    X_tr = X_train_scaled if name in scaled_models else X_train

    cv_acc = cross_val_score(model, X_tr, y_train, cv=cv, scoring="accuracy")
    cv_f1 = cross_val_score(model, X_tr, y_train, cv=cv, scoring="f1_macro")

    model.fit(X_tr, y_train)

    X_te = X_test_scaled if name in scaled_models else X_test
    y_pred = model.predict(X_te)

    results[name] = {
        "cv_accuracy_mean": cv_acc.mean(),
        "cv_accuracy_std": cv_acc.std(),
        "cv_f1_mean": cv_f1.mean(),
        "cv_f1_std": cv_f1.std(),
        "test_accuracy": accuracy_score(y_test, y_pred),
        "test_f1": f1_score(y_test, y_pred, average="macro"),
        "y_pred": y_pred,
    }

    print(f"  CV Accuracy: {cv_acc.mean():.4f} (+/- {cv_acc.std():.4f})")
    print(f"  CV F1 Macro: {cv_f1.mean():.4f} (+/- {cv_f1.std():.4f})")
    print(f"  Test Accuracy: {results[name]['test_accuracy']:.4f}")
    print(f"  Test F1 Macro: {results[name]['test_f1']:.4f}")


# ============================================================
# STEP 4: Evaluate and Visualize
# ============================================================
print("\n" + "=" * 60)
print("EVALUATION AND VISUALIZATION")
print("=" * 60)

model_names = list(results.keys())
cv_accs = [results[m]["cv_accuracy_mean"] for m in model_names]
cv_accs_std = [results[m]["cv_accuracy_std"] for m in model_names]
cv_f1s = [results[m]["cv_f1_mean"] for m in model_names]
cv_f1s_std = [results[m]["cv_f1_std"] for m in model_names]

fig, axes = plt.subplots(1, 2, figsize=(14, 6))

x = np.arange(len(model_names))
width = 0.6

axes[0].bar(x, cv_accs, width, yerr=cv_accs_std, capsize=5, color="steelblue")
axes[0].set_xlabel("Model")
axes[0].set_ylabel("Accuracy")
axes[0].set_title("Cross-Validation Accuracy")
axes[0].set_xticks(x)
axes[0].set_xticklabels(model_names, rotation=45, ha="right")
axes[0].set_ylim(0, 1)

axes[1].bar(x, cv_f1s, width, yerr=cv_f1s_std, capsize=5, color="coral")
axes[1].set_xlabel("Model")
axes[1].set_ylabel("F1 Score (Macro)")
axes[1].set_title("Cross-Validation F1 Macro")
axes[1].set_xticks(x)
axes[1].set_xticklabels(model_names, rotation=45, ha="right")
axes[1].set_ylim(0, 1)

plt.tight_layout()
plt.savefig("model_comparison.png", dpi=150)
plt.close()
print("Saved: model_comparison.png")

fig, axes = plt.subplots(2, 3, figsize=(16, 10))
axes_flat = axes.flatten()

for idx, name in enumerate(model_names):
    cm = confusion_matrix(y_test, results[name]["y_pred"])
    disp = ConfusionMatrixDisplay(cm, display_labels=class_names)
    disp.plot(ax=axes_flat[idx], cmap="Blues", colorbar=False)
    axes_flat[idx].set_title(name, fontsize=11)

plt.suptitle("Confusion Matrices - All Models", fontsize=14, y=1.02)
plt.tight_layout()
plt.savefig("confusion_matrices.png", dpi=150, bbox_inches="tight")
plt.close()
print("Saved: confusion_matrices.png")

print("\n" + "-" * 80)
print(f"{'Model':<22} {'Test Acc':>10} {'Test F1':>10} {'CV Acc':>12} {'CV F1':>12}")
print("-" * 80)

sorted_models = sorted(model_names, key=lambda m: results[m]["test_f1"], reverse=True)
for name in sorted_models:
    r = results[name]
    print(
        f"{name:<22} {r['test_accuracy']:>10.4f} {r['test_f1']:>10.4f} "
        f"{r['cv_accuracy_mean']:>8.4f}+/-{r['cv_accuracy_std']:.3f} "
        f"{r['cv_f1_mean']:>5.4f}+/-{r['cv_f1_std']:.3f}"
    )
print("-" * 80)

for name in sorted_models[:3]:
    print(f"\n--- {name} Classification Report ---")
    print(classification_report(y_test, results[name]["y_pred"], target_names=class_names))


# ============================================================
# STEP 5: Feature Importance Analysis
# ============================================================
print("\n" + "=" * 60)
print("FEATURE IMPORTANCE ANALYSIS")
print("=" * 60)

feature_names = [f"Feature_{i}" for i in range(X.shape[1])]
if len(cols_to_keep) == X.shape[1]:
    feature_names = cols_to_keep

tree_models = {
    "Decision Tree": models["Decision Tree"],
    "Random Forest": models["Random Forest"],
    "XGBoost": models["XGBoost"],
}

fig, axes = plt.subplots(1, 3, figsize=(18, 8))

for idx, (name, model) in enumerate(tree_models.items()):
    importances = model.feature_importances_
    top_indices = np.argsort(importances)[-15:]
    top_features = [feature_names[i] for i in top_indices]
    top_importances = importances[top_indices]

    axes[idx].barh(range(15), top_importances, color="teal")
    axes[idx].set_yticks(range(15))
    axes[idx].set_yticklabels(top_features, fontsize=8)
    axes[idx].set_xlabel("Importance")
    axes[idx].set_title(f"{name}\nTop 15 Features")

plt.suptitle("Feature Importance Comparison", fontsize=14)
plt.tight_layout()
plt.savefig("feature_importance.png", dpi=150, bbox_inches="tight")
plt.close()
print("Saved: feature_importance.png")

for name, model in tree_models.items():
    importances = model.feature_importances_
    top_indices = np.argsort(importances)[-5:][::-1]
    print(f"\n{name} - Top 5 Features:")
    for rank, i in enumerate(top_indices, 1):
        print(f"  {rank}. {feature_names[i]} ({importances[i]:.4f})")

print("\n" + "=" * 60)
print("PROJECT COMPLETE")
print("=" * 60)
print("Generated outputs:")
print("  - target_distribution.png")
print("  - model_comparison.png")
print("  - confusion_matrices.png")
print("  - feature_importance.png")

Your supervised learning showdown is complete. You've trained 6 algorithms, benchmarked them head-to-head, and now understand not just which models perform best, but which features drive their predictions. Next up is the Secret Mission, where you can squeeze even more performance out of your top models with hyperparameter tuning.

Secret mission

Squeeze More Performance with Hyperparameter Tuning

Your models performed well with default settings, but what if you could push them further? In this secret mission, you will use GridSearchCV to systematically search for the optimal hyperparameters for your top 2 models and quantify the improvement.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs or cloud resources to worry about. Decide whether to keep your project files for reference, or delete them to free up disk space.

Resources you used:

  • Python virtual environment with installed packages (scikit-learn, xgboost, seaborn, huggingface_hub, pandas, matplotlib, numpy).
  • Project files: match_predictor.py, match_predictor_tuned.py, and generated PNG visualization files.
  • Hugging Face cached dataset stored in ~/.cache/huggingface/hub (approximately 113 MB).

Keep everything running

No action needed. Choose this if you want to re-run the script, tweak model parameters, or extend the project further.

  • Your virtual environment stays on disk and can be reactivated any time.
  • The Hugging Face cached dataset means future runs skip the download step entirely.
  • Your PNG visualizations remain in the project folder for reference or sharing.

Pause - I'll come back to this later

Deactivate the virtual environment to free up your terminal session. Your files stay on disk for when you return.

  • Deactivate the virtual environment by running:
deactivate
  • Close VS Code if you no longer need it open.
  • To resume later, reactivate the environment with source venv/bin/activate (macOS/Linux) or venv\Scripts\activate (Windows).

Delete - I don't want to use this again

Remove all project files, the virtual environment, and the cached dataset to reclaim disk space.

  • Deactivate the virtual environment first by running:
deactivate
  • Navigate to your Desktop and delete the project folder by running:
cd ~/Desktop
rm -rf [[PROJECT_FOLDER="your project folder name (visible in VS Code title bar)"]]
  • On Windows, use this instead:
cd ~\Desktop
rmdir /s /q [[PROJECT_FOLDER="your project folder name (visible in VS Code title bar)"]]
  • Delete the Hugging Face cached dataset by running:
rm -rf ~/.cache/huggingface/hub/datasets--adibmed--football-predictions-dataset
  • On Windows, use this instead:
rmdir /s /q %USERPROFILE%\.cache\huggingface\hub\datasets--adibmed--football-predictions-dataset
  • Confirm deletion by checking your Desktop. The project folder should no longer appear.

That's a Wrap!

That's a Wrap!

Nice work. You just built a complete supervised learning benchmarking system that pits 6 classification algorithms against each other on real-world international football data.

You've learned how to:

  • Train and evaluate 6 supervised learning classifiers (Logistic Regression, KNN, Decision Tree, Random Forest, SVM, XGBoost) on a real-world 3-class classification problem using cross-validation and stratified splits.
  • Produce publication-quality visualizations including comparison bar charts, confusion matrix grids, and feature importance plots that explain model behaviour.
  • Analyze how different algorithm families (linear, distance-based, tree-based, ensemble) prioritize features differently and perform on multi-class sports prediction.
  • Secret Mission: Tuned hyperparameters with GridSearchCV to push your best models further and quantified the improvement.

Ready to quiz yourself?