Explainable Football Predictions

Add SHAP explainability and calibration to an XGBoost football predictor.

Introduction

30 Second Summary

When a weather app says "70% chance of rain," you trust that number enough to grab an umbrella. But what if that 70% was actually wrong half the time? The same problem exists in machine learning. A model can be accurate overall while giving you confidence scores you cannot trust.

In this project, you will take a trained football match prediction model and make it genuinely trustworthy by adding explainability (why did it predict this?) and calibration (can I trust the confidence scores?).

What You'll Build

An interactive Streamlit web app that displays calibrated match predictions with SHAP-powered explanations showing exactly which features drove each prediction.

By the end of this project, you'll have:

  • A SHAP explainability layer that reveals why XGBoost predicts each match outcome, with both global beeswarm and per-prediction waterfall explanations.
  • A calibration analysis comparing XGBoost vs Random Forest trustworthiness, with reliability diagrams and Brier scores proving which model's confidence scores you can actually trust.
  • A deployed Streamlit app on Streamlit Community Cloud where users pick a match, see calibrated predictions with confidence, and drill into SHAP explanations for that specific prediction.
  • Secret Mission: Add SHAP dependence plots revealing how ELO rating differences interact with other features to influence match predictions.

Are there any prerequisites?

This project builds directly on the "Supervised Learning: Football Predictions" project. You should have completed that project (or have a trained XGBoost classifier on the same dataset) before starting here. Basic Python and pandas familiarity is also expected.

Before We Start

In the previous project, you trained an XGBoost model that predicts international football match outcomes with solid accuracy. But accuracy alone doesn't tell you whether you can trust the model's confidence scores.

Before diving into explainability and calibration, you need a few extra tools in your environment. In this step, you'll install SHAP, Streamlit, and joblib into your existing virtual environment, verify they work, and create the project files.

In this step, get ready to:
  • Install SHAP, Streamlit, and joblib into your existing virtual environment.
  • Verify the installations work correctly.
  • Create the two project files you'll work in.
Activate your virtual environment and install packages

Your football-ml virtual environment from the previous project already has scikit-learn, xgboost, and pandas. You need to activate it and add three new packages.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type VS Code and press Enter to open it.
  • Open a terminal inside VS Code by pressing Ctrl+` (backtick key, same shortcut on macOS and Windows).

First, navigate to your football predictions project folder and activate the virtual environment.

macOS

  • Activate your virtual environment by running this command:
source football-ml/bin/activate

You should see (football-ml) appear at the start of your terminal prompt.

Windows

  • Activate your virtual environment by running this command:
football-ml\Scripts\activate

You should see (football-ml) appear at the start of your terminal prompt.

  • Install the three new packages with pinned versions by running this command:
pip install shap==0.52.0 streamlit==1.58.0 joblib==1.4.2

What do these packages do?

  • shap computes SHAP values that explain why a model made each prediction.
  • streamlit lets you build interactive web apps from Python scripts with minimal code.
  • joblib saves and loads Python objects (like trained models) to disk efficiently.

The install may take a minute as shap compiles some C extensions. Wait until you see the Successfully installed message before continuing.

Install failing or taking too long?

  • Make sure (football-ml) is showing in your terminal prompt. If not, re-run the activate command above.
  • If you see a permissions error, do not use sudo. Deactivate and re-activate the virtual environment instead.
  • help me troubleshoot my pip install
Verify your installations

A quick smoke test confirms that both SHAP and Streamlit installed correctly before you start writing code.

  • Verify Streamlit works by running this command:
streamlit hello

Streamlit's Hello app should appear in a new tab in your web browser. This is a built-in demo showcasing Streamlit widgets and layout features.

  • Once you see the demo page in your browser, return to your terminal and press Ctrl+C to stop the Streamlit server.
  • Verify SHAP works by running this command:
python -c "import shap; print(shap.__version__)"

You should see 0.52.0 printed in your terminal.

Streamlit not opening or import failing?

  • If streamlit hello gives a "command not found" error, try python -m streamlit hello instead.
  • If the browser tab doesn't open automatically, look in the terminal for a URL starting with http://localhost:8501 and open it manually.
  • If import shap fails with a compile error, make sure you are running Python 3.12 or higher inside the virtual environment.
  • help me fix my shap or streamlit installation
Create the project files

You'll work in two Python files throughout this project. One handles the analysis pipeline and the other holds the Streamlit app.

  • Create both project files by running this command:
touch trustworthy_predictor.py app.py
  • Confirm the files exist by running:
ls

You should see trustworthy_predictor.py and app.py in the output alongside your existing project files.

What are these files for?

  • trustworthy_predictor.py is where you'll build the SHAP explainability analysis and calibration pipeline.
  • app.py is the Streamlit web app that users interact with to see predictions and explanations.

Using Windows and touch doesn't work?

Windows PowerShell doesn't have touch. Use this command instead:

New-Item trustworthy_predictor.py, app.py

help me create empty Python files on Windows

Your environment is ready and your files are in place. Next up, you'll retrain the models and use SHAP to reveal exactly why each prediction was made.

Set Up Your Environment

This project extends your football predictions model with explainability and calibration. You already have the football-ml virtual environment with the core ML packages from the previous project.

Before writing any code, you need to confirm that the three new packages (shap, streamlit, and joblib) are correctly installed and working. A quick verification now prevents confusing import errors later.

In this step, get ready to:
  • Verify that Streamlit runs correctly in your environment.
  • Confirm that SHAP imports without errors.
  • Confirm your project files are ready for code.
Verify your Streamlit installation

Streamlit is the framework you will use to build and deploy your interactive prediction app. Running its built-in demo confirms the package is installed correctly and that Streamlit can serve a web app from your machine.

  • In your terminal with the football-ml virtual environment active, launch the Streamlit demo by running this command:
streamlit hello

A demo app opens in a new browser tab, typically at http://localhost:8501. You should see an interactive page showcasing Streamlit widgets and charts.

  • Once you see the demo app in your browser, return to your terminal and press Ctrl+C to stop the server.

Don't see the demo app?

If the browser tab does not open automatically, copy the URL shown in your terminal (usually http://localhost:8501) and paste it into your browser manually.

If you see a "command not found" error, try the long-form command: python -m streamlit hello. If that also fails, make sure your football-ml virtual environment is active (you should see (football-ml) in your terminal prompt).

Help me troubleshoot why streamlit hello is not working in my football-ml virtual environment.

Verify SHAP imports correctly

SHAP is the explainability library that will reveal why your model makes each prediction. It has compiled C extensions that can sometimes fail silently during installation, so a quick import test confirms everything built correctly.

  • Open a Python shell by running this command:
python3
  • In the Python shell, test the SHAP import by typing:
import shap
print(shap.__version__)

You should see 0.52.0 printed with no error messages.

  • Exit the Python shell by typing:
exit()

Seeing an import error?

If you see an error mentioning a missing dependency or compilation failure, make sure your virtual environment is active and try reinstalling with:

pip install --force-reinstall shap==0.52.0

Help me fix a SHAP import error in my Python 3.13 football-ml virtual environment.

Confirm your project files are ready

Your project uses two Python files: trustworthy_predictor.py for the training and analysis pipeline, and app.py for the Streamlit web app.

  • Confirm both files exist by running:
ls

You should see trustworthy_predictor.py and app.py listed in the output.

What are these files for?

The trustworthy_predictor.py script will contain the full training, SHAP explanation, and calibration pipeline. You will run it once to produce saved models, SHAP data, and calibration charts.

The app.py file will hold your Streamlit app that loads those saved artifacts and lets users interact with predictions and explanations in a browser.

Your environment is verified and ready. Next up, you will retrain your models and use SHAP to explain what drives each prediction.

Retrain Models and Generate SHAP Explanations

Your environment is ready with all the packages you need. Now it's time to answer the core question of this project: why does the model predict each match outcome?

A model that only says "home win" without explaining its reasoning is a black box. In this step, you will retrain both XGBoost and Random Forest on the football dataset, then use SHAP to crack open XGBoost's decision-making and see exactly which features drive each prediction.

In this step, get ready to:
  • Download and preprocess the football match dataset, then train XGBoost and Random Forest classifiers.
  • Generate a SHAP beeswarm plot showing global feature importance across predictions.
  • Generate a SHAP waterfall plot explaining why the model predicted a single specific match outcome.
Download the dataset, preprocess it, and train both classifiers

The football dataset lives on Hugging Face and contains international match results with numeric features like ELO ratings, goal averages, and tournament metadata.

You will reuse the same preprocessing logic from the first project: select numeric columns, drop high-null features, encode the target, and split into train and test sets.

  • Open trustworthy_predictor.py in VS Code and add the following imports at the top of the file:
import numpy as np
import pandas as pd
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
import shap
import joblib
from huggingface_hub import hf_hub_download
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.ensemble import RandomForestClassifier
from sklearn.calibration import CalibratedClassifierCV, calibration_curve
from sklearn.metrics import brier_score_loss
from xgboost import XGBClassifier

What do these imports do?

The imports pull in all the libraries you will need across this file. You are importing everything upfront so you do not need to add imports later.

  • matplotlib.use("Agg") sets a non-interactive backend so plots save to files without opening GUI windows.
  • shap provides the explainability tools for tree models.
  • joblib serializes trained models and arrays to disk so the Streamlit app can load them later.
  • Some imports (like CalibratedClassifierCV and calibration_curve) are used in the next step. Including them now keeps the file tidy.
  • You won't see a change yet. The next chunk uses these imports to load the data.
  • Below the imports, add the data download and preprocessing pipeline:


# ============================================================
# STEP 1: Download and Preprocess Data
# ============================================================
print("Downloading dataset from Hugging Face...")
file_path = hf_hub_download(
    repo_id="adibmed/football-predictions-dataset",
    filename="output/01_matches_all.csv",
    repo_type="dataset",
)
df = pd.read_csv(file_path, low_memory=False)
print(f"Dataset shape: {df.shape}")

# Derive target
if "result" not in df.columns:
    home_score_col = [
        c for c in df.columns
        if "home" in c.lower() and "score" in c.lower()
        or "home" in c.lower() and "goal" in c.lower() and "avg" not in c.lower()
    ]
    away_score_col = [
        c for c in df.columns
        if "away" in c.lower() and "score" in c.lower()
        or "away" in c.lower() and "goal" in c.lower() and "avg" not in c.lower()
    ]
    if home_score_col and away_score_col:
        df["result"] = np.where(
            df[home_score_col[0]] > df[away_score_col[0]], "home_win",
            np.where(df[home_score_col[0]] < df[away_score_col[0]], "away_win", "draw")
        )
    else:
        raise ValueError("Cannot find score columns to derive match result.")

# Select numeric features, drop high-null columns
numeric_df = df.select_dtypes(include=[np.number])
threshold = 0.3
null_pct = numeric_df.isnull().mean()
cols_to_keep = null_pct[null_pct <= threshold].index.tolist()
numeric_df = numeric_df[cols_to_keep]

# Drop remaining nulls
numeric_df = numeric_df.dropna()
df_clean = df.loc[numeric_df.index].copy()
df_clean[cols_to_keep] = numeric_df

# Prepare features and target
X = df_clean[cols_to_keep].values
le = LabelEncoder()
y = le.fit_transform(df_clean["result"])
class_names = le.classes_
feature_names = cols_to_keep

print(f"Classes: {class_names}")
print(f"Final dataset: {X.shape[0]} samples, {X.shape[1]} features")

# Train/test split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
print(f"Training set: {X_train.shape[0]} samples")
print(f"Test set: {X_test.shape[0]} samples")

What does this preprocessing do?

This pipeline mirrors what you built in the first football predictions project. It downloads the CSV from Hugging Face, derives a result column (home_win, away_win, or draw) from score columns, then filters to numeric features with fewer than 30% null values.

After dropping remaining nulls, it encodes the target with LabelEncoder and splits 80/20 for training and testing. The stratify=y argument ensures each class is proportionally represented in both splits.

  • You won't see output yet. The next chunk trains the models and produces the first terminal output.
  • Below the train/test split, add the model training and saving code:


# ============================================================
# STEP 2: Train Models and Generate SHAP Explanations
# ============================================================
print("\n" + "=" * 60)
print("TRAINING MODELS")
print("=" * 60)

# Train XGBoost
xgb_model = XGBClassifier(
    n_estimators=100, eval_metric="mlogloss", random_state=42
)
xgb_model.fit(X_train, y_train)
print("XGBoost trained.")

# Train Random Forest
rf_model = RandomForestClassifier(n_estimators=100, random_state=42)
rf_model.fit(X_train, y_train)
print("Random Forest trained.")

# Save models and preprocessing for Streamlit
joblib.dump(xgb_model, "xgb_model.joblib")
joblib.dump(rf_model, "rf_model.joblib")
joblib.dump(le, "label_encoder.joblib")
joblib.dump(feature_names, "feature_names.joblib")
joblib.dump(X_test, "X_test.joblib")
joblib.dump(y_test, "y_test.joblib")
print("Models and data saved.")

What does this code do?

You train two classifiers on the same data so you can compare them later during calibration analysis. XGBoost uses mlogloss as the evaluation metric because this is a 3-class problem (home_win, draw, away_win).

The joblib.dump calls serialize the trained models, the label encoder, feature names, and the test set to disk. Your Streamlit app will load these files later so it does not need to retrain from scratch every time a user visits.

  • Save trustworthy_predictor.py and run the script so far to verify the data loads and models train:
python trustworthy_predictor.py
  • You should see output showing the dataset shape, class names, training/test set sizes, and confirmation that both models trained and saved.

Seeing a download error?

  • Make sure you have an active internet connection. The script downloads the dataset from Hugging Face on the first run.
  • If you see a ModuleNotFoundError, confirm your football-ml virtual environment is activated.
  • If training takes longer than a minute, that is normal for 100 trees on ~49,000 samples.

Still stuck? Help me debug why trustworthy_predictor.py fails during data download or model training.

Create a SHAP TreeExplainer and generate the beeswarm summary plot

Now that both models are trained, you can use SHAP to peek inside XGBoost's decision-making. A beeswarm plot shows which features matter most across all predictions in the test set.

  • Add the following SHAP explainability code below the model-saving section in trustworthy_predictor.py:

# SHAP Explanations
print("\n" + "=" * 60)
print("SHAP EXPLAINABILITY")
print("=" * 60)

explainer = shap.TreeExplainer(xgb_model)
# Use a sample for performance
X_test_sample = X_test[:500]
shap_values = explainer(X_test_sample)

print(f"SHAP values shape: {shap_values.values.shape}")
print("Generating beeswarm summary plot...")

# Beeswarm summary plot (global importance)
plt.figure()
shap.summary_plot(
    shap_values[:, :, 0],
    X_test_sample,
    feature_names=feature_names,
    show=False,
    max_display=15,
)
plt.tight_layout()
plt.savefig("shap_summary.png", dpi=150, bbox_inches="tight")
plt.close()
print("Saved: shap_summary.png")

What does this code do?

shap.TreeExplainer uses an exact algorithm built into XGBoost's C++ implementation. It computes SHAP values in seconds because it integrates directly with the tree structure.

You subsample 500 rows from the test set for performance. Computing SHAP values on the full ~9,800 test samples with 100 trees would use significant memory. 500 samples produce stable visualizations.

The beeswarm plot (shap.summary_plot) shows the top 15 features ranked by impact. Each dot is one prediction. Its horizontal position shows whether that feature pushed the prediction higher or lower. Color indicates the feature value (red = high, blue = low).

  • Save trustworthy_predictor.py and run the script again:
python trustworthy_predictor.py
  • You should see Saved: shap_summary.png in the terminal output.
  • Confirm the file exists by running:
ls shap_summary.png
  • You should see shap_summary.png listed. Open it in your file browser or VS Code to see the beeswarm visualization.

Script hangs during SHAP computation?

  • SHAP computation on 500 samples with 100 trees can take 10-30 seconds. Give it time before assuming it is stuck.
  • If you run out of memory, close other applications. The SHAP values array for 500 samples with many features and 3 classes uses substantial RAM.

Still stuck? Help me debug why SHAP TreeExplainer hangs or crashes when computing values on 500 test samples with XGBoost.

Generate a waterfall plot for a single prediction

The beeswarm showed global importance. Now you will zoom into a single prediction and see exactly which features pushed it toward a specific outcome.

  • Add the waterfall plot code at the bottom of trustworthy_predictor.py, below the beeswarm section:

# Waterfall plot for a single prediction
print("\nGenerating waterfall plot for test sample 0...")
sample_idx = 0
prediction = xgb_model.predict(X_test_sample[sample_idx:sample_idx+1])[0]
print(f"Predicted class: {class_names[prediction]}")

plt.figure()
shap.plots.waterfall(shap_values[sample_idx, :, prediction], show=False, max_display=12)
plt.tight_layout()
plt.savefig("shap_waterfall.png", dpi=150, bbox_inches="tight")
plt.close()
print("Saved: shap_waterfall.png")

# Save explainer for Streamlit
joblib.dump(explainer, "shap_explainer.joblib")

What does this code do?

The waterfall plot visualizes one specific prediction (test sample 0). It shows the base value (the model's average prediction) and then each feature's contribution stacking on top, pushing the final score up or down.

shap_values[sample_idx, :, prediction] indexes the SHAP Explanation object: the first dimension selects sample 0, the second selects all features, and the third selects the predicted class. This gives you the SHAP values for the class the model actually chose.

Finally, the explainer itself is saved with joblib so the Streamlit app can generate waterfall plots for any match the user selects.

  • Save trustworthy_predictor.py and run the full script one more time:
python trustworthy_predictor.py
  • You should see Saved: shap_waterfall.png at the end of the output, along with the predicted class name for sample 0.
  • Confirm all saved files exist by running:
ls *.joblib *.png
  • You should see xgb_model.joblib, rf_model.joblib, label_encoder.joblib, feature_names.joblib, X_test.joblib, y_test.joblib, shap_explainer.joblib, shap_summary.png, and shap_waterfall.png listed.

Waterfall plot not saving?

  • If you see a warning about figure size, that is normal and the plot still saves correctly.
  • Make sure matplotlib.use("Agg") appears before import matplotlib.pyplot as plt in your imports. The order matters.
  • Check that show=False is passed to shap.plots.waterfall. Without it, the plot tries to display interactively and may hang.

Still stuck? Help me debug why the SHAP waterfall plot is not saving to shap_waterfall.png.

\u2714\uFE0F Awesome, I've got everything!

Great. Double check you have saved trustworthy_predictor.py and all 9 output files are present.

\u24E7 I'd like to double check the full code

Here is the complete trustworthy_predictor.py as it should look at the end of this step:

import numpy as np
import pandas as pd
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
import shap
import joblib
from huggingface_hub import hf_hub_download
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.ensemble import RandomForestClassifier
from sklearn.calibration import CalibratedClassifierCV, calibration_curve
from sklearn.metrics import brier_score_loss
from xgboost import XGBClassifier


# ============================================================
# STEP 1: Download and Preprocess Data
# ============================================================
print("Downloading dataset from Hugging Face...")
file_path = hf_hub_download(
    repo_id="adibmed/football-predictions-dataset",
    filename="output/01_matches_all.csv",
    repo_type="dataset",
)
df = pd.read_csv(file_path, low_memory=False)
print(f"Dataset shape: {df.shape}")

# Derive target
if "result" not in df.columns:
    home_score_col = [
        c for c in df.columns
        if "home" in c.lower() and "score" in c.lower()
        or "home" in c.lower() and "goal" in c.lower() and "avg" not in c.lower()
    ]
    away_score_col = [
        c for c in df.columns
        if "away" in c.lower() and "score" in c.lower()
        or "away" in c.lower() and "goal" in c.lower() and "avg" not in c.lower()
    ]
    if home_score_col and away_score_col:
        df["result"] = np.where(
            df[home_score_col[0]] > df[away_score_col[0]], "home_win",
            np.where(df[home_score_col[0]] < df[away_score_col[0]], "away_win", "draw")
        )
    else:
        raise ValueError("Cannot find score columns to derive match result.")

# Select numeric features, drop high-null columns
numeric_df = df.select_dtypes(include=[np.number])
threshold = 0.3
null_pct = numeric_df.isnull().mean()
cols_to_keep = null_pct[null_pct <= threshold].index.tolist()
numeric_df = numeric_df[cols_to_keep]

# Drop remaining nulls
numeric_df = numeric_df.dropna()
df_clean = df.loc[numeric_df.index].copy()
df_clean[cols_to_keep] = numeric_df

# Prepare features and target
X = df_clean[cols_to_keep].values
le = LabelEncoder()
y = le.fit_transform(df_clean["result"])
class_names = le.classes_
feature_names = cols_to_keep

print(f"Classes: {class_names}")
print(f"Final dataset: {X.shape[0]} samples, {X.shape[1]} features")

# Train/test split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
print(f"Training set: {X_train.shape[0]} samples")
print(f"Test set: {X_test.shape[0]} samples")


# ============================================================
# STEP 2: Train Models and Generate SHAP Explanations
# ============================================================
print("\n" + "=" * 60)
print("TRAINING MODELS")
print("=" * 60)

# Train XGBoost
xgb_model = XGBClassifier(
    n_estimators=100, eval_metric="mlogloss", random_state=42
)
xgb_model.fit(X_train, y_train)
print("XGBoost trained.")

# Train Random Forest
rf_model = RandomForestClassifier(n_estimators=100, random_state=42)
rf_model.fit(X_train, y_train)
print("Random Forest trained.")

# Save models and preprocessing for Streamlit
joblib.dump(xgb_model, "xgb_model.joblib")
joblib.dump(rf_model, "rf_model.joblib")
joblib.dump(le, "label_encoder.joblib")
joblib.dump(feature_names, "feature_names.joblib")
joblib.dump(X_test, "X_test.joblib")
joblib.dump(y_test, "y_test.joblib")
print("Models and data saved.")

# SHAP Explanations
print("\n" + "=" * 60)
print("SHAP EXPLAINABILITY")
print("=" * 60)

explainer = shap.TreeExplainer(xgb_model)
# Use a sample for performance
X_test_sample = X_test[:500]
shap_values = explainer(X_test_sample)

print(f"SHAP values shape: {shap_values.values.shape}")
print("Generating beeswarm summary plot...")

# Beeswarm summary plot (global importance)
plt.figure()
shap.summary_plot(
    shap_values[:, :, 0],
    X_test_sample,
    feature_names=feature_names,
    show=False,
    max_display=15,
)
plt.tight_layout()
plt.savefig("shap_summary.png", dpi=150, bbox_inches="tight")
plt.close()
print("Saved: shap_summary.png")

# Waterfall plot for a single prediction
print("\nGenerating waterfall plot for test sample 0...")
sample_idx = 0
prediction = xgb_model.predict(X_test_sample[sample_idx:sample_idx+1])[0]
print(f"Predicted class: {class_names[prediction]}")

plt.figure()
shap.plots.waterfall(shap_values[sample_idx, :, prediction], show=False, max_display=12)
plt.tight_layout()
plt.savefig("shap_waterfall.png", dpi=150, bbox_inches="tight")
plt.close()
print("Saved: shap_waterfall.png")

# Save explainer for Streamlit
joblib.dump(explainer, "shap_explainer.joblib")

You now have a trained XGBoost model with full SHAP explanations. You can see which features dominate globally and drill into any single prediction. Next up, you will find out whether you can actually trust the confidence scores these models produce.

Analyze Model Calibration

In the last step, you used SHAP to answer why the model predicts each outcome. But there's a second question that matters just as much for trust: when the model says "70% chance of a home win," does that actually happen about 70% of the time?

In this step, you'll measure how well-calibrated your models' probability estimates are. You'll compare XGBoost and Random Forest using reliability diagrams and Brier scores, then apply isotonic calibration to produce confidence scores you can actually trust.

In this step, get ready to:
  • Plot reliability diagrams comparing XGBoost and Random Forest calibration.
  • Apply isotonic calibration to improve XGBoost's probability estimates.
  • Compare model trustworthiness with Brier scores and determine which model to deploy.
Plot reliability diagrams for both models

A reliability diagram (also called a calibration curve) plots predicted probabilities on the x-axis against actual observed frequencies on the y-axis. A perfectly calibrated model hugs the diagonal line.

Since you have a 3-class problem (home win, draw, away win), you'll use a one-vs-rest approach: binarize each class and plot its calibration curve separately.

  • In trustworthy_predictor.py, scroll to the bottom of the file (after the line joblib.dump(explainer, "shap_explainer.joblib")).
  • Add the calibration analysis section by copying the following code below it:


# ============================================================
# STEP 3: Calibration Analysis
# ============================================================
print("\n" + "=" * 60)
print("CALIBRATION ANALYSIS")
print("=" * 60)

# Get predicted probabilities
xgb_probs = xgb_model.predict_proba(X_test)
rf_probs = rf_model.predict_proba(X_test)

# Plot calibration curves per class (one-vs-rest)
fig, axes = plt.subplots(1, 3, figsize=(15, 5))

for i, class_name in enumerate(class_names):
    y_binary = (y_test == i).astype(int)

    # XGBoost calibration curve
    prob_true_xgb, prob_pred_xgb = calibration_curve(
        y_binary, xgb_probs[:, i], n_bins=10, strategy="uniform"
    )
    # Random Forest calibration curve
    prob_true_rf, prob_pred_rf = calibration_curve(
        y_binary, rf_probs[:, i], n_bins=10, strategy="uniform"
    )

    axes[i].plot([0, 1], [0, 1], "k--", label="Perfect calibration")
    axes[i].plot(prob_pred_xgb, prob_true_xgb, "s-", label="XGBoost", color="steelblue")
    axes[i].plot(prob_pred_rf, prob_true_rf, "o-", label="Random Forest", color="coral")
    axes[i].set_xlabel("Mean predicted probability")
    axes[i].set_ylabel("Fraction of positives")
    axes[i].set_title(f"Calibration: {class_name}")
    axes[i].legend(loc="lower right")
    axes[i].grid(True, alpha=0.3)

plt.tight_layout()
plt.savefig("calibration_comparison.png", dpi=150, bbox_inches="tight")
plt.close()
print("Saved: calibration_comparison.png")

What does this code do?

predict_proba(X_test) gets each model's confidence scores for every test sample across all 3 classes.

calibration_curve bins the predictions and compares them to actual outcomes. If the model says "80% home win" for a group of matches, this checks whether home wins actually occur about 80% of the time in that group.

The dashed diagonal line represents perfect calibration. The closer a model's curve hugs that line, the more trustworthy its confidence scores are.

Now add the Brier score computation directly below. The Brier score gives you a single number summarizing calibration quality (lower is better).

  • Add the Brier score computation below the print("Saved: calibration_comparison.png") line:

# Brier scores
print("\nBrier Scores (lower is better):")
for i, class_name in enumerate(class_names):
    y_binary = (y_test == i).astype(int)
    brier_xgb = brier_score_loss(y_binary, xgb_probs[:, i])
    brier_rf = brier_score_loss(y_binary, rf_probs[:, i])
    print(f"  {class_name}: XGBoost={brier_xgb:.4f}, Random Forest={brier_rf:.4f}")

What is a Brier score?

The Brier score is the mean squared difference between predicted probabilities and actual outcomes. A score of 0.0 means perfect calibration. A score of 0.25 is no better than random guessing for a binary event.

You compute it per-class using the one-vs-rest approach, just like the reliability diagrams.

  • Save trustworthy_predictor.py.
  • Run the script in your terminal by running:
python trustworthy_predictor.py
  • You should see Saved: calibration_comparison.png printed in the terminal, followed by Brier scores for each class and each model.
  • Open calibration_comparison.png in your file explorer. You should see three side-by-side reliability diagrams (one per class) with XGBoost in blue and Random Forest in coral plotted against the perfect calibration diagonal.

Not seeing the Brier scores or plot?

Make sure you saved trustworthy_predictor.py before running it. Also check that the new code appears after the SHAP section, not inside it.

If you see an IndentationError, make sure the new section starts at the top level (no indentation) and is not nested inside the previous for loop.

Help me debug my calibration analysis code

Calibrate XGBoost with isotonic regression

Looking at your reliability diagrams, you'll likely notice that neither model hugs the diagonal perfectly. You can fix this for XGBoost using CalibratedClassifierCV, which learns a post-hoc mapping from raw probabilities to well-calibrated ones.

Why isotonic regression over sigmoid?

Isotonic calibration is more flexible because it makes no assumption about the shape of the miscalibration. Sigmoid calibration assumes a specific S-shaped correction curve, which may not hold for tree-based models.

With ~49,000 training samples in this dataset, you have plenty of data for isotonic calibration to fit reliably without overfitting.

  • Add the calibration and before/after comparison code below the Brier scores section:

# Calibrate XGBoost with isotonic regression
print("\nCalibrating XGBoost with isotonic regression...")
xgb_calibrated = CalibratedClassifierCV(xgb_model, cv=5, method="isotonic")
xgb_calibrated.fit(X_train, y_train)

# Compare before and after calibration
xgb_cal_probs = xgb_calibrated.predict_proba(X_test)

fig, axes = plt.subplots(1, 3, figsize=(15, 5))
for i, class_name in enumerate(class_names):
    y_binary = (y_test == i).astype(int)

    prob_true_before, prob_pred_before = calibration_curve(
        y_binary, xgb_probs[:, i], n_bins=10, strategy="uniform"
    )
    prob_true_after, prob_pred_after = calibration_curve(
        y_binary, xgb_cal_probs[:, i], n_bins=10, strategy="uniform"
    )

    axes[i].plot([0, 1], [0, 1], "k--", label="Perfect")
    axes[i].plot(prob_pred_before, prob_true_before, "s-", label="Before calibration", color="steelblue")
    axes[i].plot(prob_pred_after, prob_true_after, "o-", label="After calibration", color="green")
    axes[i].set_xlabel("Mean predicted probability")
    axes[i].set_ylabel("Fraction of positives")
    axes[i].set_title(f"{class_name}: Before vs After")
    axes[i].legend(loc="lower right")
    axes[i].grid(True, alpha=0.3)

plt.tight_layout()
plt.savefig("calibration_before_after.png", dpi=150, bbox_inches="tight")
plt.close()
print("Saved: calibration_before_after.png")

What does this code do?

CalibratedClassifierCV(xgb_model, cv=5, method="isotonic") wraps the trained XGBoost model and uses 5-fold cross-validation to learn an isotonic mapping from raw probabilities to calibrated ones.

The before/after plot shows the same three classes, but now compares the original XGBoost curve (blue) with the post-calibration curve (green). Wherever the green line is closer to the diagonal, calibration improved the model's honesty.

  • Save trustworthy_predictor.py and run the script again:
python trustworthy_predictor.py
  • You should see Saved: calibration_before_after.png in the terminal output.
  • Open calibration_before_after.png. You should see three plots comparing the blue (before) and green (after) calibration curves. The green curves should be noticeably closer to the diagonal on at least one class.

Calibration taking too long?

CalibratedClassifierCV with cv=5 retrains XGBoost 5 times internally. On this dataset it should take under 30 seconds. If it's much slower, make sure no other heavy processes are running.

Help me with my CalibratedClassifierCV code

Print the calibration summary and save the calibrated model

Now you'll produce a final comparison table that answers the key question: which model should you deploy if trustworthiness matters?

  • Add the summary table and model save below the print("Saved: calibration_before_after.png") line:

# Summary table
print("\n" + "-" * 60)
print("CALIBRATION SUMMARY")
print("-" * 60)
print(f"{'Model':<30} {'Brier (avg)':<15}")
print("-" * 60)

brier_avg_xgb = np.mean([brier_score_loss((y_test == i).astype(int), xgb_probs[:, i]) for i in range(3)])
brier_avg_rf = np.mean([brier_score_loss((y_test == i).astype(int), rf_probs[:, i]) for i in range(3)])
brier_avg_cal = np.mean([brier_score_loss((y_test == i).astype(int), xgb_cal_probs[:, i]) for i in range(3)])

print(f"{'XGBoost (uncalibrated)':<30} {brier_avg_xgb:<15.4f}")
print(f"{'XGBoost (calibrated)':<30} {brier_avg_cal:<15.4f}")
print(f"{'Random Forest':<30} {brier_avg_rf:<15.4f}")
print("-" * 60)

# Save calibrated model
joblib.dump(xgb_calibrated, "xgb_calibrated.joblib")
print("\nCalibrated model saved: xgb_calibrated.joblib")

print("\n" + "=" * 60)
print("PIPELINE COMPLETE")
print("=" * 60)
print("Generated outputs:")
print("  - shap_summary.png")
print("  - shap_waterfall.png")
print("  - calibration_comparison.png")
print("  - calibration_before_after.png")
print("  - xgb_model.joblib, rf_model.joblib, xgb_calibrated.joblib")
print("  - shap_explainer.joblib, label_encoder.joblib")
print("  - feature_names.joblib, X_test.joblib, y_test.joblib")

What does this code do?

The summary table averages the Brier score across all three classes for each model variant. This gives you one number to compare: lower means more trustworthy probability estimates.

joblib.dump(xgb_calibrated, "xgb_calibrated.joblib") saves the calibrated model so your Streamlit app can load it directly and serve trustworthy predictions without re-running calibration.

  • Save trustworthy_predictor.py and run the script one final time:
python trustworthy_predictor.py
  • You should see the CALIBRATION SUMMARY table in the terminal with average Brier scores for all three model variants, followed by PIPELINE COMPLETE.
  • Confirm that xgb_calibrated.joblib now exists in your project folder by running:
ls *.joblib
  • You should see xgb_calibrated.joblib listed alongside xgb_model.joblib, rf_model.joblib, and the other saved artifacts.

Brier scores look identical before and after?

Make sure you're comparing xgb_probs (uncalibrated) versus xgb_cal_probs (calibrated) in the summary table. If you accidentally used the same variable for both, the scores will match.

Also confirm that xgb_calibrated.fit(X_train, y_train) appears before the line that computes xgb_cal_probs. The calibrated model must be fitted before generating predictions.

Help me debug my calibration summary output

✔️ Awesome, I've got everything!

Great. Double check you've saved trustworthy_predictor.py and that the PIPELINE COMPLETE message printed without errors.

ⓧ I'd like to double check the full code

Here's the complete trustworthy_predictor.py file as it should look at the end of this step:

import numpy as np
import pandas as pd
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
import shap
import joblib
from huggingface_hub import hf_hub_download
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.ensemble import RandomForestClassifier
from sklearn.calibration import CalibratedClassifierCV, calibration_curve
from sklearn.metrics import brier_score_loss
from xgboost import XGBClassifier


# ============================================================
# STEP 1: Download and Preprocess Data
# ============================================================
print("Downloading dataset from Hugging Face...")
file_path = hf_hub_download(
    repo_id="adibmed/football-predictions-dataset",
    filename="output/01_matches_all.csv",
    repo_type="dataset",
)
df = pd.read_csv(file_path, low_memory=False)
print(f"Dataset shape: {df.shape}")

# Derive target
if "result" not in df.columns:
    home_score_col = [
        c for c in df.columns
        if "home" in c.lower() and "score" in c.lower()
        or "home" in c.lower() and "goal" in c.lower() and "avg" not in c.lower()
    ]
    away_score_col = [
        c for c in df.columns
        if "away" in c.lower() and "score" in c.lower()
        or "away" in c.lower() and "goal" in c.lower() and "avg" not in c.lower()
    ]
    if home_score_col and away_score_col:
        df["result"] = np.where(
            df[home_score_col[0]] > df[away_score_col[0]], "home_win",
            np.where(df[home_score_col[0]] < df[away_score_col[0]], "away_win", "draw")
        )
    else:
        raise ValueError("Cannot find score columns to derive match result.")

# Select numeric features, drop high-null columns
numeric_df = df.select_dtypes(include=[np.number])
threshold = 0.3
null_pct = numeric_df.isnull().mean()
cols_to_keep = null_pct[null_pct <= threshold].index.tolist()
numeric_df = numeric_df[cols_to_keep]

# Drop remaining nulls
numeric_df = numeric_df.dropna()
df_clean = df.loc[numeric_df.index].copy()
df_clean[cols_to_keep] = numeric_df

# Prepare features and target
X = df_clean[cols_to_keep].values
le = LabelEncoder()
y = le.fit_transform(df_clean["result"])
class_names = le.classes_
feature_names = cols_to_keep

print(f"Classes: {class_names}")
print(f"Final dataset: {X.shape[0]} samples, {X.shape[1]} features")

# Train/test split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
print(f"Training set: {X_train.shape[0]} samples")
print(f"Test set: {X_test.shape[0]} samples")


# ============================================================
# STEP 2: Train Models and Generate SHAP Explanations
# ============================================================
print("\n" + "=" * 60)
print("TRAINING MODELS")
print("=" * 60)

# Train XGBoost
xgb_model = XGBClassifier(
    n_estimators=100, eval_metric="mlogloss", random_state=42
)
xgb_model.fit(X_train, y_train)
print("XGBoost trained.")

# Train Random Forest
rf_model = RandomForestClassifier(n_estimators=100, random_state=42)
rf_model.fit(X_train, y_train)
print("Random Forest trained.")

# Save models and preprocessing for Streamlit
joblib.dump(xgb_model, "xgb_model.joblib")
joblib.dump(rf_model, "rf_model.joblib")
joblib.dump(le, "label_encoder.joblib")
joblib.dump(feature_names, "feature_names.joblib")
joblib.dump(X_test, "X_test.joblib")
joblib.dump(y_test, "y_test.joblib")
print("Models and data saved.")

# SHAP Explanations
print("\n" + "=" * 60)
print("SHAP EXPLAINABILITY")
print("=" * 60)

explainer = shap.TreeExplainer(xgb_model)
# Use a sample for performance
X_test_sample = X_test[:500]
shap_values = explainer(X_test_sample)

print(f"SHAP values shape: {shap_values.values.shape}")
print("Generating beeswarm summary plot...")

# Beeswarm summary plot (global importance)
plt.figure()
shap.summary_plot(
    shap_values[:, :, 0],
    X_test_sample,
    feature_names=feature_names,
    show=False,
    max_display=15,
)
plt.tight_layout()
plt.savefig("shap_summary.png", dpi=150, bbox_inches="tight")
plt.close()
print("Saved: shap_summary.png")

# Waterfall plot for a single prediction
print("\nGenerating waterfall plot for test sample 0...")
sample_idx = 0
prediction = xgb_model.predict(X_test_sample[sample_idx:sample_idx+1])[0]
print(f"Predicted class: {class_names[prediction]}")

plt.figure()
shap.plots.waterfall(shap_values[sample_idx, :, prediction], show=False, max_display=12)
plt.tight_layout()
plt.savefig("shap_waterfall.png", dpi=150, bbox_inches="tight")
plt.close()
print("Saved: shap_waterfall.png")

# Save explainer for Streamlit
joblib.dump(explainer, "shap_explainer.joblib")


# ============================================================
# STEP 3: Calibration Analysis
# ============================================================
print("\n" + "=" * 60)
print("CALIBRATION ANALYSIS")
print("=" * 60)

# Get predicted probabilities
xgb_probs = xgb_model.predict_proba(X_test)
rf_probs = rf_model.predict_proba(X_test)

# Plot calibration curves per class (one-vs-rest)
fig, axes = plt.subplots(1, 3, figsize=(15, 5))

for i, class_name in enumerate(class_names):
    y_binary = (y_test == i).astype(int)

    # XGBoost calibration curve
    prob_true_xgb, prob_pred_xgb = calibration_curve(
        y_binary, xgb_probs[:, i], n_bins=10, strategy="uniform"
    )
    # Random Forest calibration curve
    prob_true_rf, prob_pred_rf = calibration_curve(
        y_binary, rf_probs[:, i], n_bins=10, strategy="uniform"
    )

    axes[i].plot([0, 1], [0, 1], "k--", label="Perfect calibration")
    axes[i].plot(prob_pred_xgb, prob_true_xgb, "s-", label="XGBoost", color="steelblue")
    axes[i].plot(prob_pred_rf, prob_true_rf, "o-", label="Random Forest", color="coral")
    axes[i].set_xlabel("Mean predicted probability")
    axes[i].set_ylabel("Fraction of positives")
    axes[i].set_title(f"Calibration: {class_name}")
    axes[i].legend(loc="lower right")
    axes[i].grid(True, alpha=0.3)

plt.tight_layout()
plt.savefig("calibration_comparison.png", dpi=150, bbox_inches="tight")
plt.close()
print("Saved: calibration_comparison.png")

# Brier scores
print("\nBrier Scores (lower is better):")
for i, class_name in enumerate(class_names):
    y_binary = (y_test == i).astype(int)
    brier_xgb = brier_score_loss(y_binary, xgb_probs[:, i])
    brier_rf = brier_score_loss(y_binary, rf_probs[:, i])
    print(f"  {class_name}: XGBoost={brier_xgb:.4f}, Random Forest={brier_rf:.4f}")

# Calibrate XGBoost with isotonic regression
print("\nCalibrating XGBoost with isotonic regression...")
xgb_calibrated = CalibratedClassifierCV(xgb_model, cv=5, method="isotonic")
xgb_calibrated.fit(X_train, y_train)

# Compare before and after calibration
xgb_cal_probs = xgb_calibrated.predict_proba(X_test)

fig, axes = plt.subplots(1, 3, figsize=(15, 5))
for i, class_name in enumerate(class_names):
    y_binary = (y_test == i).astype(int)

    prob_true_before, prob_pred_before = calibration_curve(
        y_binary, xgb_probs[:, i], n_bins=10, strategy="uniform"
    )
    prob_true_after, prob_pred_after = calibration_curve(
        y_binary, xgb_cal_probs[:, i], n_bins=10, strategy="uniform"
    )

    axes[i].plot([0, 1], [0, 1], "k--", label="Perfect")
    axes[i].plot(prob_pred_before, prob_true_before, "s-", label="Before calibration", color="steelblue")
    axes[i].plot(prob_pred_after, prob_true_after, "o-", label="After calibration", color="green")
    axes[i].set_xlabel("Mean predicted probability")
    axes[i].set_ylabel("Fraction of positives")
    axes[i].set_title(f"{class_name}: Before vs After")
    axes[i].legend(loc="lower right")
    axes[i].grid(True, alpha=0.3)

plt.tight_layout()
plt.savefig("calibration_before_after.png", dpi=150, bbox_inches="tight")
plt.close()
print("Saved: calibration_before_after.png")

# Summary table
print("\n" + "-" * 60)
print("CALIBRATION SUMMARY")
print("-" * 60)
print(f"{'Model':<30} {'Brier (avg)':<15}")
print("-" * 60)

brier_avg_xgb = np.mean([brier_score_loss((y_test == i).astype(int), xgb_probs[:, i]) for i in range(3)])
brier_avg_rf = np.mean([brier_score_loss((y_test == i).astype(int), rf_probs[:, i]) for i in range(3)])
brier_avg_cal = np.mean([brier_score_loss((y_test == i).astype(int), xgb_cal_probs[:, i]) for i in range(3)])

print(f"{'XGBoost (uncalibrated)':<30} {brier_avg_xgb:<15.4f}")
print(f"{'XGBoost (calibrated)':<30} {brier_avg_cal:<15.4f}")
print(f"{'Random Forest':<30} {brier_avg_rf:<15.4f}")
print("-" * 60)

# Save calibrated model
joblib.dump(xgb_calibrated, "xgb_calibrated.joblib")
print("\nCalibrated model saved: xgb_calibrated.joblib")

print("\n" + "=" * 60)
print("PIPELINE COMPLETE")
print("=" * 60)
print("Generated outputs:")
print("  - shap_summary.png")
print("  - shap_waterfall.png")
print("  - calibration_comparison.png")
print("  - calibration_before_after.png")
print("  - xgb_model.joblib, rf_model.joblib, xgb_calibrated.joblib")
print("  - shap_explainer.joblib, label_encoder.joblib")
print("  - feature_names.joblib, X_test.joblib, y_test.joblib")

Your calibration pipeline is complete. You now have quantitative proof of which model produces the most honest confidence scores. Next up, you'll put this calibrated model into the hands of users by building and deploying an interactive Streamlit app.

Build and Deploy the Streamlit App

You have a calibrated model that produces trustworthy confidence scores and a SHAP explainer that reveals why each prediction was made. Now it is time to put these pieces together into something anyone can use.

In this step, you will build a Streamlit web app that loads your saved models, lets users pick a match, and displays calibrated predictions alongside a SHAP waterfall explanation. Then you will deploy it to Streamlit Community Cloud so it is live on the internet.

In this step, get ready to:
  • Build the Streamlit app with match selection, calibrated probabilities, and SHAP explanations.
  • Create a requirements.txt, push everything to GitHub, and deploy to Streamlit Community Cloud.
  • Test the deployed app with different match selections.
Build the Streamlit app layout

Your app needs to load the saved models once, present a match selector, and render predictions with explanations. The @st.cache_resource decorator ensures models load only once per session, keeping the app responsive.

  • Open app.py in VS Code (it is currently empty).
  • Add the imports and page configuration by pasting this code:
import numpy as np
import streamlit as st
import shap
import joblib
import matplotlib.pyplot as plt


st.set_page_config(page_title="Football Match Predictor", layout="wide")
st.title("Football Match Predictor: Explainable and Trustworthy")
st.markdown(
    "This app uses a **calibrated XGBoost model** to predict international football match outcomes. "
    "Every prediction comes with calibrated confidence scores and a SHAP explanation."
)

What does this code do?

The imports pull in your ML stack: numpy for array operations, shap for explanations, joblib for loading saved models, and matplotlib for rendering SHAP plots.

st.set_page_config sets the browser tab title and uses a wide layout. The title and markdown give users context about what the app does.

  • Next, add the model loading function and unpack its return values by pasting this below:

@st.cache_resource
def load_models():
    xgb_calibrated = joblib.load("xgb_calibrated.joblib")
    explainer = joblib.load("shap_explainer.joblib")
    le = joblib.load("label_encoder.joblib")
    feature_names = joblib.load("feature_names.joblib")
    X_test = joblib.load("X_test.joblib")
    y_test = joblib.load("y_test.joblib")
    return xgb_calibrated, explainer, le, feature_names, X_test, y_test


xgb_calibrated, explainer, le, feature_names, X_test, y_test = load_models()
class_names = le.classes_

What does this code do?

The @st.cache_resource decorator tells Streamlit to run load_models() only once and cache the results. Without caching, every user interaction would reload all the .joblib files from disk.

The function loads six artifacts you saved in the previous steps: the calibrated model, the SHAP explainer, the label encoder, feature names, test features, and test labels.

  • Add the sidebar match selector, prediction logic, and metrics display by pasting this below:

# Match selection
st.sidebar.header("Select a Match")
match_idx = st.sidebar.selectbox(
    "Choose a test match (index):",
    options=range(min(200, len(X_test))),
    format_func=lambda i: f"Match {i} (Actual: {class_names[y_test[i]]})",
)

# Prediction
sample = X_test[match_idx:match_idx + 1]
probs = xgb_calibrated.predict_proba(sample)[0]
predicted_class = np.argmax(probs)

# Display prediction with calibrated confidence
st.header("Prediction")
col1, col2, col3 = st.columns(3)
col1.metric("Home Win", f"{probs[class_names.tolist().index('home_win')]:.1%}")
col2.metric("Draw", f"{probs[class_names.tolist().index('draw')]:.1%}")
col3.metric("Away Win", f"{probs[class_names.tolist().index('away_win')]:.1%}")

st.markdown(
    f"**Predicted outcome:** {class_names[predicted_class]} | "
    f"**Actual outcome:** {class_names[y_test[match_idx]]}"
)

What does this code do?

st.sidebar.selectbox creates a dropdown in the sidebar showing matches 0 through 199. The format_func displays the actual outcome next to each index so users can verify predictions against reality.

The calibrated model's predict_proba returns honest confidence scores (thanks to isotonic calibration from the previous step). The three st.metric widgets display these as large, readable percentages in a three-column layout.

  • Finally, add the SHAP explanation section and footer by pasting this below:

# SHAP explanation
st.header("Why did the model predict this?")
st.markdown(
    "The waterfall plot below shows which features pushed the prediction "
    "toward or away from the predicted class."
)

shap_values_single = explainer(sample)
fig, ax = plt.subplots()
shap.plots.waterfall(shap_values_single[0, :, predicted_class], show=False, max_display=12)
plt.tight_layout()
st.pyplot(fig)
plt.close()

# Footer
st.markdown("---")
st.markdown(
    "Built with SHAP for explainability, sklearn CalibratedClassifierCV for "
    "calibration, and Streamlit for deployment."
)

What does this code do?

The SHAP explainer computes feature contributions for the selected match. The waterfall plot visualizes exactly which features pushed the prediction toward or away from the predicted class, capped at 12 features for readability.

st.pyplot(fig) renders the matplotlib figure directly in the Streamlit app. The footer provides attribution for the tools used.

  • Save app.py.
  • Test the app locally by running this command:
streamlit run app.py

Your browser opens automatically to a local URL. You should see the app title, a sidebar dropdown, three metric cards showing percentages, and a SHAP waterfall chart below.

App not loading or showing errors?

  • If you see a FileNotFoundError for a .joblib file, make sure you ran trustworthy_predictor.py to completion in the same directory.
  • If the SHAP plot does not render, ensure shap is installed in your active football-ml virtual environment.
  • Press Ctrl+C in the terminal when you are done testing locally.

help me debug my Streamlit app that won't load or shows a missing file error

✔️ Awesome, I've got everything!

Great. Double check you have saved app.py and that the local app loaded without errors.

ⓧ I'd like to double check the full code

Here is the complete app.py file:

import numpy as np
import streamlit as st
import shap
import joblib
import matplotlib.pyplot as plt


st.set_page_config(page_title="Football Match Predictor", layout="wide")
st.title("Football Match Predictor: Explainable and Trustworthy")
st.markdown(
    "This app uses a **calibrated XGBoost model** to predict international football match outcomes. "
    "Every prediction comes with calibrated confidence scores and a SHAP explanation."
)


@st.cache_resource
def load_models():
    xgb_calibrated = joblib.load("xgb_calibrated.joblib")
    explainer = joblib.load("shap_explainer.joblib")
    le = joblib.load("label_encoder.joblib")
    feature_names = joblib.load("feature_names.joblib")
    X_test = joblib.load("X_test.joblib")
    y_test = joblib.load("y_test.joblib")
    return xgb_calibrated, explainer, le, feature_names, X_test, y_test


xgb_calibrated, explainer, le, feature_names, X_test, y_test = load_models()
class_names = le.classes_

# Match selection
st.sidebar.header("Select a Match")
match_idx = st.sidebar.selectbox(
    "Choose a test match (index):",
    options=range(min(200, len(X_test))),
    format_func=lambda i: f"Match {i} (Actual: {class_names[y_test[i]]})",
)

# Prediction
sample = X_test[match_idx:match_idx + 1]
probs = xgb_calibrated.predict_proba(sample)[0]
predicted_class = np.argmax(probs)

# Display prediction with calibrated confidence
st.header("Prediction")
col1, col2, col3 = st.columns(3)
col1.metric("Home Win", f"{probs[class_names.tolist().index('home_win')]:.1%}")
col2.metric("Draw", f"{probs[class_names.tolist().index('draw')]:.1%}")
col3.metric("Away Win", f"{probs[class_names.tolist().index('away_win')]:.1%}")

st.markdown(
    f"**Predicted outcome:** {class_names[predicted_class]} | "
    f"**Actual outcome:** {class_names[y_test[match_idx]]}"
)

# SHAP explanation
st.header("Why did the model predict this?")
st.markdown(
    "The waterfall plot below shows which features pushed the prediction "
    "toward or away from the predicted class."
)

shap_values_single = explainer(sample)
fig, ax = plt.subplots()
shap.plots.waterfall(shap_values_single[0, :, predicted_class], show=False, max_display=12)
plt.tight_layout()
st.pyplot(fig)
plt.close()

# Footer
st.markdown("---")
st.markdown(
    "Built with SHAP for explainability, sklearn CalibratedClassifierCV for "
    "calibration, and Streamlit for deployment."
)
Create requirements.txt, push to GitHub, and deploy

Streamlit Community Cloud reads a requirements.txt from your repository to know which packages to install. You need to pin every dependency so the cloud environment matches your local setup exactly.

  • Create requirements.txt in your project folder and paste this content:
streamlit==1.58.0
shap==0.52.0
xgboost==2.1.4
scikit-learn==1.6.1
pandas==2.2.3
numpy==2.2.3
matplotlib==3.10.1
joblib==1.4.2
huggingface_hub==0.30.2

Why pin versions?

Pinning exact versions ensures Streamlit Community Cloud installs the same package versions you developed with locally. Without pins, a new release of any dependency could break your deployed app unexpectedly.

  • Save requirements.txt.
  • Initialize a Git repository and make your first commit by running these commands:
git init
git add .
git commit -m "Initial commit: trustworthy football predictor"

What do these commands do?

git init creates a new Git repository in your project folder. git add . stages all files (Python scripts, models, PNGs, requirements). git commit saves a snapshot of everything.

  • Go to GitHub in your browser and create a new public repository (name it something like football-predictor-explainable).
  • Do not initialize it with a README or .gitignore (your local repo already has content).
  • Copy the remote URL GitHub provides and connect your local repo by running:
git remote add origin [[REPO_URL="https://github.com/your-username/football-predictor-explainable.git"]]
git branch -M main
git push -u origin main
  • Verify the push succeeded by refreshing your GitHub repository page. You should see app.py, trustworthy_predictor.py, requirements.txt, and all .joblib files listed.

Push rejected or authentication error?

  • Make sure you are signed into GitHub in your terminal. If prompted, use a personal access token instead of a password.
  • If the remote URL is wrong, fix it with git remote set-url origin <correct-url> and try pushing again.

help me fix a git push authentication error to GitHub

Now deploy to Streamlit Community Cloud.

  • Go to share.streamlit.io in your browser.
  • Click Continue with GitHub and authorize Streamlit to access your repositories.
  • In the upper-right corner of your workspace, click Create app.
  • When asked if you already have an app, select Yes, I have an app.
  • Fill in your repository name, branch (main), and main file path (app.py).
  • Click Deploy.

Streamlit Community Cloud will clone your repository, install your dependencies from requirements.txt, and launch your app. This typically takes 2-5 minutes.

What is Streamlit Community Cloud?

Streamlit Community Cloud is a free hosting service for public Streamlit apps. It connects directly to your GitHub repository and automatically redeploys whenever you push new code.

Your deployed app gets a permanent URL in the format your-app-name.streamlit.app that anyone can visit.

Deployment failing with dependency errors?

  • Click Manage app in the lower-right corner of the deployed app to view logs.
  • Check that your requirements.txt is in the repository root (not inside a subfolder).
  • If you see memory errors, ensure all .joblib files were pushed to GitHub (they should total less than 50MB combined).

help me debug a Streamlit Community Cloud deployment failure

Test the deployed app

Once deployment finishes, Streamlit redirects you to your live app URL. The app should look identical to your local version.

  • In the sidebar dropdown, select Match 0 and confirm the prediction metrics and waterfall chart appear.
  • Switch to Match 5 and verify that the probabilities and SHAP waterfall update to reflect a different prediction.
  • Try a match where the model got it wrong (look for one where the predicted outcome differs from the actual outcome in the dropdown label). Inspect the SHAP waterfall to see which features misled the model.

Each time you select a new match, the calibrated probabilities update and the SHAP waterfall recomputes. This confirms the full pipeline works end to end.

App loads but waterfall chart is blank?

  • This can happen if the SHAP explainer was trained on a different version of XGBoost than what Streamlit Community Cloud installed. Confirm that xgboost==2.1.4 is in your requirements.txt.
  • Try rebooting the app from the Manage app menu to force a fresh install.

help me fix a blank SHAP waterfall plot in my deployed Streamlit app

Your prediction system is now live on the internet. Anyone with the link can explore match predictions, see calibrated confidence scores, and understand exactly why the model made each call.

Secret mission

SHAP Dependence Plots for Feature Interactions

Your beeswarm plot shows which features matter most globally. But it hides something important: how features interact with each other to change predictions. Two matches with the same ELO difference can get wildly different predictions depending on recent form. Can you reveal those hidden interactions?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely on free services. Your Streamlit Community Cloud app runs at no cost on the free tier, and your GitHub repository is free.

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Streamlit Community Cloud app (free tier, no ongoing cost).
  • GitHub repository (free).
  • Local files: trustworthy_predictor.py, app.py, requirements.txt, saved models (.joblib files), and PNG visualizations.

Keep everything running

No action needed. Choose this if you are still actively building or want your app to stay accessible.

  • Your Streamlit Community Cloud app stays live indefinitely on the free tier. There is no cost.
  • If your app receives no traffic for 12 hours, Streamlit Community Cloud automatically puts it to sleep to conserve resources. Anyone who visits the URL can wake it up by clicking the on-screen prompt.
  • Your GitHub repository remains available at no charge.
  • Your local files remain in place on your machine.

Pause - I'll come back to this later

Free up local disk space while keeping your cloud resources intact. Your app will sleep automatically when it has no visitors.

  • Streamlit Community Cloud apps without traffic for 12 hours go to sleep automatically. There is no manual pause button needed. When someone visits the URL later, they can wake it up instantly.
  • Optionally remove the local model files to free disk space by running:
rm -f xgb_model.joblib rf_model.joblib xgb_calibrated.joblib shap_explainer.joblib label_encoder.joblib feature_names.joblib X_test.joblib y_test.joblib
  • Keep trustworthy_predictor.py and app.py so you can regenerate the models later by re-running the script.

Delete - I don't want to use this again

Remove all project resources completely and start fresh.

  • Delete your Streamlit Community Cloud app:
  • Visit your deployed app URL (your .streamlit.app address).
  • Click Manage app in the lower-right corner.
  • Click Delete app.
  • Delete your GitHub repository:
  • On GitHub, navigate to the main page of your repository.
  • Click Settings.
  • Scroll down to the Danger Zone section and click Delete this repository.
  • Confirm by typing the repository name, then click Delete this repository.
  • Delete your local project files:

On macOS/Linux:

cd ~/Desktop
rm -rf trustworthy_predictor.py app.py requirements.txt *.joblib *.png

On Windows:

cd ~\Desktop
Remove-Item trustworthy_predictor.py, app.py, requirements.txt, *.joblib, *.png -Force

That's a Wrap!

That's a Wrap!

Nice work! You have taken a trained machine learning model and transformed it into something you can actually explain and trust.

You now have a live, deployed web app that proves you can move a model from "it works" to "I can explain why and trust its confidence scores."

You've learned how to:

  • Use SHAP TreeExplainer to generate both global (beeswarm) and local (waterfall) explanations that answer "why did the model predict this?" for each match outcome.
  • Run a calibration analysis comparing model trustworthiness with reliability diagrams and Brier scores, then apply isotonic calibration to improve XGBoost's probability estimates.
  • Build and deploy a Streamlit app to Community Cloud that lets users interact with calibrated predictions and per-prediction SHAP explanations.
  • Secret Mission: Added SHAP dependence plots revealing feature interaction effects and integrated them into the Streamlit app.

Ready to quiz yourself?