Build an ML FPL Transfer Shortlist

Build a Streamlit app that ranks affordable FPL transfer replacements.

Introduction

30 Second Summary

A Fantasy Premier League transfer can leave you comparing dozens of possible replacements. A strong recent score can hide unreliable minutes or injury risk.

In this project, you will build a local Streamlit app that turns FPL Core Insights Gameweek data into a ranked shortlist of affordable replacements. You will use chronological validation to compare a regression model with a rolling-average baseline.

What You'll Build

Your browser will show the latest populated Gameweek beside a sortable shortlist of affordable same-position replacements.

By the end of this project, you'll have:

  • A budget-aware player search that updates the transfer budget from your outgoing player and bank balance.
  • A ranked top-ten comparison showing predicted points gain, price, value per million, availability risk, and current news.
  • A model scorecard showing baseline and model mean absolute error on the final six historical Gameweeks.
  • Secret Mission: Stress-test one recommendation with bank values of £0.0, £0.5, and £1.0. Then write a transfer rule that weighs predicted gain, value, and news.

What experience do I need?

Basic Python and introductory machine-learning experience are enough. The first setup step checks compatibility and installs the required packages on Windows.

Before We Start

Before any hands-on work begins, choose one real outgoing-player transfer decision for your app to support. This keeps the project focused on a decision that matters to your FPL team.

Set Up the Local App

Your transfer decision is clear. The project now needs a dependable local foundation before any FPL data enters the picture.

The pinned pandas release requires Python 3.11 or newer. The pinned scikit-learn release has the same requirement. This step prepares the files in Visual Studio Code before Streamlit serves the starter page.

In this step, get ready to:
  • Create the project folder with its two starter files.
  • Verify Python 3.11 or newer.
  • Get the starter Streamlit app running with its pinned dependencies.
Create the project files

A dedicated project folder keeps your Python file beside its dependency list. Opening that folder in Visual Studio Code also gives the integrated terminal the correct starting location.

  • Press the Windows key to open Windows search.
  • Type Visual Studio Code into the search field.
  • Press Enter to launch Visual Studio Code.
  • Click File in the top menu.
  • Select Open Folder.
  • Select Desktop in the folder window.
  • Click New Folder.

Keep the project easy to find

The Desktop gives this project a predictable location. Visual Studio Code uses the folder you select as the home for its Explorer sidebar and integrated terminal.

  • Type fpl-transfer-shortlist as the folder name.
  • Press Enter to create the folder.
  • Click Select Folder.
  • Confirm that you trust the folder if Visual Studio Code shows a trust prompt.
  • Click the New File icon in the Explorer sidebar.
  • Enter requirements.txt as the file name.
  • Press Enter to create the file.

Your dependency list

The requirements.txt file records the exact packages that power the app. You will fill it after confirming that Python supports those packages.

  • Click the New File icon in the Explorer sidebar again.
  • Enter app.py as the file name.
  • Press Enter to create the file.
  • Click Terminal in the top menu.
  • Select New Terminal.
  • Click the profile dropdown beside the plus button in the terminal panel.
  • Select Command Prompt.

You should see requirements.txt beside app.py in the Explorer sidebar. The Command Prompt path should end with fpl-transfer-shortlist.

Your workspace is now anchored to one folder. Every command from this point lands beside the app files.

Verify Python compatibility

The dependency versions in this project need Python 3.11 or newer. Checking first prevents the package installation from failing because of an incompatible interpreter.

  • Check the installed Python version by running this command:
python --version

What does this command do?

The command prints the Python version available to this Command Prompt. It exits after displaying the version number.

✔️ I see version 3.11 or newer

Compatibility is settled. Your installed Python can support every pinned package in this project.

ⓧ I see an older version

The installed Python version is too old for the pinned pandas and scikit-learn releases. Install the current 64-bit stable release before continuing.

  • Open the official Python downloads for Windows page in your browser.
  • Click Download Windows installer (64-bit) for the current stable release.
  • Run the downloaded installer.
  • Follow the installer prompts to finish the installation.
  • Close the current Command Prompt in Visual Studio Code.
  • Select Command Prompt from the terminal profile dropdown again.
  • Run the version check command from above again.

ⓧ Command not found

Windows cannot find a Python installation through the current Command Prompt. Install the current 64-bit stable release before continuing.

  • Open the official Python downloads for Windows page in your browser.
  • Click Download Windows installer (64-bit) for the current stable release.
  • Run the downloaded installer.
  • Follow the installer prompts to finish the installation.
  • Close the current Command Prompt in Visual Studio Code.
  • Select Command Prompt from the terminal profile dropdown again.
  • Run the version check command from above again.

Still seeing the wrong Python version?

Close every terminal in Visual Studio Code after the installation finishes. Create a fresh Command Prompt so it loads the updated Windows configuration.

If the result still shows an older version, help me identify which Python installation Windows is using.

Install dependencies and launch the app

Pinned dependencies keep the project on known package versions. The starter file gives you a quick browser check that Python can import Streamlit and serve a local page.

  • Select requirements.txt in the Explorer sidebar.
  • Add the pinned dependencies by pasting this code into requirements.txt:
pandas==3.0.5
scikit-learn==1.9.1
streamlit==1.63.0

What do these dependencies provide?

  • The pandas==3.0.5 pin supplies the table operations used for FPL Gameweek data.
  • The scikit-learn==1.9.1 pin supplies the machine-learning model and evaluation metric used later.
  • The streamlit==1.63.0 pin supplies the local interactive app interface.
  • Save requirements.txt by pressing Ctrl+S.
  • Install the pinned dependencies by running this command:
py -m pip install -r requirements.txt

What does this command do?

The Python launcher starts pip through the verified interpreter. Pip reads requirements.txt before installing the three pinned package versions.

The Command Prompt returns after the dependencies finish installing. That is the dependency hurdle cleared.

Package installation failed?

  • Confirm that the Command Prompt path ends with fpl-transfer-shortlist.
  • Check that every line in requirements.txt matches the pinned dependency list above.
  • Confirm that the earlier version check reports Python 3.11 or newer.

If the installation still fails, help me diagnose my pinned package installation.

  • Select app.py in the Explorer sidebar.
  • Add the starter page by pasting this code into app.py:
import streamlit as st

st.write("# FPL Transfer Shortlist")

What does this code do?

  • The import streamlit as st line makes Streamlit available through the shorter st name.
  • The st.write() call sends the starter heading to the browser page.
  • Save app.py by pressing Ctrl+S.

Before you launch the app, what do you expect the browser to show from these two lines of Python?

  • Launch the local app by running this command:
streamlit run app.py

What does this command do?

The command starts a local Streamlit server from app.py. It opens the app in a new browser tab.

The server keeps this Command Prompt busy while the app runs. Press Ctrl+C in the terminal when you need to stop it.

You should see a browser tab with the heading FPL Transfer Shortlist. This proves the pinned packages and starter Python file work together.

Starter page not showing?

  • Confirm that the running Command Prompt path ends with fpl-transfer-shortlist.
  • Check that app.py contains the two starter lines shown above.
  • Save app.py before launching the app again.

If the browser still does not show the heading, help me troubleshoot my local Streamlit starter app.

You have your first visible win. The local app can now turn Python code into a browser page.

✔️ Awesome, I've got everything!

Your browser shows the starter heading. Keep the Streamlit server running for the next step.

ⓧ I'd like to double check the full code

Compare your two saved files with these complete versions.

pandas==3.0.5
scikit-learn==1.9.1
streamlit==1.63.0

What should match?

Your requirements.txt file should contain these three package pins in the same order. Each package belongs on its own line.

import streamlit as st

st.write("# FPL Transfer Shortlist")

What should match here?

Your app.py file should contain the Streamlit import followed by the starter heading. Keep the blank line between them.

Your local foundation is ready. Next, you will load populated FPL Gameweek snapshots and turn them into a visible rolling-average ranking.

Build the Naive Shortlist

Your starter Streamlit page now proves the local app can run. The next milestone is a ranking built from current FPL Gameweek snapshots.

A three-Gameweek points average creates a fast first ranking. It gives you a visible baseline for comparing players.

Recent points cannot show whether returns came from reliable minutes. They leave underlying involvement outside the ranking. Current availability is missing too.

In this step, get ready to:
  • Connect the app to populated FPL Core Insights Gameweek snapshots.
  • Calculate the latest three-Gameweek points average for each player.
  • Display a temporary table of the highest-ranked current players.
Load populated Gameweek snapshots

The repository stores one player snapshot for each Gameweek. A URL builder lets the app request the same file from each numbered folder.

  • Switch back to app.py in VS Code.
  • Replace import streamlit as st with the data-source foundation below:
import pandas as pd
import streamlit as st

BASE_URL = (
    "https://raw.githubusercontent.com/olbauday/"
    "FPL-Core-Insights/main/data"
)
CURRENT_SEASON = "2026-2027"

ROLLING_COLUMNS = {
    "event_points": "points_avg3",
    "minutes": "minutes_avg3",
    "expected_goal_involvements": "xgi_avg3",
    "bonus": "bonus_avg3",
    "starts": "starts_avg3",
}


def gameweek_url(season, gameweek, filename):
    return (
        f"{BASE_URL}/{season}/By%20Gameweek/"
        f"GW{gameweek}/{filename}"
    )

What does this foundation do?

  • The pandas import gives the app tools for loading and transforming tabular data.
  • BASE_URL points to the repository's raw data directory.
  • CURRENT_SEASON selects the current dataset.
  • ROLLING_COLUMNS pairs each source statistic with its three-Gameweek feature name.
  • gameweek_url() assembles the season folder, Gameweek folder, and requested filename.
  • Save app.py.
  • Refresh the existing browser tab.

The starter heading should load successfully. This confirms the new imports and data-source configuration are valid Python.

Does the starter heading disappear?

Check that both imports remain at the top of app.py. Compare the parentheses and quotation marks with the code above.

Ask for help if the page reports a Python syntax or import problem. Help me check the imports, constants, and gameweek_url function in my app.py file.

Some Gameweek folders are prepared before real data arrives. The loader checks every folder from Gameweek 1 through Gameweek 38 and rejects any snapshot with fewer than 100 player rows.

  • Insert the loader below gameweek_url() in app.py by adding this code:
def load_gameweeks(season):
    frames = []
    for gameweek in range(1, 39):
        try:
            frame = pd.read_csv(
                gameweek_url(
                    season,
                    gameweek,
                    "player_gameweek_stats.csv",
                ),
                low_memory=False,
            )
        except Exception:
            continue

        if len(frame) < 100:
            continue

        frame["gw"] = gameweek
        frames.append(frame)

    if not frames:
        raise RuntimeError(f"No populated Gameweeks found for {season}.")

    return pd.concat(frames, ignore_index=True)

How does the loader work?

  • The loop requests player_gameweek_stats.csv from each Gameweek folder.
  • pd.read_csv() loads each CSV into a table.
  • The exception handler skips folders that do not contain a readable snapshot.
  • The row-count check removes pre-created placeholder files.
  • pd.concat() combines every populated snapshot into one current-season table.
  • Save app.py.
  • Refresh the browser tab.

The starter heading should remain visible. This confirms the loader definition can be parsed successfully.

Does the loader break the page?

Check that load_gameweeks() starts at the left edge of the file. Make sure every nested line keeps the indentation shown above.

Use this prompt if the page reports an indentation or syntax problem. Help me compare my load_gameweeks function with the expected indentation.

The loader is ready to be called. The page can now report which populated Gameweek it detected as the latest one.

  • Replace the starter heading line at the bottom of app.py with this page setup:
st.set_page_config(
    page_title="FPL Transfer Shortlist",
    layout="wide",
)

st.write("# ML FPL Transfer Shortlist")
current = load_gameweeks(CURRENT_SEASON)
latest_gameweek = int(current["gw"].max())
st.write(f"Current data: Gameweek {latest_gameweek}.")

What does this page setup do?

  • st.set_page_config() gives the app a wide layout.
  • load_gameweeks() retrieves every populated current-season snapshot.
  • latest_gameweek stores the highest Gameweek number found in the combined data.
  • st.write() displays that detected Gameweek in the browser.
  • Save app.py.
  • Refresh the browser tab.

You should see the ML FPL Transfer Shortlist heading. A current-data line should show the latest populated Gameweek detected by the loader.

Does the data fail to load?

Confirm that your internet connection can reach the repository. Check that BASE_URL matches the code above exactly.

Use this prompt if the heading loads without the current Gameweek. Help me find why load_gameweeks is not returning populated FPL data.

Calculate three-Gameweek rolling averages

Raw snapshot columns can contain missing values or text-formatted numbers. Normalization turns the selected statistics into numeric values before the app calculates each rolling window.

  • Insert the normalization helper immediately above st.set_page_config( by adding this code:
def normalize_numeric(frame):
    result = frame.copy()
    numeric_columns = list(ROLLING_COLUMNS) + [
        "now_cost",
        "selected_by_percent",
    ]

    for column in numeric_columns:
        result[column] = pd.to_numeric(
            result[column], errors="coerce"
        )

    for column in ROLLING_COLUMNS:
        result[column] = result[column].fillna(0.0)

    result["now_cost"] = result["now_cost"].fillna(0.0)
    result["selected_by_percent"] = result[
        "selected_by_percent"
    ].fillna(0.0)
    return result

Why normalize these columns?

  • result is a copy that protects the original table from direct changes.
  • numeric_columns includes the rolling statistics, price, and ownership percentage.
  • pd.to_numeric() converts usable values into numbers.
  • fillna(0.0) gives missing rolling inputs a consistent numeric fallback.
  • Save app.py.
  • Refresh the browser tab.

The current-data line should remain visible after the rerun. This confirms the normalization helper loads without breaking the app.

Does normalization trigger a problem?

Check that each name in numeric_columns matches the source columns shown above. Make sure return result stays inside normalize_numeric().

Ask for a focused comparison if the app stops loading. Help me debug the normalize_numeric function in my app.py file.

Each player has one row per populated Gameweek. Sorting by player and Gameweek keeps those performances in time order before the three-match average is calculated.

  • Insert the current-player feature builder below normalize_numeric() by adding this code:
def build_current_rows(current):
    current = normalize_numeric(current)
    current = current.sort_values(["id", "gw"]).copy()
    grouped = current.groupby("id", sort=False)

    for source, feature in ROLLING_COLUMNS.items():
        current[feature] = grouped[source].transform(
            lambda values: values.rolling(
                3, min_periods=1
            ).mean()
        )

    return current.groupby("id", sort=False).tail(1).copy()

How are the rolling features built?

  • sort_values() orders every player's rows by Gameweek.
  • groupby() keeps one player's performances separate from every other player.
  • rolling() averages up to the latest three available performances.
  • tail(1) keeps the newest row for each player after the features are calculated.
  • Save app.py.
  • Refresh the browser tab.

The current-data line should load again. This confirms the rolling-feature function is ready for the page to use.

Does the rolling function fail to load?

Check that grouped is created after the data is sorted. Compare the indentation inside the for loop with the code above.

Use this prompt for help with the rolling calculation. Help me debug build_current_rows and its three-Gameweek rolling averages.

Display the naive shortlist

The page can now turn the latest row for every player into a temporary ranking. Sorting by recent average points exposes what this baseline prioritizes.

  • Replace the page section beginning with st.set_page_config( with this completed display section:
st.set_page_config(
    page_title="FPL Transfer Shortlist",
    layout="wide",
)

st.write("# ML FPL Transfer Shortlist")
current = load_gameweeks(CURRENT_SEASON)
candidates = build_current_rows(current)
latest_gameweek = int(current["gw"].max())
st.write(f"Current data: Gameweek {latest_gameweek}.")
st.dataframe(
    candidates.sort_values("points_avg3", ascending=False)[
        ["web_name", "points_avg3"]
    ].head(10),
    hide_index=True,
)

How is the shortlist ranked?

  • candidates contains one latest row for each current player.
  • sort_values() puts the highest points_avg3 value first.
  • head(10) limits the temporary ranking to ten players.
  • st.dataframe() renders the result as an interactive table.
  • Save app.py.
  • Return to the Command Prompt from earlier.
  • Press Ctrl+C to stop the running Streamlit process.

Expect the full data load to take longer than the starter page because it checks all 38 Gameweek folders.

Before you restart the app, do you expect one recent high-scoring performance to keep a player near the top of a three-Gameweek average?

  • Restart the local app from the fpl-transfer-shortlist folder by running this command:
streamlit run app.py

The browser should show the latest populated current-season Gameweek. Below it, you should see ten players ordered by points_avg3.

This is the intended shortfall. The table cannot constrain replacements by transfer budget or position. Availability context is absent from the ranking.

Do you see a heading without the table?

  • Check that candidates = build_current_rows(current) appears before the table code.
  • Check that points_avg3 uses the same spelling in ROLLING_COLUMNS and the display section.
  • Allow the Gameweek requests to finish before refreshing the browser.

Use this prompt if the table remains missing. Help me trace why my Streamlit app loads FPL data but does not display the points_avg3 table.

✔️ Awesome, I've got everything!

Great work. Save app.py and confirm the browser shows the latest Gameweek plus the rolling-average table.

ⓧ I'd like to double check the full code

Compare your complete app.py file with this version:

import pandas as pd
import streamlit as st

BASE_URL = (
    "https://raw.githubusercontent.com/olbauday/"
    "FPL-Core-Insights/main/data"
)
CURRENT_SEASON = "2026-2027"

ROLLING_COLUMNS = {
    "event_points": "points_avg3",
    "minutes": "minutes_avg3",
    "expected_goal_involvements": "xgi_avg3",
    "bonus": "bonus_avg3",
    "starts": "starts_avg3",
}


def gameweek_url(season, gameweek, filename):
    return (
        f"{BASE_URL}/{season}/By%20Gameweek/"
        f"GW{gameweek}/{filename}"
    )


def load_gameweeks(season):
    frames = []
    for gameweek in range(1, 39):
        try:
            frame = pd.read_csv(
                gameweek_url(
                    season,
                    gameweek,
                    "player_gameweek_stats.csv",
                ),
                low_memory=False,
            )
        except Exception:
            continue

        if len(frame) < 100:
            continue

        frame["gw"] = gameweek
        frames.append(frame)

    if not frames:
        raise RuntimeError(f"No populated Gameweeks found for {season}.")

    return pd.concat(frames, ignore_index=True)


def normalize_numeric(frame):
    result = frame.copy()
    numeric_columns = list(ROLLING_COLUMNS) + [
        "now_cost",
        "selected_by_percent",
    ]

    for column in numeric_columns:
        result[column] = pd.to_numeric(
            result[column], errors="coerce"
        )

    for column in ROLLING_COLUMNS:
        result[column] = result[column].fillna(0.0)

    result["now_cost"] = result["now_cost"].fillna(0.0)
    result["selected_by_percent"] = result[
        "selected_by_percent"
    ].fillna(0.0)
    return result


def build_current_rows(current):
    current = normalize_numeric(current)
    current = current.sort_values(["id", "gw"]).copy()
    grouped = current.groupby("id", sort=False)

    for source, feature in ROLLING_COLUMNS.items():
        current[feature] = grouped[source].transform(
            lambda values: values.rolling(
                3, min_periods=1
            ).mean()
        )

    return current.groupby("id", sort=False).tail(1).copy()


st.set_page_config(
    page_title="FPL Transfer Shortlist",
    layout="wide",
)

st.write("# ML FPL Transfer Shortlist")
current = load_gameweeks(CURRENT_SEASON)
candidates = build_current_rows(current)
latest_gameweek = int(current["gw"].max())
st.write(f"Current data: Gameweek {latest_gameweek}.")
st.dataframe(
    candidates.sort_values("points_avg3", ascending=False)[
        ["web_name", "points_avg3"]
    ].head(10),
    hide_index=True,
)

Your first FPL ranking is live. Next, you'll test whether its recent-points logic can support a trustworthy forecast.

Make the Forecast Leak-Resistant

Your Streamlit app already turns current-season FPL Core Insights snapshots into a visible form ranking. That gives you a useful benchmark for the forecasting work ahead.

An unshifted rolling average includes the same Gameweek that it tries to predict. This step creates lagged features from earlier performances. It also uses chronological validation so the final six historical Gameweeks provide a realistic test.

In this step, get ready to:
  • Define the historical season and forecasting features.
  • Shift each three-Gameweek rolling window behind its target.
  • Measure the rolling baseline on a chronological holdout.
Add the historical feature plan

Historical training needs a separate season label. The feature list also fixes which inputs later models can use.

  • In app.py, select the header from import pandas as pd through the closing brace of ROLLING_COLUMNS.
  • Replace the selected header with this code:
import pandas as pd
import streamlit as st
from sklearn.metrics import mean_absolute_error

BASE_URL = (
    "https://raw.githubusercontent.com/olbauday/"
    "FPL-Core-Insights/main/data"
)
HISTORICAL_SEASON = "2025-2026"
CURRENT_SEASON = "2026-2027"

ROLLING_COLUMNS = {
    "event_points": "points_avg3",
    "minutes": "minutes_avg3",
    "expected_goal_involvements": "xgi_avg3",
    "bonus": "bonus_avg3",
    "starts": "starts_avg3",
}

FEATURES = [
    "points_avg3",
    "minutes_avg3",
    "xgi_avg3",
    "bonus_avg3",
    "starts_avg3",
    "now_cost",
    "selected_by_percent",
    "availability",
]

What does this configuration do?

  • The mean_absolute_error import provides the evaluation metric for the baseline.
  • HISTORICAL_SEASON points the training workflow at the completed 2025-2026 snapshots.
  • FEATURES lists the rolling form signals and current player context used by the later model.
  • Save app.py.
  • Return to the browser tab from earlier.
  • Refresh the page.

You should see the current ten-player ranking reload without an import failure. This confirms that the historical configuration works alongside the existing shortlist.

Does the app stop at the new import?

Confirm that the terminal uses the project environment where scikit-learn was installed. Check that the import matches the code above.

Help me diagnose why the metrics import prevents my Streamlit app from loading.

Prepare availability for training

The historical rows contain percentage fields that can arrive as text or missing values. Numeric normalization gives each row a consistent shape before feature engineering begins.

  • In app.py, select the entire normalize_numeric() function.
  • Replace the selected function with this code:
def normalize_numeric(frame):
    result = frame.copy()
    numeric_columns = list(ROLLING_COLUMNS) + [
        "now_cost",
        "selected_by_percent",
        "chance_of_playing_this_round",
        "chance_of_playing_next_round",
    ]

    for column in numeric_columns:
        result[column] = pd.to_numeric(
            result[column], errors="coerce"
        )

    for column in ROLLING_COLUMNS:
        result[column] = result[column].fillna(0.0)

    result["now_cost"] = result["now_cost"].fillna(0.0)
    result["selected_by_percent"] = result[
        "selected_by_percent"
    ].fillna(0.0)
    return result

What does this normalization change?

  • The two availability percentage columns join the existing numeric inputs.
  • pd.to_numeric() converts usable values into numbers. Values that cannot be converted become missing values.
  • The existing form statistics still receive zeroes where their source values are missing.
  • Save app.py.
  • Return to the browser tab.
  • Refresh the page.

You should see the current player table reload with ten rows. This confirms that the expanded normalization still processes the current-season snapshots.

Does normalization fail on a column?

Check that both availability column names sit inside numeric_columns. Confirm that every comma and closing bracket matches the snippet.

Help me compare my normalization function with the required column list.

Availability percentages need a usable fallback. Healthy players with no reported percentage receive full availability. Players marked injured, suspended, or unavailable receive zero when the percentage is missing.

  • Add this function below normalize_numeric() in app.py:
def add_availability(frame, source_column):
    result = frame.copy()
    source = result[source_column]
    result["availability"] = source.fillna(100.0)
    unavailable_without_percentage = (
        source.isna() & result["status"].isin(["i", "s", "u"])
    )
    result.loc[unavailable_without_percentage, "availability"] = 0.0
    return result

How is availability filled?

  • source_column lets the same helper work with historical or next-Gameweek percentages.
  • availability starts with the reported percentage. Missing percentages receive 100.0.
  • unavailable_without_percentage identifies missing percentages attached to unavailable status values. Those rows receive 0.0.
  • Save app.py.
  • Return to the browser tab.
  • Refresh the page.

You should see the ten-player table load again. The successful rerun proves that availability can be derived from the loaded snapshot schema.

Does the availability helper fail?

Check the indentation inside add_availability(). Confirm that the status values use lowercase letters inside the list.

Help me find the problem in my availability helper.

Shift the features and measure the baseline

Each training target is one player's event_points for a specific Gameweek. Shifting before the three-Gameweek rolling calculation ensures that target only sees performances from earlier Gameweeks.

  • Add this function below the existing build_current_rows() function in app.py:
def build_training_rows(history):
    history = normalize_numeric(history)
    history = add_availability(
        history, "chance_of_playing_this_round"
    )
    history = history.sort_values(["id", "gw"]).copy()
    grouped = history.groupby("id", sort=False)

    for source, feature in ROLLING_COLUMNS.items():
        history[feature] = grouped[source].transform(
            lambda values: values.shift(1).rolling(
                3, min_periods=1
            ).mean()
        )

    return history.dropna(subset=["points_avg3"])

How does the shift prevent leakage?

  • groupby() keeps every rolling history attached to one player.
  • shift(1) moves each source statistic back by one Gameweek before the rolling mean is calculated.
  • rolling() averages up to three earlier performances for each feature.
  • dropna() removes the first player rows that have no earlier points history.
  • Save app.py.
  • Return to the browser tab.
  • Refresh the page.

You should see the existing current-season table reload. This confirms that the new historical builder is valid before it becomes the main app workflow.

Does the rolling transformation fail?

Confirm that shift(1) comes before rolling(). Check that the nested lines remain indented inside the loop.

Help me debug the shifted rolling feature calculation.

The final six historical Gameweeks now form the holdout. Mean absolute error measures the average size of the baseline's prediction mistakes in points. Lower values indicate smaller errors.

  • Select the entire build_current_rows() function in app.py.
  • Delete the selected function.
  • Select the page code from st.set_page_config() through the end of app.py.
  • Replace the selected page code with this code:
st.set_page_config(
    page_title="FPL Transfer Shortlist",
    layout="wide",
)

st.write("# ML FPL Transfer Shortlist")
history = load_gameweeks(HISTORICAL_SEASON)
training_rows = build_training_rows(history)
holdout_start = int(training_rows["gw"].max()) - 5
test_rows = training_rows[training_rows["gw"] >= holdout_start]
baseline_mae = mean_absolute_error(
    test_rows["event_points"], test_rows["points_avg3"]
)
st.write(
    f"Rolling-average baseline MAE, GW{holdout_start}-GW38: "
    f"{baseline_mae:.2f} points"
)

How is the holdout evaluated?

  • holdout_start identifies the first of the final six historical Gameweeks.
  • test_rows contains only rows from that later period.
  • baseline_mae compares each later target with its shifted three-Gameweek points average.
  • The page reports the measured result with two decimal places.
  • Save app.py.

Before you restart the app, consider whether the baseline uses every historical Gameweek or only a later holdout.

  • Stop the existing Streamlit process from the terminal panel.
  • Restart the completed app from the fpl-transfer-shortlist folder by running:
streamlit run app.py

What does this check prove?

The command launches the updated app.py workflow. A successful page load proves that the historical snapshots can be transformed into shifted training rows and evaluated.

Your browser should show a rolling-average baseline MAE for the Gameweek range ending at GW38. The value appears as a number of points with two decimal places.

You have closed the leakage gap. The baseline now predicts later historical Gameweeks using only information available before each target.

Does the baseline page fail to load?

Confirm that build_current_rows() has been removed. Check that the page code calls build_training_rows() with history.

If loading stops during data retrieval, confirm that your internet connection can reach the historical snapshot files.

Help me diagnose why the chronological baseline page does not load.

✔️ Awesome, I've got everything!

Great. Double check that app.py is saved. Your browser should show the rolling-average baseline MAE for the final six historical Gameweeks.

ⓧ I'd like to double check the full code

Compare your complete app.py file with this reference:

import pandas as pd
import streamlit as st
from sklearn.metrics import mean_absolute_error

BASE_URL = (
    "https://raw.githubusercontent.com/olbauday/"
    "FPL-Core-Insights/main/data"
)
HISTORICAL_SEASON = "2025-2026"
CURRENT_SEASON = "2026-2027"

ROLLING_COLUMNS = {
    "event_points": "points_avg3",
    "minutes": "minutes_avg3",
    "expected_goal_involvements": "xgi_avg3",
    "bonus": "bonus_avg3",
    "starts": "starts_avg3",
}

FEATURES = [
    "points_avg3",
    "minutes_avg3",
    "xgi_avg3",
    "bonus_avg3",
    "starts_avg3",
    "now_cost",
    "selected_by_percent",
    "availability",
]


def gameweek_url(season, gameweek, filename):
    return (
        f"{BASE_URL}/{season}/By%20Gameweek/"
        f"GW{gameweek}/{filename}"
    )


def load_gameweeks(season):
    frames = []
    for gameweek in range(1, 39):
        try:
            frame = pd.read_csv(
                gameweek_url(
                    season,
                    gameweek,
                    "player_gameweek_stats.csv",
                ),
                low_memory=False,
            )
        except Exception:
            continue

        if len(frame) < 100:
            continue

        frame["gw"] = gameweek
        frames.append(frame)

    if not frames:
        raise RuntimeError(f"No populated Gameweeks found for {season}.")

    return pd.concat(frames, ignore_index=True)


def normalize_numeric(frame):
    result = frame.copy()
    numeric_columns = list(ROLLING_COLUMNS) + [
        "now_cost",
        "selected_by_percent",
        "chance_of_playing_this_round",
        "chance_of_playing_next_round",
    ]

    for column in numeric_columns:
        result[column] = pd.to_numeric(
            result[column], errors="coerce"
        )

    for column in ROLLING_COLUMNS:
        result[column] = result[column].fillna(0.0)

    result["now_cost"] = result["now_cost"].fillna(0.0)
    result["selected_by_percent"] = result[
        "selected_by_percent"
    ].fillna(0.0)
    return result


def add_availability(frame, source_column):
    result = frame.copy()
    source = result[source_column]
    result["availability"] = source.fillna(100.0)
    unavailable_without_percentage = (
        source.isna() & result["status"].isin(["i", "s", "u"])
    )
    result.loc[unavailable_without_percentage, "availability"] = 0.0
    return result


def build_training_rows(history):
    history = normalize_numeric(history)
    history = add_availability(
        history, "chance_of_playing_this_round"
    )
    history = history.sort_values(["id", "gw"]).copy()
    grouped = history.groupby("id", sort=False)

    for source, feature in ROLLING_COLUMNS.items():
        history[feature] = grouped[source].transform(
            lambda values: values.shift(1).rolling(
                3, min_periods=1
            ).mean()
        )

    return history.dropna(subset=["points_avg3"])


st.set_page_config(
    page_title="FPL Transfer Shortlist",
    layout="wide",
)

st.write("# ML FPL Transfer Shortlist")
history = load_gameweeks(HISTORICAL_SEASON)
training_rows = build_training_rows(history)
holdout_start = int(training_rows["gw"].max()) - 5
test_rows = training_rows[training_rows["gw"] >= holdout_start]
baseline_mae = mean_absolute_error(
    test_rows["event_points"], test_rows["points_avg3"]
)
st.write(
    f"Rolling-average baseline MAE, GW{holdout_start}-GW38: "
    f"{baseline_mae:.2f} points"
)

How to use this reference

Compare the imports, constants, helper functions, and page code in order. Save your file after correcting any differences.

Your historical evaluation now has a trustworthy baseline. Next up, you'll train a regression model on the earlier Gameweeks and compare its holdout error with this result.

Train and Test the ML Model

Your Streamlit app now keeps later outcomes out of each training row. Chronological validation gives the rolling average an honest benchmark.

A three-Gameweek average treats recent points as the whole signal. You will use Regression to combine all eight FEATURES inputs through scikit-learn's histogram gradient boosting estimator.

In this step, get ready to:
  • Define an absolute-error histogram gradient boosting model.
  • Compare model MAE with the rolling baseline on the chronological holdout.
  • Create next-Gameweek predictions from every valid historical row.
Define the histogram gradient boosting model

The estimator uses absolute-error loss to focus training on absolute point differences. A reusable new_model() function keeps the holdout test model consistent with the final prediction model.

  • In app.py from earlier, replace the import block at the top with this code:
import pandas as pd
import streamlit as st
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.metrics import mean_absolute_error

What Changed in the Imports?

  • The HistGradientBoostingRegressor import provides the model class used for training.
  • The existing mean_absolute_error import remains available for evaluating both approaches.
  • Save app.py.
  • Return to the running app in your browser.

You should still see the rolling-average baseline MAE. This confirms that the new estimator import loads successfully.

Does the App Stop Loading?

Compare the four import lines with the code above. Confirm that the project still uses the pinned scikit-learn package from requirements.txt.

Ask for help with the import if the page still fails to load: Help me diagnose why HistGradientBoostingRegressor does not import in app.py.

  • Add the model factory below build_training_rows() in app.py by pasting this code:
def new_model():
    return HistGradientBoostingRegressor(
        loss="absolute_error",
        learning_rate=0.05,
        max_iter=150,
        max_leaf_nodes=15,
        min_samples_leaf=20,
        l2_regularization=0.1,
        random_state=42,
    )

Why Histogram Gradient Boosting?

Histogram gradient boosting can learn nonlinear interactions between the eight model inputs. It also supports missing values.

  • The absolute_error loss aligns training with absolute point differences.
  • The learning settings control how quickly the model adapts to the historical rows.
  • The fixed random_state keeps repeated training runs consistent.
  • Save app.py.
  • Return to the browser tab containing the running app.

The baseline MAE should render again. This proves that Python can create the model factory without interrupting the app.

Does the Model Factory Break the App?

Check that new_model() sits outside build_training_rows(). Its def line should align with the other function definitions.

Use this prompt if the indentation still looks uncertain: Help me check the placement and indentation of new_model in app.py.

Test the model on later Gameweeks

The model trains on rows before holdout_start. The final six Gameweeks stay untouched until prediction time.

  • Replace the runtime section that starts with history = load_gameweeks(HISTORICAL_SEASON) and ends with the existing baseline output block by pasting this code:
history = load_gameweeks(HISTORICAL_SEASON)
training_rows = build_training_rows(history)
holdout_start = int(training_rows["gw"].max()) - 5
fit_rows = training_rows[training_rows["gw"] < holdout_start]
test_rows = training_rows[training_rows["gw"] >= holdout_start]

model = new_model()
model.fit(fit_rows[FEATURES], fit_rows["event_points"])
test_predictions = model.predict(test_rows[FEATURES])
model_mae = mean_absolute_error(
    test_rows["event_points"], test_predictions
)
baseline_mae = mean_absolute_error(
    test_rows["event_points"], test_rows["points_avg3"]
)

st.write(
    f"Rolling-average baseline MAE, GW{holdout_start}-GW38: "
    f"{baseline_mae:.2f} points"
)
st.write(
    f"Histogram gradient boosting MAE, "
    f"GW{holdout_start}-GW38: {model_mae:.2f} points"
)

How Does the Holdout Test Work?

  • The fit_rows selection contains only Gameweeks before the holdout.
  • The model learns how the eight FEATURES inputs relate to event_points.
  • The test_predictions values estimate points for the held-out rows.
  • Both MAE calculations use the same test_rows targets for a fair comparison.

Before you save, do you expect the model MAE to be automatically lower than the rolling-average baseline MAE?

  • Save app.py.
  • Return to the running app in your browser.

You should see a rolling-average baseline MAE followed by a histogram gradient boosting MAE for the same final six Gameweeks.

That is an important checkpoint. Your model is now judged on Gameweeks it did not train on.

How Should You Read the MAE Values?

A lower MAE represents smaller average absolute prediction errors. Claim an improvement only when the displayed model MAE is lower than the displayed baseline MAE.

Are the Two MAE Values Missing?

  • Confirm that fit_rows uses Gameweeks below holdout_start.
  • Confirm that test_rows uses Gameweeks from holdout_start onward.
  • Check that both calls to mean_absolute_error use test_rows["event_points"] as the first input.

Ask for help if the holdout calculation still fails: Help me debug the chronological model evaluation in app.py.

Predict the next Gameweek

Holdout evaluation measures past performance. Transfer advice needs one row per current player built from their latest completed Gameweeks.

  • Add build_current_rows() between build_training_rows() and new_model() in app.py by pasting this code:
def build_current_rows(current):
    current = normalize_numeric(current)
    current = current.sort_values(["id", "gw"]).copy()
    grouped = current.groupby("id", sort=False)

    for source, feature in ROLLING_COLUMNS.items():
        current[feature] = grouped[source].transform(
            lambda values: values.rolling(
                3, min_periods=1
            ).mean()
        )

    latest = current.groupby("id", sort=False).tail(1).copy()
    return add_availability(
        latest, "chance_of_playing_next_round"
    )

What Do the Current Rows Represent?

  • The rolling calculation summarizes each player's latest completed performances.
  • The tail(1) selection keeps the newest row for each player.
  • The final availability value comes from the next-round percentage.
  • Save app.py.
  • Return to the browser tab containing the running app.

Both MAE values should render again. This confirms that the current-row builder loads correctly.

Does Current-Row Preparation Fail?

Check that build_current_rows() uses rolling values without shift(1). These rows use completed current-season performances to predict the following Gameweek.

Use this prompt if the function still fails: Help me debug build_current_rows without changing the lagged training rows.

  • Replace the runtime code from the historical data load through the baseline_mae calculation with this expanded code:
history = load_gameweeks(HISTORICAL_SEASON)
current = load_gameweeks(CURRENT_SEASON)
training_rows = build_training_rows(history)
holdout_start = int(training_rows["gw"].max()) - 5
fit_rows = training_rows[training_rows["gw"] < holdout_start]
test_rows = training_rows[training_rows["gw"] >= holdout_start]

model = new_model()
model.fit(fit_rows[FEATURES], fit_rows["event_points"])
test_predictions = model.predict(test_rows[FEATURES])
model_mae = mean_absolute_error(
    test_rows["event_points"], test_predictions
)
baseline_mae = mean_absolute_error(
    test_rows["event_points"], test_rows["points_avg3"]
)

model = new_model()
model.fit(training_rows[FEATURES], training_rows["event_points"])
candidates = build_current_rows(current)
candidate_predictions = model.predict(candidates[FEATURES])
candidates["predicted_points"] = pd.Series(
    candidate_predictions,
    index=candidates.index,
).clip(lower=0.0)
latest_gameweek = int(current["gw"].max())

How Are the Final Predictions Created?

  • The holdout model still produces model_mae from earlier historical rows.
  • A fresh model trains on every valid row after evaluation is complete.
  • The retrained model predicts one score for each latest current-season candidate.
  • The clip(lower=0.0) operation prevents negative predicted points from entering the shortlist.
  • Save app.py.
  • Return to the running app in your browser.

The two MAE values should render after the current-season rows are processed. This confirms that predicted_points now exists for each latest candidate.

Does Candidate Prediction Stop the App?

  • Confirm that candidates comes from build_current_rows(current).
  • Confirm that candidate_predictions uses the same FEATURES list as model training.
  • Check that the prediction series uses candidates.index.

Ask for targeted help if the prediction stage still fails: Help me debug the current candidate prediction pipeline.

  • Add the prediction target message immediately before the two MAE output blocks by pasting this code:
st.write(
    f"Current data: Gameweek {latest_gameweek}. "
    f"Predictions target Gameweek {latest_gameweek + 1}."
)

What Does the Target Message Confirm?

The detected latest_gameweek comes from the populated current-season files. Adding one names the Gameweek that the candidate predictions target.

Before you check the page, which Gameweek should the app name relative to the latest populated Gameweek?

  • Save app.py.
  • Restart the app from the Windows Command Prompt in the fpl-transfer-shortlist folder if the earlier Streamlit process has stopped by running this command:
streamlit run app.py

What Does This Command Do?

The command serves app.py as a local Streamlit app. It reruns the historical evaluation before creating current-player predictions.

  • Return to the app in your browser.

You should see the latest populated current-season Gameweek followed by the next Gameweek as the prediction target. You should also see baseline MAE and model MAE for the same final six historical Gameweeks.

You now have an evaluated model plus current-player predictions. The lower displayed holdout MAE tells you which tested approach produced smaller errors.

Is the Target Gameweek Missing?

  • Confirm that latest_gameweek is assigned after current-season candidates are created.
  • Confirm that the target message appears before both MAE output blocks.
  • Check that the target uses latest_gameweek + 1.

Ask for help if the target or metrics still fail to appear: Help me verify the final model evaluation and target Gameweek output.

✔️ Awesome, I've got everything!

Great. Double check that app.py is saved. Your app should show the prediction target plus both chronological holdout MAE values.

ⓧ I'd like to double check the full code

Compare your complete app.py file with this version:

import pandas as pd
import streamlit as st
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.metrics import mean_absolute_error

BASE_URL = (
    "https://raw.githubusercontent.com/olbauday/"
    "FPL-Core-Insights/main/data"
)
HISTORICAL_SEASON = "2025-2026"
CURRENT_SEASON = "2026-2027"

ROLLING_COLUMNS = {
    "event_points": "points_avg3",
    "minutes": "minutes_avg3",
    "expected_goal_involvements": "xgi_avg3",
    "bonus": "bonus_avg3",
    "starts": "starts_avg3",
}

FEATURES = [
    "points_avg3",
    "minutes_avg3",
    "xgi_avg3",
    "bonus_avg3",
    "starts_avg3",
    "now_cost",
    "selected_by_percent",
    "availability",
]


def gameweek_url(season, gameweek, filename):
    return (
        f"{BASE_URL}/{season}/By%20Gameweek/"
        f"GW{gameweek}/{filename}"
    )


def load_gameweeks(season):
    frames = []
    for gameweek in range(1, 39):
        try:
            frame = pd.read_csv(
                gameweek_url(
                    season,
                    gameweek,
                    "player_gameweek_stats.csv",
                ),
                low_memory=False,
            )
        except Exception:
            continue

        if len(frame) < 100:
            continue

        frame["gw"] = gameweek
        frames.append(frame)

    if not frames:
        raise RuntimeError(f"No populated Gameweeks found for {season}.")

    return pd.concat(frames, ignore_index=True)


def normalize_numeric(frame):
    result = frame.copy()
    numeric_columns = list(ROLLING_COLUMNS) + [
        "now_cost",
        "selected_by_percent",
        "chance_of_playing_this_round",
        "chance_of_playing_next_round",
    ]

    for column in numeric_columns:
        result[column] = pd.to_numeric(
            result[column], errors="coerce"
        )

    for column in ROLLING_COLUMNS:
        result[column] = result[column].fillna(0.0)

    result["now_cost"] = result["now_cost"].fillna(0.0)
    result["selected_by_percent"] = result[
        "selected_by_percent"
    ].fillna(0.0)
    return result


def add_availability(frame, source_column):
    result = frame.copy()
    source = result[source_column]
    result["availability"] = source.fillna(100.0)
    unavailable_without_percentage = (
        source.isna() & result["status"].isin(["i", "s", "u"])
    )
    result.loc[unavailable_without_percentage, "availability"] = 0.0
    return result


def build_training_rows(history):
    history = normalize_numeric(history)
    history = add_availability(
        history, "chance_of_playing_this_round"
    )
    history = history.sort_values(["id", "gw"]).copy()
    grouped = history.groupby("id", sort=False)

    for source, feature in ROLLING_COLUMNS.items():
        history[feature] = grouped[source].transform(
            lambda values: values.shift(1).rolling(
                3, min_periods=1
            ).mean()
        )

    return history.dropna(subset=["points_avg3"])


def build_current_rows(current):
    current = normalize_numeric(current)
    current = current.sort_values(["id", "gw"]).copy()
    grouped = current.groupby("id", sort=False)

    for source, feature in ROLLING_COLUMNS.items():
        current[feature] = grouped[source].transform(
            lambda values: values.rolling(
                3, min_periods=1
            ).mean()
        )

    latest = current.groupby("id", sort=False).tail(1).copy()
    return add_availability(
        latest, "chance_of_playing_next_round"
    )


def new_model():
    return HistGradientBoostingRegressor(
        loss="absolute_error",
        learning_rate=0.05,
        max_iter=150,
        max_leaf_nodes=15,
        min_samples_leaf=20,
        l2_regularization=0.1,
        random_state=42,
    )


st.set_page_config(
    page_title="FPL Transfer Shortlist",
    layout="wide",
)

st.write("# ML FPL Transfer Shortlist")
history = load_gameweeks(HISTORICAL_SEASON)
current = load_gameweeks(CURRENT_SEASON)
training_rows = build_training_rows(history)
holdout_start = int(training_rows["gw"].max()) - 5
fit_rows = training_rows[training_rows["gw"] < holdout_start]
test_rows = training_rows[training_rows["gw"] >= holdout_start]

model = new_model()
model.fit(fit_rows[FEATURES], fit_rows["event_points"])
test_predictions = model.predict(test_rows[FEATURES])
model_mae = mean_absolute_error(
    test_rows["event_points"], test_predictions
)
baseline_mae = mean_absolute_error(
    test_rows["event_points"], test_rows["points_avg3"]
)

model = new_model()
model.fit(training_rows[FEATURES], training_rows["event_points"])
candidates = build_current_rows(current)
candidate_predictions = model.predict(candidates[FEATURES])
candidates["predicted_points"] = pd.Series(
    candidate_predictions,
    index=candidates.index,
).clip(lower=0.0)
latest_gameweek = int(current["gw"].max())

st.write(
    f"Current data: Gameweek {latest_gameweek}. "
    f"Predictions target Gameweek {latest_gameweek + 1}."
)
st.write(
    f"Rolling-average baseline MAE, GW{holdout_start}-GW38: "
    f"{baseline_mae:.2f} points"
)
st.write(
    f"Histogram gradient boosting MAE, "
    f"GW{holdout_start}-GW38: {model_mae:.2f} points"
)

Your evaluated predictions are ready. Next up, you will turn them into a position-aware transfer shortlist that responds to the outgoing player and bank balance.

Turn Predictions Into Transfer Advice

Your Streamlit app now produces next-Gameweek predictions from a chronologically tested model. Those predictions give each current player a forecast grounded in earlier performances.

A raw score still cannot settle an FPL transfer. A useful recommendation must match the outgoing player's position. It must also fit the sale price plus the available bank balance.

In this step, get ready to:
  • Merge current candidates with their position labels.
  • Add outgoing-player controls with a transfer budget.
  • Display affordable replacements with gain, value, availability, risk, and news.
Merge position labels and cache preparation

The current prediction rows do not contain the verified position labels needed for same-position filtering. The latest FPL Core Insights `players.csv` snapshot supplies those labels.

Preparing two seasons of data also takes repeated work. Caching the preparation function lets the app reuse its result for six hours.

  • In the VS Code Explorer, select app.py.
  • Find the blank line between the end of new_model() and st.set_page_config(.
  • Start the cached preparation function by inserting this code in that blank line:
@st.cache_data(
    ttl="6h",
    show_spinner="Loading FPL data and training the model...",
)
def prepare_app_data():
    history = load_gameweeks(HISTORICAL_SEASON)
    current = load_gameweeks(CURRENT_SEASON)
    training_rows = build_training_rows(history)

    holdout_start = int(training_rows["gw"].max()) - 5
    fit_rows = training_rows[training_rows["gw"] < holdout_start]
    test_rows = training_rows[training_rows["gw"] >= holdout_start]

    model = new_model()
    model.fit(fit_rows[FEATURES], fit_rows["event_points"])
    test_predictions = model.predict(test_rows[FEATURES])

    model_mae = mean_absolute_error(
        test_rows["event_points"], test_predictions
    )
    baseline_mae = mean_absolute_error(
        test_rows["event_points"], test_rows["points_avg3"]
    )

What does this code do?

  • The st.cache_data decorator stores the prepared result for six hours. A rerun can reuse that result instead of downloading every snapshot again.
  • The function loads both seasons. It builds the same leak-resistant training rows from earlier.
  • The chronological holdout still provides separate baseline and model MAE values.
  • Continue prepare_app_data() directly below the baseline_mae calculation by adding this code:
    model = new_model()
    model.fit(
        training_rows[FEATURES],
        training_rows["event_points"],
    )

    candidates = build_current_rows(current)
    candidate_predictions = model.predict(candidates[FEATURES])
    candidates["predicted_points"] = pd.Series(
        candidate_predictions,
        index=candidates.index,
    ).clip(lower=0.0)

How are current predictions prepared?

The second model trains on every valid historical row after evaluation is complete. It predicts the next Gameweek for each player's latest current-season row.

Clipping at zero prevents negative point forecasts from entering the transfer shortlist.

  • Save app.py.
  • Return to the browser tab from earlier to confirm the Gameweek and MAE values still load.

The page reloads with the current Gameweek plus both MAE values. This confirms the cached function code parses alongside the existing app flow.

  • Finish prepare_app_data() directly below the predicted_points assignment by adding this code:
    latest_gameweek = int(current["gw"].max())
    players = pd.read_csv(
        gameweek_url(CURRENT_SEASON, latest_gameweek, "players.csv")
    )
    players["player_id"] = pd.to_numeric(
        players["player_id"], errors="coerce"
    )
    players = players[["player_id", "position"]].rename(
        columns={"player_id": "id"}
    )
    candidates = candidates.merge(players, on="id", how="left")

    return (
        candidates,
        latest_gameweek,
        baseline_mae,
        model_mae,
        holdout_start,
    )

How does the position merge work?

  • The latest populated Gameweek chooses the matching players.csv snapshot.
  • The player_id column becomes numeric before being renamed to id.
  • The merge adds position to each candidate while preserving the prediction rows.
  • The returned tuple gives the interface every dataset and metric it needs.
  • Save app.py.
  • Return to the browser to confirm the existing Gameweek and MAE values still render.

Your completed preparation function now returns candidates with position labels. The current page output confirms the new function remains valid before it replaces the inline preparation flow.

  • Find the existing st.set_page_config( line near the bottom of app.py.
  • Select everything from that line through the closing parenthesis of the existing model MAE st.write block.
  • Replace the selected code with this interface setup:
st.set_page_config(
    page_title="FPL Transfer Shortlist",
    layout="wide",
)

st.write("# ML FPL Transfer Shortlist")
st.write(
    "Choose a player to sell and enter your bank balance. "
    "The app ranks affordable replacements in the same position."
)

(
    candidates,
    latest_gameweek,
    baseline_mae,
    model_mae,
    holdout_start,
) = prepare_app_data()

st.write(
    f"Current data: Gameweek {latest_gameweek}. "
    f"Predictions target Gameweek {latest_gameweek + 1}."
)

What changed in the app flow?

The page now calls prepare_app_data() once to receive candidates plus the evaluation details. The introduction explains the transfer decision that the controls are about to support.

  • Add the metric layout directly below the current Gameweek message:
metric_left, metric_right = st.columns(2)
with metric_left:
    st.write(
        f"Rolling-average baseline MAE, GW{holdout_start}-GW38: "
        f"{baseline_mae:.2f} points"
    )
with metric_right:
    st.write(
        f"Histogram gradient boosting MAE, "
        f"GW{holdout_start}-GW38: {model_mae:.2f} points"
    )

Why place the metrics together?

The two columns put the baseline and model MAE side by side. This keeps the comparison visible without claiming the model wins before you inspect the measured values.

  • Save app.py.
  • Return to the browser after Streamlit reloads the file.

You should see the transfer-focused description below the heading. You should also see the baseline and model MAE displayed in two columns.

Add transfer controls and budget logic

A transfer budget combines the outgoing player's sale price with the money in the bank. The outgoing-player list also needs valid position labels so every later comparison has the right context.

  • Add the outgoing-player options and controls directly below the metric columns:
eligible = candidates[
    candidates["position"].notna() & (candidates["status"] != "u")
].copy()
eligible = eligible.sort_values(["position", "web_name"])
eligible["choice"] = (
    eligible["web_name"].astype(str)
    + " | "
    + eligible["position"].astype(str)
    + " | ID "
    + eligible["id"].astype(int).astype(str)
)

control_left, control_right = st.columns(2)
with control_left:
    selected_choice = st.selectbox(
        "Outgoing player",
        eligible["choice"].tolist(),
    )
with control_right:
    bank = st.number_input(
        "Money in the bank (£m)",
        min_value=0.0,
        max_value=20.0,
        value=0.0,
        step=0.1,
        format="%.1f",
    )

How do the controls stay useful?

  • The eligible pool keeps players with a known position. It also removes records with unavailable departed status.
  • Each choice combines the player's name, position, and ID. This distinguishes players who share similar names.
  • The bank input accepts values from £0.0m to £20.0m in £0.1m steps.
  • Calculate the active transfer budget directly below the bank input block by adding this code:
outgoing = eligible.loc[
    eligible["choice"] == selected_choice
].iloc[0]
transfer_budget = float(outgoing["now_cost"] + bank)

What defines the transfer budget?

The selected choice identifies one outgoing-player row. The app adds that player's current price to the bank value to produce transfer_budget.

  • Save app.py.
  • Return to the browser after the app reloads.

You should see a searchable outgoing-player control beside the bank input. Changing the bank value updates the control immediately.

Don't see the transfer controls?

  • Confirm the control code sits below both metric-column blocks.
  • Check that eligible uses the merged candidates data returned by prepare_app_data().

Ask for help with the control setup: Why are my outgoing-player and bank controls missing from my Streamlit app?

Filter and display the shortlist

The shortlist now needs to turn budget and position constraints into a decision table. Each row will show the expected gain while keeping availability risk and player news visible.

  • Build the ranked shortlist directly below the transfer_budget calculation by adding this code:
shortlist = candidates[
    (candidates["position"] == outgoing["position"])
    & (candidates["id"] != outgoing["id"])
    & (candidates["now_cost"] <= transfer_budget)
    & (candidates["status"] != "u")
].copy()

shortlist["predicted_gain"] = (
    shortlist["predicted_points"]
    - float(outgoing["predicted_points"])
)
shortlist["value_per_million"] = (
    shortlist["predicted_points"] / shortlist["now_cost"]
)
shortlist["availability_risk"] = 100.0 - shortlist["availability"]
shortlist = shortlist.sort_values(
    ["predicted_gain", "predicted_points"],
    ascending=False,
).head(10)

st.write(
    f"Selling {outgoing['web_name']} at £{outgoing['now_cost']:.1f}m "
    f"with £{bank:.1f}m in the bank gives a £{transfer_budget:.1f}m budget."
)

How does a player reach the shortlist?

  • A candidate must share the outgoing player's position.
  • A candidate must be a different player with a price inside transfer_budget.
  • Predicted gain compares each candidate's forecast with the outgoing player's forecast.
  • Value per million rewards forecast points relative to price. Availability risk converts availability into a percentage where higher means riskier.
  • Save app.py.
  • Return to the browser after the app reloads.
  • Change the bank value in the Money in the bank (£m) control.

The budget sentence should update with the selected player's price, the bank value, and the resulting transfer budget. This proves the input now drives the shortlist calculation.

  • Add the empty-state check and display-column selection directly below the budget sentence:
if shortlist.empty:
    st.write("No same-position replacements fit this budget.")
else:
    display = shortlist[
        [
            "web_name",
            "position",
            "now_cost",
            "predicted_points",
            "predicted_gain",
            "value_per_million",
            "availability",
            "availability_risk",
            "news",
        ]
    ].copy()

Why include an empty state?

Some outgoing-player and bank combinations may have no affordable replacement. The empty-state message explains that result instead of leaving a blank page.

When candidates exist, display keeps only the columns needed for the transfer decision.

  • Finish the display logic directly below the display column selection by adding this code:
    for column in [
        "now_cost",
        "predicted_points",
        "predicted_gain",
        "value_per_million",
        "availability",
        "availability_risk",
    ]:
        display[column] = display[column].round(2)

    display = display.rename(
        columns={
            "web_name": "Player",
            "position": "Position",
            "now_cost": "Price (£m)",
            "predicted_points": "Predicted points",
            "predicted_gain": "Predicted gain",
            "value_per_million": "Points per £m",
            "availability": "Availability (%)",
            "availability_risk": "Risk (%)",
            "news": "News",
        }
    )
    st.dataframe(display, hide_index=True, width="stretch")

st.write(
    "Model outputs are estimates. Review availability and news before "
    "making an FPL transfer."
)

What does the final table show?

  • Numeric values are rounded to two decimal places so candidates are easier to compare.
  • Readable column labels turn the model data into a sortable transfer table.
  • The final reminder keeps model output in context. Availability and news can still outweigh the highest predicted gain.
  • Save app.py.

Before you restart the app, do you expect increasing the bank value to keep the same shortlist or make more same-position players affordable?

  • Stop the current Streamlit process in the Command Prompt from earlier if it is still active.
  • Restart the completed app from the fpl-transfer-shortlist folder by running this command:
streamlit run app.py

What does this command do?

This starts app.py as a local Streamlit app. The browser receives the completed transfer controls and shortlist table.

The browser should show the latest populated Gameweek plus both MAE values. You should also see the outgoing-player selector, bank input, budget sentence, and transfer table.

  • Select an outgoing player from the Outgoing player control.
  • Change the Money in the bank (£m) value.

The table should update to show only affordable replacements in the same position. Its top ten rows should include predicted gain, value, availability, risk, and news.

Shortlist missing or unchanged?

  • Check that the selected outgoing player has a position in the merged current-season data.
  • Confirm the shortlist filters use outgoing["position"] and transfer_budget exactly.
  • Try a larger bank value if the page reports that no same-position replacements fit the current budget.

Ask for help with the shortlist logic: Why is my Streamlit transfer shortlist empty or failing to update when I change the bank value?

✔️ Awesome, I've got everything!

Great work. Save app.py and confirm the shortlist responds to both the outgoing-player choice and bank value.

ⓧ I'd like to double check the full code

Compare your completed app.py with this reference:

import pandas as pd
import streamlit as st
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.metrics import mean_absolute_error

BASE_URL = (
    "https://raw.githubusercontent.com/olbauday/"
    "FPL-Core-Insights/main/data"
)
HISTORICAL_SEASON = "2025-2026"
CURRENT_SEASON = "2026-2027"

ROLLING_COLUMNS = {
    "event_points": "points_avg3",
    "minutes": "minutes_avg3",
    "expected_goal_involvements": "xgi_avg3",
    "bonus": "bonus_avg3",
    "starts": "starts_avg3",
}

FEATURES = [
    "points_avg3",
    "minutes_avg3",
    "xgi_avg3",
    "bonus_avg3",
    "starts_avg3",
    "now_cost",
    "selected_by_percent",
    "availability",
]


def gameweek_url(season, gameweek, filename):
    return (
        f"{BASE_URL}/{season}/By%20Gameweek/"
        f"GW{gameweek}/{filename}"
    )


def load_gameweeks(season):
    frames = []
    for gameweek in range(1, 39):
        try:
            frame = pd.read_csv(
                gameweek_url(
                    season,
                    gameweek,
                    "player_gameweek_stats.csv",
                ),
                low_memory=False,
            )
        except Exception:
            continue

        if len(frame) < 100:
            continue

        frame["gw"] = gameweek
        frames.append(frame)

    if not frames:
        raise RuntimeError(f"No populated Gameweeks found for {season}.")

    return pd.concat(frames, ignore_index=True)


def normalize_numeric(frame):
    result = frame.copy()
    numeric_columns = list(ROLLING_COLUMNS) + [
        "now_cost",
        "selected_by_percent",
        "chance_of_playing_this_round",
        "chance_of_playing_next_round",
    ]

    for column in numeric_columns:
        result[column] = pd.to_numeric(
            result[column], errors="coerce"
        )

    for column in ROLLING_COLUMNS:
        result[column] = result[column].fillna(0.0)

    result["now_cost"] = result["now_cost"].fillna(0.0)
    result["selected_by_percent"] = result[
        "selected_by_percent"
    ].fillna(0.0)
    return result


def add_availability(frame, source_column):
    result = frame.copy()
    source = result[source_column]
    result["availability"] = source.fillna(100.0)
    unavailable_without_percentage = (
        source.isna() & result["status"].isin(["i", "s", "u"])
    )
    result.loc[unavailable_without_percentage, "availability"] = 0.0
    return result


def build_training_rows(history):
    history = normalize_numeric(history)
    history = add_availability(
        history, "chance_of_playing_this_round"
    )
    history = history.sort_values(["id", "gw"]).copy()
    grouped = history.groupby("id", sort=False)

    for source, feature in ROLLING_COLUMNS.items():
        history[feature] = grouped[source].transform(
            lambda values: values.shift(1).rolling(
                3, min_periods=1
            ).mean()
        )

    return history.dropna(subset=["points_avg3"])


def build_current_rows(current):
    current = normalize_numeric(current)
    current = current.sort_values(["id", "gw"]).copy()
    grouped = current.groupby("id", sort=False)

    for source, feature in ROLLING_COLUMNS.items():
        current[feature] = grouped[source].transform(
            lambda values: values.rolling(
                3, min_periods=1
            ).mean()
        )

    latest = current.groupby("id", sort=False).tail(1).copy()
    return add_availability(
        latest, "chance_of_playing_next_round"
    )


def new_model():
    return HistGradientBoostingRegressor(
        loss="absolute_error",
        learning_rate=0.05,
        max_iter=150,
        max_leaf_nodes=15,
        min_samples_leaf=20,
        l2_regularization=0.1,
        random_state=42,
    )


@st.cache_data(
    ttl="6h",
    show_spinner="Loading FPL data and training the model...",
)
def prepare_app_data():
    history = load_gameweeks(HISTORICAL_SEASON)
    current = load_gameweeks(CURRENT_SEASON)
    training_rows = build_training_rows(history)

    holdout_start = int(training_rows["gw"].max()) - 5
    fit_rows = training_rows[training_rows["gw"] < holdout_start]
    test_rows = training_rows[training_rows["gw"] >= holdout_start]

    model = new_model()
    model.fit(fit_rows[FEATURES], fit_rows["event_points"])
    test_predictions = model.predict(test_rows[FEATURES])

    model_mae = mean_absolute_error(
        test_rows["event_points"], test_predictions
    )
    baseline_mae = mean_absolute_error(
        test_rows["event_points"], test_rows["points_avg3"]
    )

    model = new_model()
    model.fit(
        training_rows[FEATURES],
        training_rows["event_points"],
    )

    candidates = build_current_rows(current)
    candidate_predictions = model.predict(candidates[FEATURES])
    candidates["predicted_points"] = pd.Series(
        candidate_predictions,
        index=candidates.index,
    ).clip(lower=0.0)

    latest_gameweek = int(current["gw"].max())
    players = pd.read_csv(
        gameweek_url(CURRENT_SEASON, latest_gameweek, "players.csv")
    )
    players["player_id"] = pd.to_numeric(
        players["player_id"], errors="coerce"
    )
    players = players[["player_id", "position"]].rename(
        columns={"player_id": "id"}
    )
    candidates = candidates.merge(players, on="id", how="left")

    return (
        candidates,
        latest_gameweek,
        baseline_mae,
        model_mae,
        holdout_start,
    )


st.set_page_config(
    page_title="FPL Transfer Shortlist",
    layout="wide",
)

st.write("# ML FPL Transfer Shortlist")
st.write(
    "Choose a player to sell and enter your bank balance. "
    "The app ranks affordable replacements in the same position."
)

(
    candidates,
    latest_gameweek,
    baseline_mae,
    model_mae,
    holdout_start,
) = prepare_app_data()

st.write(
    f"Current data: Gameweek {latest_gameweek}. "
    f"Predictions target Gameweek {latest_gameweek + 1}."
)

metric_left, metric_right = st.columns(2)
with metric_left:
    st.write(
        f"Rolling-average baseline MAE, GW{holdout_start}-GW38: "
        f"{baseline_mae:.2f} points"
    )
with metric_right:
    st.write(
        f"Histogram gradient boosting MAE, "
        f"GW{holdout_start}-GW38: {model_mae:.2f} points"
    )

eligible = candidates[
    candidates["position"].notna() & (candidates["status"] != "u")
].copy()
eligible = eligible.sort_values(["position", "web_name"])
eligible["choice"] = (
    eligible["web_name"].astype(str)
    + " | "
    + eligible["position"].astype(str)
    + " | ID "
    + eligible["id"].astype(int).astype(str)
)

control_left, control_right = st.columns(2)
with control_left:
    selected_choice = st.selectbox(
        "Outgoing player",
        eligible["choice"].tolist(),
    )
with control_right:
    bank = st.number_input(
        "Money in the bank (£m)",
        min_value=0.0,
        max_value=20.0,
        value=0.0,
        step=0.1,
        format="%.1f",
    )

outgoing = eligible.loc[
    eligible["choice"] == selected_choice
].iloc[0]
transfer_budget = float(outgoing["now_cost"] + bank)

shortlist = candidates[
    (candidates["position"] == outgoing["position"])
    & (candidates["id"] != outgoing["id"])
    & (candidates["now_cost"] <= transfer_budget)
    & (candidates["status"] != "u")
].copy()

shortlist["predicted_gain"] = (
    shortlist["predicted_points"]
    - float(outgoing["predicted_points"])
)
shortlist["value_per_million"] = (
    shortlist["predicted_points"] / shortlist["now_cost"]
)
shortlist["availability_risk"] = 100.0 - shortlist["availability"]
shortlist = shortlist.sort_values(
    ["predicted_gain", "predicted_points"],
    ascending=False,
).head(10)

st.write(
    f"Selling {outgoing['web_name']} at £{outgoing['now_cost']:.1f}m "
    f"with £{bank:.1f}m in the bank gives a £{transfer_budget:.1f}m budget."
)

if shortlist.empty:
    st.write("No same-position replacements fit this budget.")
else:
    display = shortlist[
        [
            "web_name",
            "position",
            "now_cost",
            "predicted_points",
            "predicted_gain",
            "value_per_million",
            "availability",
            "availability_risk",
            "news",
        ]
    ].copy()

    for column in [
        "now_cost",
        "predicted_points",
        "predicted_gain",
        "value_per_million",
        "availability",
        "availability_risk",
    ]:
        display[column] = display[column].round(2)

    display = display.rename(
        columns={
            "web_name": "Player",
            "position": "Position",
            "now_cost": "Price (£m)",
            "predicted_points": "Predicted points",
            "predicted_gain": "Predicted gain",
            "value_per_million": "Points per £m",
            "availability": "Availability (%)",
            "availability_risk": "Risk (%)",
            "news": "News",
        }
    )
    st.dataframe(display, hide_index=True, width="stretch")

st.write(
    "Model outputs are estimates. Review availability and news before "
    "making an FPL transfer."
)

Use this full-file reference to compare imports, indentation, filtering conditions, labels, and code order.

You did it. Your model now supports a real transfer decision with position, budget, value, and availability constraints.

Secret mission

Audit the Recommendation

Test one transfer recommendation with £0.0, £0.5, and £1.0 in the bank. You will find a candidate that survives multiple budgets. Then you will write a rule that checks predicted gain, value, and player news.

Clean Up Your Resources

Clean Up Your Resources

Your app runs entirely on your Windows PC at no ongoing cost. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • The running Streamlit app process.
  • The local fpl-transfer-shortlist folder.

Keep everything running

No action is needed. Choose this if you plan to keep testing players as new Gameweeks arrive.

  • Leave the local Streamlit process running while you compare more transfer scenarios.
  • Keep your fpl-transfer-shortlist folder as a reusable copy of the app.

Pause - I'll come back to this later

Shut down the running app to free memory. Your project files stay in place for your next transfer check.

  • Return to the Visual Studio Code terminal that is running the app.
  • Use the terminal's stop control to end the Streamlit process.
  • Confirm the terminal returns to a command prompt.
  • Keep the fpl-transfer-shortlist folder in its current location.

Delete - I don't want to use this again

Deleting the folder becomes permanent once you remove it from the Recycle Bin. The steps below stop the app before removing its files.

  • Return to the Visual Studio Code terminal that is running the app.
  • Use the terminal's stop control to end the Streamlit process.
  • In the Visual Studio Code Explorer sidebar, right-click the fpl-transfer-shortlist folder.
  • Choose the context-menu option that reveals the folder in File Explorer.
  • Move to the folder's parent location in File Explorer.
  • Delete the fpl-transfer-shortlist folder.
  • Permanently delete only the fpl-transfer-shortlist folder from the Recycle Bin.
  • Confirm the fpl-transfer-shortlist folder no longer appears in its parent location.

Nice Work!

Nice Work!

You did it! Your local Streamlit app now turns a player sale into a ranked shortlist within the available transfer budget.

You've learned how to:

  • Load populated FPL Core Insights snapshots while skipping placeholder Gameweeks. Your app detects the latest populated current-season Gameweek automatically.
  • Create leak-resistant rolling features from earlier player performances. You used chronological validation to compare the baseline with later Gameweek results using mean absolute error.
  • Train a histogram gradient boosting model to predict next-Gameweek points. You converted those predictions into a position-aware transfer shortlist filtered by price plus availability risk.
  • Complete the Secret Mission by auditing one recommendation across bank values of £0.0, £0.5, and £1.0. You created a transfer rule that weighs predicted gain against value plus player news.

Ready to quiz yourself?