Predict FPL Points with SageMaker

Train an XGBoost model on FPL data and deploy it to a SageMaker endpoint.

Introduction

30 Second Summary

Millions of Fantasy Premier League managers spend hours each week trying to figure out which players will score big in the next gameweek. What if you could use machine learning to make those predictions for you?

In this project, you will build a machine learning pipeline on Amazon SageMaker that predicts Fantasy Premier League player points from live data. You will go from pulling raw stats off a public API to querying a deployed XGBoost model for a ranked transfer shortlist.

What You'll Build

You will query your live SageMaker prediction endpoint with current-gameweek player stats and see a ranked list of the top 10 players your model predicts will earn the most points next week.

By the end of this project, you'll have:

  • A live prediction endpoint you can send any player's current stats to and get back a predicted points score for the next gameweek.
  • A top-10 transfer shortlist generated by running every active Premier League player through your trained model and ranking them by predicted points.
  • A per-position accuracy breakdown that reveals goalkeeper predictions are significantly worse than forward predictions. You will discover why production FPL predictors use position-specific models.
  • Secret Mission: Configure SageMaker Model Monitor to detect data drift on your endpoint automatically as player form changes each gameweek.

Are there any prerequisites?

You will need an AWS account with free tier available. You should also be comfortable with Python (especially pandas and requests) and familiar with navigating the AWS console.

Before We Start

Before any code or cloud resources, let's lock in what you're about to build and why it matters.

Set Up Your SageMaker Notebook

You have committed to building an FPL predictor. Now you need somewhere to write the code that pulls data, trains the model, and deploys it.

Amazon SageMaker notebook instances are managed Jupyter environments that come pre-loaded with the ML libraries and IAM permissions the rest of the project depends on. In this step, you will create a notebook instance, open JupyterLab, and verify that everything is wired together.

In this step, get ready to:
  • Create a SageMaker notebook instance with a new IAM role.
  • Open JupyterLab and create a new notebook with the conda_python3 kernel.
  • Verify the SageMaker SDK and boto3 are available by printing the SDK version and default S3 bucket.
Create a SageMaker notebook instance

A notebook instance is a managed ML compute environment running on a Jupyter server. SageMaker provisions the EC2 instance, pre-installs Anaconda libraries and the AWS SDKs, and attaches an IAM role so your code can access other AWS services.

  • Sign in to the AWS Management Console at console.aws.amazon.com.
  • In the search bar at the top, type SageMaker and select Amazon SageMaker from the results.
  • In the left navigation pane, choose Notebook, then choose Notebook instances.
  • Click Create notebook instance.

You are now on the Create notebook instance page. Fill in the details below.

  • For Notebook instance name, enter fpl-predictor.
  • For Notebook Instance type, select ml.t3.medium. This is eligible for the SageMaker free tier.
  • Leave Platform Identifier at the default notebook-al2023-v1. This gives you JupyterLab 4 on Amazon Linux 2023.

What is ml.t3.medium?

It is a burstable CPU instance with 2 vCPUs and 4 GiB of RAM. It handles lightweight notebooks comfortably and is the recommended default for CPU-based SageMaker images.

The SageMaker free tier gives you 250 hours per month on ml.t3.medium notebook instances for the first 2 months after your first SageMaker resource is created.

Now scroll down to the Permissions and encryption section to set up the IAM role.

  • For IAM role, select Create a new role from the dropdown.
  • A dialog box appears asking about S3 bucket access. Select Any S3 bucket.
  • Click Create role.

What does this IAM role do?

The role grants your notebook instance permission to call AWS services on your behalf. SageMaker automatically attaches the AmazonSageMakerFullAccess managed policy, which covers S3 buckets with sagemaker in the name.

Selecting Any S3 bucket ensures the role can read from and write to any bucket. You need this because your training data and model artifacts will be stored in the default SageMaker S3 bucket.

  • Leave all other settings at their defaults.
  • Click Create notebook instance at the bottom of the page.

You are taken back to the Notebook instances list. Your new fpl-predictor instance shows a status of Pending.

  • Wait for the status to change to InService. This takes 2 to 4 minutes.

Stuck on Pending for more than 5 minutes?

Check that your AWS region supports ml.t3.medium notebook instances. If it does not, stop the instance and update the instance type to ml.t2.medium.

Also confirm you have not hit your SageMaker service quota for notebook instances in this region.

Ask for help: My SageMaker notebook instance is stuck on Pending. What should I check?

Open JupyterLab and create a new notebook

With the instance running, you can open JupyterLab directly from the SageMaker console. JupyterLab is the IDE where you will write all of the Python code for this project.

  • On the Notebook instances page, find your fpl-predictor instance.
  • Click Open JupyterLab next to it. JupyterLab opens in a new browser tab.

You should see the JupyterLab Launcher with options to create notebooks, open a terminal, and browse files.

  • From the JupyterLab menu bar, click File.
  • Select New, then select Notebook.
  • In the Select Kernel dialog, choose conda_python3 and click Select.

Why conda_python3?

This kernel ships with Python 3.10, pandas, NumPy, scikit-learn, and the SageMaker SDK pre-installed. It saves you from manually installing dependencies.

A new untitled notebook opens. Rename it so you can find it later.

  • Click File, then Save Notebook As....
  • Enter fpl_predictor.ipynb as the file name and click Save.

The tab title now shows fpl_predictor.ipynb. You have a blank notebook ready for code.

Verify the SageMaker SDK and default S3 bucket

Before writing any ML code, confirm that the libraries you need are importable and that SageMaker knows which S3 bucket to use for training data and model artifacts.

  • Click inside the first empty cell in your notebook and paste the following code:
# Import the SageMaker SDK and AWS SDK
import sagemaker
import boto3

# Print the SageMaker SDK version to confirm it loaded
print('SageMaker SDK version:', sagemaker.__version__)
  • Run the cell by pressing Shift+Enter.

You should see a version number printed below the cell, confirming the SDK is installed.

What does this code do?

  • sagemaker is the Python SDK that lets you launch training jobs, deploy models, and manage endpoints from code.
  • boto3 is the general-purpose AWS SDK for Python. SageMaker uses it under the hood to talk to S3, IAM, and other services.
  • Printing the version confirms the library loaded without import errors.
  • Click inside the next empty cell and paste this code:
# Create a SageMaker session and get the default S3 bucket
sess = sagemaker.Session()
bucket = sess.default_bucket()
region = sess.boto_region_name

print('Default S3 bucket:', bucket)
print('Region:', region)
  • Run the cell by pressing Shift+Enter.

What does this code do?

  • sagemaker.Session() creates a session object that manages interactions with SageMaker and AWS services.
  • sess.default_bucket() returns the name of the default S3 bucket SageMaker uses for this account and region. The bucket name follows the pattern sagemaker-{region}-{account-id}. If the bucket does not exist yet, SageMaker creates it for you.
  • You will use this bucket throughout the project to store training data and model artifacts.

Before checking the output, predict: what do you think the bucket name will look like?

You should see two lines printed. The bucket name starts with sagemaker- followed by your region and account ID. The region matches the AWS region you selected in the console.

Seeing an import error or permission error?

  • If you see ModuleNotFoundError, make sure you selected the conda_python3 kernel when creating the notebook. You can change the kernel by clicking the kernel name in the top-right corner of the notebook and selecting conda_python3.
  • If you see an AccessDenied error when calling default_bucket(), the IAM role may not have S3 permissions. Go back to the SageMaker console, stop the instance, update the IAM role to include AmazonS3FullAccess, then restart.
  • Ask for help: I'm getting an error when importing sagemaker or calling default_bucket. How do I fix this?

✔️ Awesome, I've got everything!

Great. Make sure you have saved fpl_predictor.ipynb with both cells executed and their output visible.

ⓧ I'd like to double check the full code

Your notebook should have two cells. Scroll up and confirm these match what you have.

Cell 1:

# Import the SageMaker SDK and AWS SDK
import sagemaker
import boto3

# Print the SageMaker SDK version to confirm it loaded
print('SageMaker SDK version:', sagemaker.__version__)

Cell 2:

# Create a SageMaker session and get the default S3 bucket
sess = sagemaker.Session()
bucket = sess.default_bucket()
region = sess.boto_region_name

print('Default S3 bucket:', bucket)
print('Region:', region)

Your ML environment is ready. You have a running SageMaker notebook with the SDK confirmed working and an S3 bucket identified for storing data. Next up, you will pull live player data from the FPL API and engineer the features your model needs to predict points.

Pull FPL Data and Engineer Features

Your SageMaker notebook is running and the SDK is confirmed working. The model needs training data, and the FPL API gives you raw stats for every Premier League player.

Raw stats alone are not good features. You need to engineer features that capture what actually predicts next-gameweek points: recent form, fixture difficulty, position, and cost. In this step, you will pull live data from the FPL API, engineer those features, and upload the result to S3 as training data for XGBoost.

In this step, get ready to:
  • Pull all player data from the FPL API and build a DataFrame of gameweek-level stats.
  • Engineer predictive features like rolling averages, goals per 90, and fixture difficulty.
  • Split the data into training and validation sets and upload them to S3.
Pull player data from the FPL API

The FPL API provides two key endpoints. The bootstrap-static endpoint returns every player's metadata in one call. The element-summary endpoint returns per-gameweek performance history for each player individually.

  • In your fpl_predictor.ipynb notebook, add a new cell with the following imports and API call to fetch all player data:
import requests
import pandas as pd
import time
import numpy as np

# Fetch all player and team data from the FPL API
fpl_url = "https://fantasy.premierleague.com/api/bootstrap-static/"
response = requests.get(fpl_url)
fpl_data = response.json()

# Extract player list
players = fpl_data["elements"]
print(f"Total players in the API: {len(players)}")

What does this code do?

The bootstrap-static endpoint returns all player data, team data, and gameweek metadata in a single JSON response. No authentication is required.

The elements array contains every player in the game with fields like element_type (position), now_cost (price), and cumulative season stats.

  • Run the cell.

You should see a count of all Premier League players (around 600).

Seeing a connection error?

Make sure your SageMaker notebook instance has internet access. The default VPC configuration allows outbound connections. If you see a timeout, try running the cell again.

Help me troubleshoot this connection error

Now you need per-gameweek history for each player. This requires calling the element-summary endpoint once per player. But first, you need fixture difficulty data from a separate endpoint.

  • Add a new cell to fetch fixture difficulty ratings:
# Fetch fixture difficulty ratings from the fixtures endpoint
fixtures_resp = requests.get("https://fantasy.premierleague.com/api/fixtures/")
all_fixtures = fixtures_resp.json()

# Build a lookup: fixture_id -> difficulty for home and away teams
fixture_difficulty = {}
for f in all_fixtures:
    fid = f["id"]
    fixture_difficulty[fid] = {
        "team_h": f["team_h"],
        "team_a": f["team_a"],
        "team_h_difficulty": f["team_h_difficulty"],
        "team_a_difficulty": f["team_a_difficulty"]
    }

print(f"Loaded difficulty ratings for {len(fixture_difficulty)} fixtures")

Why fetch fixtures separately?

The per-gameweek history entries do not include a fixture difficulty field. You need to join each player's gameweek fixture ID with the fixtures endpoint to get the team_h_difficulty and team_a_difficulty values (rated 1 to 5).

  • Run the cell. You should see difficulty ratings loaded for 380 fixtures.
  • Add a new cell to loop through active players and collect their gameweek history:
# Filter to players who have played minutes this season
active_players = [p for p in players if p["minutes"] > 0]
print(f"Active players to fetch: {len(active_players)}")

# Fetch gameweek history for each active player
all_gw_data = []
for i, player in enumerate(active_players):
    pid = player["id"]
    url = f"https://fantasy.premierleague.com/api/element-summary/{pid}/"
    resp = requests.get(url)
    history = resp.json().get("history", [])

    for gw in history:
        gw["element_type"] = player["element_type"]
        gw["now_cost"] = player["now_cost"]
    all_gw_data.extend(history)

    time.sleep(0.1)  # Respect API rate limit
    if (i + 1) % 50 == 0:
        print(f"Fetched {i+1}/{len(active_players)} players...")

print(f"\nTotal gameweek rows collected: {len(all_gw_data)}")
df = pd.DataFrame(all_gw_data)

What does this code do?

For each player who has played minutes, the code calls element-summary/{player_id}/ and collects the history array. Each entry is one gameweek's stats for that player.

The time.sleep(0.1) call respects the API's rate limit. The element_type and now_cost are attached from bootstrap-static since they are not in the per-gameweek history.

  • Run the cell. This takes 30 to 60 seconds depending on how many players have played. You will see progress updates every 50 players.

You should see a total gameweek row count printed at the end. The exact number depends on how many gameweeks have been played in the current season.

Seeing very few rows?

If the season just started, you may only have one gameweek of data per player. The code works with whatever data is available. More gameweeks will produce a richer training set.

Help me understand if my data is enough

Engineer features and set the target variable

Raw API fields need transformation to become useful predictors. Features like rolling averages capture recent form, and fixture difficulty captures opponent strength.

  • Add a new cell to compute the fixture difficulty, home/away flag, and ICT index features:
# Add fixture difficulty rating for each gameweek row
def get_difficulty(row):
    fid = row["fixture"]
    if fid in fixture_difficulty:
        fix = fixture_difficulty[fid]
        if row["was_home"]:
            return fix["team_h_difficulty"]
        else:
            return fix["team_a_difficulty"]
    return 3  # Default to medium difficulty

df["fixture_difficulty_rating"] = df.apply(get_difficulty, axis=1)
df["home_away_flag"] = df["was_home"].astype(int)
df["ict_index"] = df["ict_index"].astype(float)

print(f"Added fixture difficulty, home/away flag, and ICT index.")
print(df[["fixture_difficulty_rating", "home_away_flag", "ict_index"]].head())

What does this code do?

The get_difficulty function joins each gameweek entry with the fixtures data using the fixture ID. It picks team_h_difficulty or team_a_difficulty depending on whether the player was home or away.

The ict_index comes from the API as a string, so it needs to be cast to a float.

  • Run the cell. You should see a preview of the three new columns with numeric values.
  • Add a new cell to compute rolling averages and goals per 90:
# Sort by player and gameweek for rolling calculations
df = df.sort_values(["element", "round"])

# Rolling average points over last 3 gameweeks per player
df["rolling_avg_pts_3gw"] = df.groupby("element")["total_points"].transform(
    lambda x: x.shift(1).rolling(window=3, min_periods=1).mean()
).fillna(0)

# Total minutes played up to this gameweek
df["total_minutes"] = df.groupby("element")["minutes"].cumsum()

# Goals + assists per 90 minutes
df["goals_assists_per90"] = np.where(
    df["total_minutes"] > 0,
    (df.groupby("element")["goals_scored"].cumsum() +
     df.groupby("element")["assists"].cumsum()) / df["total_minutes"] * 90,
    0
)

print(f"Engineered features. DataFrame shape: {df.shape}")
print(df[["total_points", "rolling_avg_pts_3gw", "total_minutes",
          "goals_assists_per90", "clean_sheets", "fixture_difficulty_rating",
          "home_away_flag", "element_type", "now_cost", "ict_index"]].head())

What does this code do?

The rolling_avg_pts_3gw uses a shifted rolling mean so it only looks at previous gameweeks, not the current one. This prevents data leakage.

The goals_assists_per90 normalises goals and assists by minutes played. The clean_sheets field comes directly from the API. Together with position, cost, and ICT index, these form 10 features that capture a player's scoring potential.

  • Run the cell. You should see the DataFrame shape and a preview of the first few rows with all engineered features.
Split the data and upload to S3

SageMaker's built-in XGBoost algorithm requires a specific CSV format. The target variable must be in the first column, and there must be no header row.

  • Add a new cell to select the feature columns and build the final dataset:
# Select target and feature columns
feature_cols = [
    "rolling_avg_pts_3gw", "total_minutes", "goals_assists_per90",
    "clean_sheets", "fixture_difficulty_rating", "home_away_flag",
    "element_type", "now_cost", "ict_index"
]
target_col = "total_points"

# Build the final dataset with target in the first column
model_df = df[[target_col] + feature_cols].dropna()
print(f"Model dataset shape: {model_df.shape}")
print(f"Features: {feature_cols}")

Why is the target column first?

SageMaker's built-in XGBoost expects CSV files with the target variable in the first column and no header row. This format is the most common gotcha for new SageMaker users. Getting it wrong produces confusing training errors.

  • Run the cell. You should see the shape (rows, 10 columns) and the list of 9 feature names.
  • Add a new cell to split the data by gameweek into training and validation sets:
# Split by gameweek: 80% train, 20% validation
max_round = df["round"].max()
split_round = int(max_round * 0.8) or 1
train_df = model_df[df["round"] <= split_round]
val_df = model_df[df["round"] > split_round]

# If only one gameweek, use all data for training
if len(val_df) == 0:
    split_idx = int(len(model_df) * 0.8)
    train_df = model_df.iloc[:split_idx]
    val_df = model_df.iloc[split_idx:]

print(f"Train set: {train_df.shape}, Validation set: {val_df.shape}")

Why split by gameweek?

Splitting by gameweek prevents data leakage. If you split randomly, the model could train on a player's Gameweek 5 data and validate on Gameweek 3, which leaks future information. Splitting by time keeps the validation set as "future" data.

If only one gameweek has been played, the code falls back to an 80/20 row-based split so the model still has validation data to evaluate against.

  • Run the cell. You should see two shapes printed with non-zero row counts for both sets.
  • Add the final cell to save the CSVs and upload them to S3:
import boto3
import sagemaker

# Get the default SageMaker S3 bucket
sess = sagemaker.Session()
bucket = sess.default_bucket()
prefix = "fpl-xgboost/data"

# Save as headerless CSV (SageMaker XGBoost requirement)
train_csv = train_df.to_csv(index=False, header=False)
val_csv = val_df.to_csv(index=False, header=False)

# Upload to S3
s3 = boto3.client("s3")
s3.put_object(Bucket=bucket, Key=f"{prefix}/train.csv", Body=train_csv)
s3.put_object(Bucket=bucket, Key=f"{prefix}/val.csv", Body=val_csv)

print(f"Uploaded train.csv and val.csv to s3://{bucket}/{prefix}/")

What does this code do?

The header=False argument removes column headers from the CSV. The index=False argument stops pandas from writing row numbers as an extra column.

The files are uploaded under the fpl-xgboost/data/ prefix in your default SageMaker S3 bucket.

  • Run the cell. You should see the S3 path printed confirming both files were uploaded.

Before you move on, what do you expect to see when you check the training and validation set shapes? Do you think both sets have rows?

  • Verify the data by running this in a new cell:
print(f"Train: {train_df.shape}, Validation: {val_df.shape}")
print(f"Features: {len(feature_cols)} columns")
print(f"S3 location: s3://{bucket}/{prefix}/")
  • Run the cell. You should see a training set with 10 columns (1 target + 9 features) and hundreds of rows, plus a separate validation set.

Seeing zero rows in your validation set?

If only one gameweek has been played, the gameweek-based split may put everything in training. The fallback code handles this by doing an 80/20 row split instead. Check that both train_df.shape and val_df.shape show non-zero row counts.

Help me fix my empty validation set

✔️ Awesome, I've got everything!

Great. Make sure you have saved your notebook. Your training and validation CSVs are in S3 and ready for the model.

ⓧ I'd like to double check the full code

This step added several cells to your notebook. Scroll back through your cells and confirm you have each of these in order:

  • The imports and bootstrap-static API call.
  • The fixtures endpoint call and difficulty lookup dictionary.
  • The active player loop fetching element-summary histories.
  • The fixture difficulty, home/away flag, and ICT index feature engineering.
  • The rolling averages and goals per 90 calculations.
  • The feature column selection and model DataFrame.
  • The train/validation split by gameweek.
  • The headerless CSV export and S3 upload.
  • The verification cell printing shapes, feature count, and S3 location.

Your training data is sitting in S3, formatted exactly how SageMaker XGBoost expects it. Next up, you will point a SageMaker training job at this data and train your model with a single API call.

Train the XGBoost Model

Step 3: Train the XGBoost Model

You have training data sitting in S3, but data alone does not make predictions. You need a trained model that has learned the patterns in your player stats.

SageMaker's built-in XGBoost algorithm lets you launch a managed training job without writing a single line of training code. You point it at your data, set a few hyperparameters, and SageMaker handles the rest.

In this step, get ready to:
  • Configure the XGBoost built-in algorithm container and set training hyperparameters.
  • Point the training job at your S3 data and launch it.
  • Confirm the training job completes with decreasing validation error.
Retrieve the XGBoost container and configure hyperparameters

SageMaker's built-in algorithms are packaged as Docker containers. To use one, you retrieve its container image URI for your AWS region, then pass it to a ModelTrainer along with the hyperparameters that control how the model learns.

  • In your fpl_predictor.ipynb notebook, create a new cell and retrieve the XGBoost container image by pasting this code:
from sagemaker.train import ModelTrainer
from sagemaker.train.configs import Compute, InputData
from sagemaker.core.helper.session_helper import get_execution_role
from sagemaker.core import image_uris

# Get the current AWS region from our existing session
region = sagemaker.Session().boto_region_name

# Retrieve the XGBoost built-in algorithm container for this region
xgboost_container = image_uris.retrieve("xgboost", region, "1.5-1")
print(f"XGBoost container URI: {xgboost_container}")

What does this code do?

The image_uris.retrieve("xgboost", region, "1.5-1") call looks up the exact Docker image URI for the XGBoost 1.5-1 built-in algorithm in your AWS region. This is the container SageMaker spins up to run your training job.

You never write a training script. The built-in algorithm container already knows how to read CSV data, run XGBoost, and save the trained model.

  • Run the cell. You should see a long container URI printed, something like 683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-xgboost:1.5-1 (the numbers and region will vary).

Seeing an import error?

If you see ModuleNotFoundError for sagemaker.train, your SageMaker SDK may be outdated. Run !pip install -qU sagemaker in a new cell, then restart the kernel from the Kernel menu before re-running.

Help me fix my SageMaker import error

Now configure the ModelTrainer with hyperparameters that tell XGBoost how to learn from your FPL data.

  • Set up the training compute and model trainer by pasting this code into a new cell:
# Set the S3 output path for the trained model artifact
bucket = sagemaker.Session().default_bucket()
prefix = 'fpl-xgboost'

# Configure the training instance (free tier eligible)
compute = Compute(
    instance_type='ml.m5.xlarge',
    instance_count=1,
    volume_size_in_gb=5
)

# Configure the ModelTrainer with XGBoost hyperparameters
# All hyperparameter values must be strings
model_trainer = ModelTrainer(
    training_image=xgboost_container,
    hyperparameters={
        "objective": "reg:squarederror",
        "num_round": "100",
        "max_depth": "5",
        "eta": "0.2",
        "subsample": "0.7"
    },
    role=get_execution_role(),
    compute=compute
)

print("ModelTrainer configured successfully")

What do these hyperparameters control?

Each hyperparameter shapes how the XGBoost model learns from your FPL data:

  • objective reg:squarederror tells XGBoost this is a regression problem (predicting a number, not a category).
  • num_round 100 means 100 boosting rounds. Each round adds a new tree that corrects the errors of the previous trees.
  • max_depth 5 limits how deep each tree can grow. Deeper trees capture more complexity but risk overfitting.
  • eta 0.2 is the learning rate. Lower values make the model learn more slowly but often more accurately.
  • subsample 0.7 trains each tree on 70% of the data, reducing overfitting.

Every hyperparameter value is a string (not an integer or float). This is a SageMaker requirement for built-in algorithms.

  • Run the cell. You should see "ModelTrainer configured successfully" printed.

Seeing a role error?

If get_execution_role() fails, your notebook's IAM role may not have the right permissions. Confirm that the role attached to your fpl-predictor notebook instance has the AmazonSageMakerFullAccess and AmazonS3FullAccess policies attached.

Help me fix my SageMaker IAM role permissions

Point the training job at your S3 data and launch it

The ModelTrainer knows which algorithm to use and how to tune it. Now you need to tell it where your training and validation data live in S3.

  • Define the input data channels and confirm your S3 paths by pasting this code into a new cell:
# Define the S3 paths for the training and validation data
train_input = InputData(
    channel_name="train",
    data_source=f"s3://{bucket}/{prefix}/data/train.csv"
)
validation_input = InputData(
    channel_name="validation",
    data_source=f"s3://{bucket}/{prefix}/data/val.csv"
)

print(f"Train data: s3://{bucket}/{prefix}/data/train.csv")
print(f"Validation data: s3://{bucket}/{prefix}/data/val.csv")

What are data channels?

SageMaker training jobs receive data through named channels. The built-in XGBoost algorithm expects a train channel for learning and optionally a validation channel for measuring accuracy during training.

Each channel points to the S3 path where you uploaded the CSV files in the previous step.

  • Run the cell to confirm your S3 paths print correctly.

Before you launch the training job, think about what you expect to happen. SageMaker will spin up an ml.m5.xlarge instance, download your data from S3, and run 100 rounds of XGBoost. Do you think the validation error will go up or down as more rounds complete?

  • Launch the training job by pasting this code into a new cell and running it:
# Launch the SageMaker training job
# This will display progress logs and wait until training completes
model_trainer.train(input_data_config=[train_input, validation_input])

What happens when you run this?

SageMaker provisions an ml.m5.xlarge instance, pulls the XGBoost container image, downloads your CSVs from S3, runs 100 boosting rounds, then saves the trained model (model.tar.gz) back to S3. The cell streams the training logs in real time while you wait.

This takes about 3 to 5 minutes. The free tier covers 50 hours per month of ml.m5.xlarge training time during the first 2 months after your first SageMaker resource.

Training job stuck or failed?

If the job stays at "Starting" for more than 5 minutes, the instance may be provisioning. This is normal for a first-time job in a region.

If you see an error about the data format, double-check that your CSVs have no header row and the target column (total_points) is in the first column. This is a strict requirement for SageMaker's built-in XGBoost.

Help me debug my SageMaker XGBoost training job

Review the training logs to confirm decreasing RMSE

Once the training job completes, the logs show the validation error for each boosting round. You should see the RMSE (root mean square error) decrease as more trees are added.

  • Scroll through the training output in your notebook. Look for lines that show validation-rmse values. The numbers should generally decrease from the first round to round 100.
  • Confirm the model artifact was saved by pasting this code into a new cell and running it:
# Check the training job completed and print the model location
training_job = model_trainer._latest_training_job
print(f"Training job name: {training_job.training_job_name}")
print(f"Trained model saved to: {training_job.model_artifacts.s3_model_artifacts}")

What is the model artifact?

SageMaker packages the trained model into a file called model.tar.gz and uploads it to your S3 bucket. This compressed file contains everything needed to make predictions. In the next step, you will deploy this artifact to a live endpoint.

  • Run the cell. You should see the training job name and an S3 path printed, confirming the model was saved successfully.

Not seeing the model path?

If model_trainer._latest_training_job returns None, the training job cell may not have finished. Go back to the cell with model_trainer.train() and wait for it to complete before running this cell.

Help me find my trained model artifact in S3

✔️ Awesome, I've got everything!

Great. Make sure you have saved your notebook.

ⓧ I'd like to double check the full code

Here is all the code you should have added in this step. Scroll through your notebook and confirm each cell is present in order:

  • Cell 1: imports, region, and xgboost_container retrieval using image_uris.retrieve("xgboost", region, "1.5-1").
  • Cell 2: Compute configuration and ModelTrainer setup with all five hyperparameters as strings.
  • Cell 3: InputData channels for train and validation.
  • Cell 4: model_trainer.train() call.
  • Cell 5: model_trainer._latest_training_job and model artifacts path print.

Your model is trained and the artifact is safely stored in S3. Next up, you will deploy it to a real-time endpoint and query it with live player stats to see which players your model predicts will score the most points.

Deploy and Query Your Predictor

Your trained XGBoost model is sitting in S3 as a compressed artifact. That artifact is just a file. To actually get predictions from it, you need to deploy it behind a live HTTPS endpoint that accepts player data and returns predicted points.

Deploying to a real-time endpoint is how ML models serve predictions in production. It is also a core Domain 3 skill on the MLA-C01 exam. In this step, you will deploy your model, send it current-gameweek data for every active player, and print a ranked shortlist of the top 10 transfer picks.

In this step, get ready to:
  • Deploy the trained model to a live SageMaker real-time endpoint.
  • Send current-gameweek player features to the endpoint and receive predictions.
  • Print a top 10 player recommendation list sorted by predicted points.
Deploy the model to a real-time endpoint

SageMaker's ModelBuilder class takes your trained model_trainer object and chains the training output directly to a deployable endpoint. You do not need to write any serving code.

  • In a new cell in your notebook, deploy the trained model by running this code:
from sagemaker.serve import ModelBuilder

# Build and deploy the trained model to a real-time endpoint
model_builder = ModelBuilder(
    model=model_trainer,
    role_arn=role,
    instance_type='ml.m5.xlarge'
)

# Build creates a SageMaker Model resource
model = model_builder.build()

# Deploy creates a SageMaker Endpoint resource
endpoint = model_builder.deploy(endpoint_name='fpl-xgboost-endpoint')

What does this code do?

ModelBuilder takes your model_trainer object and automatically locates the trained model artifact in S3. It wraps the artifact in the XGBoost serving container.

build() registers a SageMaker Model resource. deploy() provisions an ml.m5.xlarge instance, loads the model onto it, and exposes a live HTTPS endpoint named fpl-xgboost-endpoint.

  • Wait for the cell to finish running. Deployment takes approximately 5 to 8 minutes. You will see log output as SageMaker provisions the instance and loads the model.
  • Confirm the endpoint is live by running this in a new cell:
# Confirm the endpoint name
print(endpoint.endpoint_name)

You should see fpl-xgboost-endpoint printed.

What about the free tier?

SageMaker free tier includes 125 hours per month of ml.m5.xlarge real-time inference for the first 2 months. This project stays well within that allowance, but remember to delete the endpoint when you are done.

Deployment failing or timing out?

Check that your notebook's IAM role has the AmazonSageMakerFullAccess policy attached. Without it, the endpoint creation call will fail with an access denied error.

If the deployment hangs beyond 10 minutes, check the SageMaker console under Inference > Endpoints for error messages on the endpoint status.

Help me debug my SageMaker endpoint deployment

Prepare current-gameweek features and query the endpoint

Your endpoint is live. Now you need to send it player data. The endpoint expects CSV rows with feature values in the same order as your training data, but without the target column.

  • In a new cell, pull the latest player data from the FPL API and engineer the same features you used for training:
import requests
import pandas as pd
import numpy as np
import time

# Pull current player data from the FPL API
base_url = 'https://fantasy.premierleague.com/api/'
bootstrap = requests.get(base_url + 'bootstrap-static/').json()

# Build a lookup for team fixture difficulty
teams = {t['id']: t for t in bootstrap['teams']}

# Get all player data
players = bootstrap['elements']
player_df = pd.DataFrame(players)

# Filter to active players with minutes > 0
active = player_df[player_df['minutes'] > 0].copy()

Why filter to active players?

Players with zero minutes have no meaningful stats to feed the model. Filtering them out avoids sending empty feature rows that would produce meaningless predictions.

  • Confirm active players were loaded by running this in a new cell:
print(f'Active players: {len(active)}')

You should see a number in the hundreds, representing all Premier League players who have played at least one minute this season.

  • In a new cell, fetch per-player gameweek histories and engineer prediction features:
# Fetch recent history for each active player and build features
prediction_rows = []

for _, player in active.iterrows():
    pid = player['id']
    time.sleep(0.1)  # Respect API rate limit
    detail = requests.get(f'{base_url}element-summary/{pid}/').json()
    history = detail.get('history', [])

    if len(history) < 1:
        continue

    # Rolling average points over last 3 gameweeks
    recent = history[-3:] if len(history) >= 3 else history
    rolling_avg_pts = np.mean([gw['total_points'] for gw in recent])

    # Aggregate stats from history
    total_mins = sum(gw['minutes'] for gw in history)
    total_goals = sum(gw['goals_scored'] for gw in history)
    total_assists = sum(gw['assists'] for gw in history)
    total_cs = sum(gw['clean_sheets'] for gw in history)
    per_90 = (total_goals + total_assists) / (total_mins / 90) if total_mins > 0 else 0

    # Latest gameweek info for fixture difficulty
    latest = history[-1]
    fixture_diff = latest.get('difficulty', 3)
    was_home = int(latest.get('was_home', False))

    prediction_rows.append({
        'player_id': pid,
        'name': player['web_name'],
        'position': player['element_type'],
        'rolling_avg_pts': rolling_avg_pts,
        'total_minutes': total_mins,
        'ga_per_90': per_90,
        'clean_sheets': total_cs,
        'fixture_difficulty': fixture_diff,
        'was_home': was_home,
        'element_type': player['element_type'],
        'now_cost': player['now_cost'],
        'ict_index': float(player['ict_index'])
    })

pred_df = pd.DataFrame(prediction_rows)
print(f'Players with features: {len(pred_df)}')

What are these features?

These are the same features you engineered in Step 2 for training: rolling average points, total minutes, goals and assists per 90, clean sheets, fixture difficulty, home/away flag, position, cost, and ICT index. The feature order and names must match the training data exactly.

The time.sleep(0.1) call between API requests respects the FPL API's rate limit so your requests are not blocked.

API requests timing out?

The FPL API is a public endpoint with no authentication. If requests are failing, check that your notebook instance has internet access. SageMaker notebook instances have outbound internet by default.

If you have hundreds of active players, the loop may take a few minutes. This is normal.

Help me debug my FPL API requests in a SageMaker notebook

  • In a new cell, prepare the feature matrix and send it to the endpoint:
# Select feature columns in training order (no target column)
feature_cols = [
    'rolling_avg_pts', 'total_minutes', 'ga_per_90',
    'clean_sheets', 'fixture_difficulty', 'was_home',
    'element_type', 'now_cost', 'ict_index'
]

# Convert features to CSV string (no header, no index)
feature_matrix = pred_df[feature_cols].values
csv_lines = '\n'.join([','.join(map(str, row)) for row in feature_matrix])

# Send to the endpoint
response = endpoint.invoke(
    body=csv_lines,
    content_type='text/csv'
)

# Parse the response
result = response.body.read().decode('utf-8')
predictions = [float(x) for x in result.strip().split(',')]
print(f'Predictions received: {len(predictions)}')

What does this code do?

The endpoint expects CSV-formatted input with no header row and no target column. Each row is one player's feature values in the same column order as the training data.

endpoint.invoke() sends the CSV string to the XGBoost serving container. The container deserializes it into an XGBoost DMatrix, runs predictions, and returns the results as a comma-separated string of float values.

Getting a model error on invoke?

The most common cause is a feature count mismatch. The number of columns in your CSV must match the number of features the model was trained on. Double check that feature_cols contains the same features in the same order as your training data.

Help me debug my SageMaker XGBoost endpoint invocation error

Parse predictions and print your FPL transfer shortlist

You have raw prediction scores for every active player. Now you need to attach player names and sort them to find the best transfer picks.

  • In a new cell, merge predictions with player names and print the top 10:
# Map position codes to labels
position_map = {1: 'GKP', 2: 'DEF', 3: 'MID', 4: 'FWD'}

# Attach predictions to the player DataFrame
pred_df['predicted_points'] = predictions
pred_df['position_label'] = pred_df['position'].map(position_map)

# Sort by predicted points descending
top_players = pred_df.sort_values('predicted_points', ascending=False).head(10)

# Print the top 10 recommended players
print('\nTop 10 Players by Predicted Next-GW Points:')
print('=' * 50)
for i, (_, row) in enumerate(top_players.iterrows(), 1):
    cost = row['now_cost'] / 10
    print(f"{i}. {row['name']} ({row['position_label']}) "
          f"- Predicted: {row['predicted_points']:.2f} pts "
          f"- Cost: £{cost:.1f}m")

What does this code do?

The position map converts the FPL API's numeric codes (1, 2, 3, 4) to readable labels (GKP, DEF, MID, FWD). The now_cost value is divided by 10 because the API stores cost in 0.1m units.

Sorting by predicted_points descending gives you the players the model thinks will score the most points next gameweek. This is your FPL transfer shortlist.

Before you check the output, take a moment to predict: what position do you think will dominate the top 10? Midfielders, forwards, or a mix?

  • Run the cell and review the printed list.

You should see a ranked list of 10 player names with their position, predicted points score, and cost. The model is making real predictions based on live FPL data.

✔️ I see 10 players with predicted points

Your endpoint is working and returning predictions. Take a look at the player names. Do they match your FPL intuition? The model tends to favour players with high recent form and strong ICT index scores.

ⓧ I see fewer than 10 players or an error

If you see fewer than 10, check that your pred_df has enough rows by running len(pred_df). If the DataFrame is very small, the FPL API calls may have been rate-limited. Increase the sleep interval and re-run the history-fetching cell.

If you see an error about mismatched prediction count, make sure the number of predictions equals the number of rows in pred_df.

Help me debug my FPL prediction results

Your model is live and making predictions from real FPL data. But how good are these predictions, really? In the next step, you will measure the model's accuracy using RMSE and discover a surprising weakness hiding in the position breakdown.

Evaluate and Discover the Position Problem

In the last step, you deployed your XGBoost model to a live endpoint and printed a top-10 player recommendation list. The predictor is running and returning numbers.

But how good are those numbers? A model that predicts every player will score 3 points looks like it works, but it would be useless for picking differential transfers. You need to measure prediction error. More importantly, you need to check whether the model performs equally well across all player positions.

In this step, get ready to:
  • Calculate the overall RMSE on the validation set using the deployed endpoint.
  • Break down RMSE by position to check for prediction bias.
  • Understand why certain positions are harder to predict than others.
Calculate overall RMSE on the validation set

RMSE (Root Mean Squared Error) tells you how far off your predictions are from the actual values, in the same units as the target. An RMSE of 2.0 means your model's predictions are off by about 2 points on average.

  • In your fpl_predictor.ipynb notebook, create a new cell and add the following code to predict on the validation set and calculate RMSE:
import numpy as np
from sklearn.metrics import mean_squared_error

# Get actual target values from the validation DataFrame
actuals = val_df['total_points'].values

# Prepare features in the same column order used for training
feature_cols = [col for col in val_df.columns if col != 'total_points']
val_features = val_df[feature_cols].values

# Send validation features to the endpoint for predictions
result = predictor.predict(val_features)
val_predictions = np.array(
    [float(v) for v in result.decode('utf-8').strip().split()]
)

# Calculate overall RMSE
overall_rmse = mean_squared_error(
    actuals, val_predictions, squared=False
)
print(f'Overall RMSE: {overall_rmse:.2f}')

What does this code do?

This cell takes the validation data you set aside in Step 2 and runs it through the live endpoint to get predictions. It then compares those predictions against the actual total_points each player scored.

  • feature_cols extracts every column except the target, preserving the same column order the model was trained on.
  • predictor.predict() sends the features to your fpl-xgboost-endpoint endpoint. The response comes back as bytes, so you decode and parse each value into a float.
  • squared=False makes mean_squared_error return the root (RMSE) instead of plain MSE. This keeps the error in the same unit as total_points.
  • Run the cell.

You should see a single RMSE value printed. The exact number depends on your data, but a value between 1 and 4 is typical for FPL point predictions.

Seeing a decode or shape error?

  • If you see a decode error, make sure your predictor was deployed with CSVSerializer() in Step 4. The response format depends on the serializer configuration.
  • If the number of predictions does not match the number of validation rows, double-check that val_df still exists in your notebook. If you restarted the kernel, re-run the data preparation cells from Step 2 first.

help me debug my SageMaker XGBoost predictor response parsing

Break down RMSE by player position

A single overall RMSE can hide serious problems. The model might be great at predicting midfielders but terrible at predicting goalkeepers. The only way to find out is to split the error by position.

Before you run the next cell, think about this: do you expect the model to perform equally well for all four positions (GKP, DEF, MID, FWD)? Or do you think some positions will be harder to predict?

  • Create a new cell and add the following code to calculate RMSE per position:
# Map position codes to readable labels
position_map = {1: 'GKP', 2: 'DEF', 3: 'MID', 4: 'FWD'}

# Build an evaluation DataFrame with positions and predictions
val_eval = val_df.copy()
val_eval['position'] = val_eval['element_type'].map(position_map)
val_eval['predicted'] = val_predictions

# Calculate RMSE for each position
position_rmse = val_eval.groupby('position').apply(
    lambda g: mean_squared_error(
        g['total_points'], g['predicted'], squared=False
    )
).reset_index(name='RMSE')
position_rmse = position_rmse.sort_values('RMSE', ascending=False)

print('RMSE by Position:')
print(position_rmse.to_string(index=False))

What does this code do?

  • position_map translates the numeric element_type codes from the FPL API into human-readable labels: 1=GKP, 2=DEF, 3=MID, 4=FWD.
  • The groupby('position').apply() call calculates a separate RMSE for each position group, so you can compare accuracy across player types.
  • Sorting by RMSE descending puts the worst-predicted position at the top of the table.
  • Run the cell.

Look at the table. The GKP (goalkeeper) row has a noticeably higher RMSE than MID or FWD. Your model is significantly worse at predicting goalkeeper points.

This is the position problem. You trained a single model on all positions, and it struggles with goalkeepers because they score points in completely different ways from outfield players.

All RMSE values look the same?

  • Make sure element_type is one of the feature columns in val_df. If it was dropped during feature engineering in Step 2, the position breakdown will not work correctly.
  • Check that val_predictions from the previous cell is still available. If you cleared outputs or restarted the kernel, re-run the RMSE cell above first.

help me understand my per-position RMSE results for FPL predictions

Understand why goalkeepers are harder to predict

This result is not a bug. It reveals a real limitation of using a single model for all positions.

Why is goalkeeper RMSE so much higher?

Goalkeepers score Fantasy Premier League points through saves, penalties saved, and clean sheets. These events have very different statistical distributions from the goals and assists that midfielders and forwards rely on.

A single XGBoost model tries to learn one set of feature weights for all positions. It ends up optimising for the patterns that appear most often in the data, which are outfield player patterns. Goalkeeper scoring is too different to be captured well by the same model.

This is why production FPL predictors use position-specific models. The OpenFPL paper, for example, trains a separate ensemble for each position. That approach lets each model learn the unique scoring patterns for GKP, DEF, MID, and FWD independently.

What does this teach you about ML in production?

This is a key machine learning concept: different subgroups in your data can have very different error profiles. A single aggregate metric like overall RMSE can hide these differences. Always break down model performance by meaningful segments before trusting the results.

On the MLA-C01 exam, this maps to Domain 2 (Model Development) and Domain 4 (Monitoring). Identifying subgroup bias and knowing when to split models is a core competency.

✔️ Awesome, I've got everything!

Great. Confirm you have saved your notebook with both new cells.

ⓧ I'd like to double check the full code

Scroll up in your notebook and confirm these two cells are present, in order:

  • Cell 1: imports numpy and mean_squared_error, extracts actuals and features from val_df, calls predictor.predict(), parses the response, and prints the overall RMSE.
  • Cell 2: maps element_type to position labels (GKP, DEF, MID, FWD), groups by position, calculates per-position RMSE, sorts descending, and prints the table.

Secret mission

Add Model Monitor for Data Drift Detection

Your model predicts FPL points today, but player data changes every gameweek. Injuries, transfers, and form swings mean the data your model sees in production will slowly drift away from the data it trained on. In this Secret Mission, you will configure SageMaker Model Monitor to catch that drift automatically.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project uses Amazon SageMaker free tier resources. The notebook instance, training job, and endpoint are all covered for the first 2 months from your first SageMaker resource creation.

Cost warning

Your SageMaker endpoint runs continuously and consumes hosting hours from your free tier allowance (125 hrs/month on ml.m5.xlarge). If you leave it running 24/7, you will exhaust the free tier in about 5 days.

If you completed the Secret Mission, the Model Monitor schedule also spins up processing job instances on a recurring basis. Delete the monitoring schedule and endpoint when you are not actively using them.

Resources you used:

  • SageMaker notebook instance (fpl-predictor, ml.t3.medium). Free tier: 250 hrs/month for 2 months.
  • SageMaker real-time endpoint (fpl-xgboost-endpoint, ml.m5.xlarge). Free tier: 125 hrs/month for 2 months.
  • SageMaker training job. Free tier: 50 hrs/month for 2 months.
  • SageMaker Model Monitor schedule (fpl-monitor-schedule). Uses processing job instance hours on each run.
  • S3 bucket with training data, validation data, model artifacts, baseline files, and data capture logs.
  • IAM role with AmazonS3FullAccess and AmazonSageMakerFullAccess policies.

Keep everything running

No action needed. Choose this if you are still actively building or experimenting with your FPL predictor. Your endpoint, notebook, and monitoring schedule will continue running and consuming free tier hours.

  • Your endpoint consumes ml.m5.xlarge hosting hours continuously.
  • Your notebook instance consumes ml.t3.medium hours while it has InService status.
  • If you completed the Secret Mission, the Model Monitor schedule launches processing jobs on its hourly cadence.
  • Monitor your free tier usage in the AWS Billing console to avoid surprise charges.

Pause - I'll come back to this later

Shut down running resources to stop the billing clock, but keep your model artifacts and notebook storage so you can re-deploy later.

  • If you completed the Secret Mission, delete the monitoring schedule first. In your notebook, run:
  • Delete the endpoint to stop hosting charges. In your notebook, run:
# Delete the monitoring schedule (Secret Mission only)
my_monitor.delete_monitoring_schedule()

# Delete the endpoint and its configuration
predictor.delete_endpoint()
  • Stop the notebook instance to stop compute charges. In the SageMaker console, click Notebook instances in the left navigation.
  • Select fpl-predictor, choose Actions, then choose Stop.
  • Wait for the Status to change to Stopped. This takes a few minutes.
  • Your model artifact stays in S3. To re-deploy later, start the notebook and create a new endpoint from the saved model.

Delete - I don't want to use this again

Remove all project resources and start fresh. Follow these steps in order.

If you completed the Secret Mission, start by deleting the monitoring schedule. The monitoring schedule must be deleted before the endpoint. In your notebook, run:

# Delete the monitoring schedule (Secret Mission only)
my_monitor.delete_monitoring_schedule()

Next, delete the endpoint and its backing model. Run this in the same notebook:

# Delete the endpoint, endpoint configuration, and model
predictor.delete_endpoint()
predictor.delete_model()
  • Stop and delete the notebook instance. In the SageMaker console, click Notebook instances in the left navigation.
  • Select fpl-predictor, choose Actions, then choose Stop.
  • Wait for the Status to change to Stopped.
  • Choose Actions, then choose Delete. This permanently removes the notebook instance and its attached storage.
  • Delete the S3 data. Open the S3 console, find the SageMaker default bucket, and delete the fpl-xgboost/ prefix (or the entire bucket if you created it only for this project).
  • Delete CloudWatch log groups. Open the CloudWatch console, go to Log groups, and delete any log groups with names starting with /aws/sagemaker/.
  • Delete the IAM role. Open the IAM console, go to Roles, find the SageMaker execution role you created, and delete it.

What about the training job?

Training job records and their logs are retained indefinitely by AWS and cannot be deleted. They do not incur ongoing charges.

Nice Work!

Nice Work!

Nice work! You just built a complete machine learning pipeline on AWS, from raw API data to a live predictor you can query for next-gameweek player recommendations.

You've learned how to:

  • Pulled live Fantasy Premier League player data from a REST API, engineered predictive features, and stored training datasets in Amazon S3.
  • Trained an XGBoost regression model using Amazon SageMaker's built-in algorithm and deployed it to a real-time endpoint that ranks players by predicted points.
  • Evaluated model accuracy using RMSE, discovered position-specific bias in a single model, and connected the full pipeline to all four MLA-C01 exam domains.
  • Completed an optional Secret Mission to push your skills further.

Ready to quiz yourself?