Predict S&P 500 Direction with ML

Train a Random Forest classifier to predict weekly S&P 500 market direction.

Introduction

30 Second Summary

Everyone wants to know whether the stock market will go up or down next week. While no one can predict the future with certainty, machine learning models can analyze decades of historical patterns to make informed probability estimates.

In this project, you will build a Python script that downloads over 20 years of S&P 500 price data, engineers predictive features from it, trains a Random Forest classifier with proper time-based methodology, and outputs a probability prediction for the market's direction next week.

What You'll Build

A single Python script (sp500_predictor.py) that pulls historical S&P 500 data from Yahoo Finance, creates technical features capturing momentum and volatility, and trains a model to predict weekly market direction.

By the end of this project, you'll have:

  • A complete sp500_predictor.py script that downloads real market data from Yahoo Finance and engineers six predictive features from raw price data.
  • A trained Random Forest classifier that outputs calibrated probability predictions for the upcoming trading week.
  • Printed evaluation metrics (accuracy, ROC-AUC) proving your model performs better than random guessing.
  • Secret Mission: Add feature importance analysis to discover which market signals your model relies on most.

Are there any prerequisites?

You should be comfortable with basic Python concepts like variables, functions, loops, and lists. No prior machine learning or finance knowledge is needed.

Set Up Your Project Environment

Before you can build a machine learning model, you need the right tools in place. Python alone cannot download stock data or train a classifier.

In this step, you will create a dedicated project folder, set up a virtual environment to keep your libraries isolated, and install the three packages that power the entire prediction script.

In this step, get ready to:
  • Create a project folder and virtual environment.
  • Install yfinance, pandas, and scikit-learn.
  • Verify everything works by running a test script.
Create a project folder and virtual environment

A virtual environment is an isolated container for your Python libraries. It prevents conflicts between different projects on your machine.

  • In VS Code, open a terminal by clicking Terminal in the menu bar, then New Terminal.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder, move into it, and set up a virtual environment by running these commands:

macOS

mkdir sp500-predictor
cd sp500-predictor
python3 -m venv venv
source venv/bin/activate

Windows

mkdir sp500-predictor
cd sp500-predictor
python -m venv venv
venv\Scripts\activate

What do these commands do?

  • mkdir sp500-predictor creates a new folder called sp500-predictor on your Desktop.
  • cd sp500-predictor moves your terminal into that folder.
  • python3 -m venv venv creates a virtual environment named venv inside the folder.
  • source venv/bin/activate (macOS) or venv\Scripts\activate (Windows) turns on the virtual environment so any libraries you install stay isolated here.

You should see (venv) appear at the start of your terminal prompt. This confirms the virtual environment is active.

Don't see (venv) in your prompt?

  • Make sure you ran the activate command for your operating system. macOS uses source venv/bin/activate and Windows uses venv\Scripts\activate.
  • On Windows, if you see a permissions error about running scripts, run this command first and then try activating again:
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

help me activate my Python virtual environment in VS Code

Install the required libraries

You need three libraries for this project: one to download stock data, one to manipulate DataFrames, and one to train the machine learning model.

  • Install all three libraries by running this command:
pip install yfinance==1.4.1 pandas==3.0.3 scikit-learn==1.9.0

What are these libraries?

  • yfinance downloads historical stock and index data from Yahoo Finance.
  • pandas provides DataFrames for organizing and transforming tabular data.
  • scikit-learn is a machine learning library that includes the Random Forest classifier you will train.
  • Wait for the installation to finish. You should see a message containing Successfully installed followed by the package names.

Seeing an error during installation?

  • Confirm (venv) is showing in your terminal prompt. If it is not, go back and activate your virtual environment first.
  • If you see a network timeout error, check your internet connection and try running the install command again.
  • help me fix a pip install error for yfinance, pandas, or scikit-learn
Verify the installation with a test script

The fastest way to confirm everything installed correctly is to import the libraries and print their version numbers.

  • In VS Code, create a new file by clicking File in the menu bar, then New File.
  • Save the file immediately as sp500_predictor.py inside your sp500-predictor folder.
  • Paste the following code into sp500_predictor.py:
import yfinance as yf
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, roc_auc_score, classification_report

print(yf.__version__)
print(pd.__version__)
import sklearn; print(sklearn.__version__)

What does this code do?

  • The first four lines import the exact modules you will use throughout the project.
  • The print statements output each library's version number to confirm they loaded successfully.
  • If any import fails, Python raises an error immediately, telling you exactly which library has a problem.
  • Save sp500_predictor.py by pressing Cmd+S (macOS) or Ctrl+S (Windows).
  • Run the script in your terminal by entering this command:
python3 sp500_predictor.py

You should see three version numbers printed, one per line: 1.4.1, 3.0.3, and 1.9.0.

Seeing a ModuleNotFoundError?

  • This means Python cannot find one of the libraries. Confirm (venv) is still showing in your prompt. If it disappeared, activate your virtual environment again and rerun the script.
  • Make sure your terminal is inside the sp500-predictor folder (the same folder where sp500_predictor.py lives).
  • On Windows, use python sp500_predictor.py instead of python3 if the command is not found.
  • help me fix a ModuleNotFoundError in my Python project

✔️ Awesome, I've got everything!

Great. Double check you have saved your file before moving on.

ⓧ I'd like to double check the full code

Here is the complete sp500_predictor.py at this point:

import yfinance as yf
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, roc_auc_score, classification_report

print(yf.__version__)
print(pd.__version__)
import sklearn; print(sklearn.__version__)

Your environment is ready and all three libraries are loading correctly. Next up, you will use yfinance to download over 20 years of S&P 500 daily price data and start exploring it.

Download and Explore S&P 500 Data

Your environment is set up and ready to go. Now it's time to get the raw material your model needs: historical stock market data.

In this step, you'll pull over 20 years of daily S&P 500 price data from Yahoo Finance using yfinance. Then you'll transform that raw data into something a machine learning model can learn from: a binary target that answers "did the market go up or down this week?"

In this step, get ready to:
  • Download daily S&P 500 price data from 2000 to today.
  • Create a binary target variable that captures weekly market direction.
  • Clean the dataset and verify its structure for model training.
Download S&P 500 data from Yahoo Finance

The S&P 500 index tracks the 500 largest US companies. Its ticker symbol on Yahoo Finance is ^GSPC. You'll use the yf.download() function to pull daily OHLCV data (Open, High, Low, Close, Volume) stretching back to the year 2000.

  • In sp500_predictor.py, find and delete the three test print lines at the bottom of the file:
print(yf.__version__)
print(pd.__version__)
import sklearn; print(sklearn.__version__)
  • Replace those lines with the following download code:
# Step 2: Download S&P 500 data
print("Downloading S&P 500 data...")
data = yf.download("^GSPC", start="2000-01-01", progress=False)
print(f"Downloaded {data.shape[0]} rows of daily data")
print(data.head())

What does this code do?

  • yf.download("^GSPC", start="2000-01-01", progress=False) fetches every trading day's price data for the S&P 500 index from January 2000 until today.
  • progress=False suppresses the download progress bar so your terminal output stays clean.
  • data.head() prints the first 5 rows so you can see the columns: Open, High, Low, Close, and Volume.
  • Save sp500_predictor.py.
  • Run the script by entering this command in your terminal:
python3 sp500_predictor.py
  • You should see "Downloaded 6000+ rows of daily data" followed by a table showing the first few rows with columns for Open, High, Low, Close, and Volume.

Seeing a connection or download error?

  • Make sure you have an active internet connection. The yf.download() function fetches live data from Yahoo Finance's servers.
  • Check that your virtual environment is activated. You should see (venv) at the start of your terminal prompt.
  • Confirm the ticker symbol is exactly "^GSPC" with the caret symbol. A typo here returns an empty DataFrame.

Why can't my script download S&P 500 data with yfinance?

Create the binary target variable

Your model needs a clear question to answer. In binary classification, that question has a yes/no answer: "Will the S&P 500 be higher 5 trading days from now?"

To create this target, you'll calculate the forward return (how much the price changed over the next 5 days) and convert it to a 1 (went up) or 0 (went down).

  • Add the following code below the print(data.head()) line in sp500_predictor.py:
# Create target: did the market rise over the next 5 trading days?
data["Forward_Return"] = data["Close"].shift(-5) / data["Close"] - 1
data["Target"] = (data["Forward_Return"] > 0).astype(int)
data = data.dropna(subset=["Target"])

print(f"\nClass balance: {data['Target'].value_counts(normalize=True).to_dict()}")

What does this code do?

  • data["Close"].shift(-5) looks 5 rows into the future to get the Close price 5 trading days from now.
  • Dividing the future price by today's price and subtracting 1 gives you the percentage return over that period.
  • (data["Forward_Return"] > 0).astype(int) converts the return into a binary label: 1 if positive (market went up), 0 if negative (market went down).
  • data.dropna(subset=["Target"]) removes the last 5 rows where no future price exists yet, since those rows can't have a valid target.

Why drop NaN rows?

The shift(-5) operation looks 5 days into the future. For the last 5 rows in the dataset, there is no future data yet, so those cells are NaN ("not a number"). Dropping them ensures your model only trains on complete, valid examples.

  • Save sp500_predictor.py and run it again:
python3 sp500_predictor.py
  • You should see the download output from before, plus a new line showing the class balance. It should show roughly 55-60% for class 1 (up weeks) and 40-45% for class 0 (down weeks).

Seeing a KeyError or empty output?

  • Make sure the column names use exact capitalization: "Close", "Forward_Return", and "Target".
  • Check that the target code is placed below the download block, not above it. The data variable must exist before you can add columns to it.

Why am I getting a KeyError when creating my target variable?

Verify the final dataset

Your terminal output tells you two important things about the data you'll feed into the model.

  • Confirm the download line shows approximately 6,000+ rows of daily data. This means you have over 20 years of trading history to work with.
  • Check the class balance output. You should see roughly 55-60% for class 1 (up weeks). This tells you the S&P 500 trends upward over time, but not so heavily that your model can cheat by always predicting "up."

Why does class balance matter?

If 99% of weeks were positive, a model could achieve 99% accuracy by always predicting "up" without learning anything useful. Your ~57% base rate means the model has to actually learn patterns to beat the baseline.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

Here's what your complete sp500_predictor.py should look like at this point:

import yfinance as yf
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, roc_auc_score, classification_report

# Step 2: Download S&P 500 data
print("Downloading S&P 500 data...")
data = yf.download("^GSPC", start="2000-01-01", progress=False)
print(f"Downloaded {data.shape[0]} rows of daily data")
print(data.head())

# Create target: did the market rise over the next 5 trading days?
data["Forward_Return"] = data["Close"].shift(-5) / data["Close"] - 1
data["Target"] = (data["Forward_Return"] > 0).astype(int)
data = data.dropna(subset=["Target"])

print(f"\nClass balance: {data['Target'].value_counts(normalize=True).to_dict()}")

You now have a clean dataset with over 6,000 rows of daily S&P 500 data and a binary target column. Next up, you'll engineer predictive features that give your model something to learn from.

Engineer Predictive Features

Your script now downloads S&P 500 data and knows what it's trying to predict. But raw prices alone are poor inputs for a model.

In this step, you'll create features that summarize market behavior into numbers a Random Forest can learn from. Each feature captures a different signal: momentum, trend, or volatility.

In this step, get ready to:
  • Create rolling return features that capture short, medium, and long-term momentum.
  • Create moving average ratio and volatility features.
  • Clean up NaN values and confirm the final feature set.
Create rolling return features

Feature engineering is the process of turning raw data into informative inputs for your model. The first set of features you'll build captures momentum: how much has the market moved over different timeframes?

  • In sp500_predictor.py, add the following code at the bottom of your file:
# Step 3: Engineer features
data["return_5d"] = data["Close"].pct_change(5)
data["return_20d"] = data["Close"].pct_change(20)
data["return_60d"] = data["Close"].pct_change(60)

What does this code do?

The pct_change() function calculates the percentage change between today's Close price and the Close price N days ago.

  • return_5d captures short-term momentum (1 trading week).
  • return_20d captures medium-term momentum (1 trading month).
  • return_60d captures long-term momentum (1 trading quarter).

If the market has been consistently rising over 60 days, that's a very different signal than if it only rose yesterday. These three features give the model multiple "time horizons" to learn from.

You won't see a change yet. The next chunk adds the remaining features, and then you'll run the script to verify everything together.

Create moving average and volatility features

Momentum tells you how much the price moved. But the model also needs to know how the current price relates to its recent trend, and how turbulent the market has been.

  • Add these three feature calculations directly below the momentum features in sp500_predictor.py:
data["sma_20_ratio"] = data["Close"] / data["Close"].rolling(20).mean()
data["sma_50_ratio"] = data["Close"] / data["Close"].rolling(50).mean()
data["volatility_20d"] = data["Close"].pct_change().rolling(20).std()

What does this code do?

  • sma_20_ratio and sma_50_ratio compare today's price to its 20-day and 50-day moving averages. A ratio above 1.0 means the price is above its average (bullish). Below 1.0 means it's below average (bearish).
  • volatility_20d measures how wild daily price swings have been over the past 20 days. High volatility often signals uncertainty or fear in the market.

You won't see a change yet. The next chunk cleans up NaN values and prints the results.

Drop NaN values and verify the feature set

Rolling calculations need historical data to compute. The 60-day return, for example, can't be calculated for the first 60 rows because there aren't 60 prior days to look back on. These rows end up with NaN values that need to be removed.

  • Add the cleanup and verification code at the bottom of sp500_predictor.py:
# Drop NaN rows from rolling calculations
feature_columns = ["return_5d", "return_20d", "return_60d", "sma_20_ratio", "sma_50_ratio", "volatility_20d"]
data = data.dropna(subset=feature_columns)

print(f"\nFinal dataset: {data.shape[0]} rows, features: {feature_columns}")
print(data[feature_columns].head())

What does this code do?

  • feature_columns defines the list of all 6 features in one place. You'll reuse this list throughout the rest of the script.
  • dropna(subset=feature_columns) removes only the rows where any feature column is NaN. This keeps as much data as possible.
  • The print statements confirm how many rows survived and show the first few rows of feature values so you can verify there are no NaN entries left.
  • Save sp500_predictor.py.
  • Run the script by entering this command in your terminal:
python3 sp500_predictor.py
  • You should see output showing roughly 6,000+ rows in the final dataset, a list of 6 feature column names, and a table of the first few feature values with no NaN entries.

Seeing NaN values in the output?

  • Make sure the data = data.dropna(subset=feature_columns) line comes AFTER all six feature columns have been created. If it runs before a feature is added, that column won't be cleaned.
  • Check that feature_columns lists all six names exactly as spelled in the code above. A typo in a column name will cause a KeyError.

Why are my features showing NaN values after dropna?

Why these specific features?

Each feature captures a different type of market signal. Returns measure momentum ("is the market trending?"). SMA ratios measure position relative to trend ("is the price above or below its average?"). Volatility measures fear ("how uncertain is the market right now?").

Together, they give the model a multi-dimensional view of market conditions at each point in time.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

Here's the complete sp500_predictor.py so far:

import yfinance as yf
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, roc_auc_score, classification_report

# Step 2: Download S&P 500 data
print("Downloading S&P 500 data...")
data = yf.download("^GSPC", start="2000-01-01", progress=False)
print(f"Downloaded {data.shape[0]} rows of daily data")
print(data.head())

# Create target: did the market rise over the next 5 trading days?
data["Forward_Return"] = data["Close"].shift(-5) / data["Close"] - 1
data["Target"] = (data["Forward_Return"] > 0).astype(int)
data = data.dropna(subset=["Target"])

print(f"\nClass balance: {data['Target'].value_counts(normalize=True).to_dict()}")

# Step 3: Engineer features
data["return_5d"] = data["Close"].pct_change(5)
data["return_20d"] = data["Close"].pct_change(20)
data["return_60d"] = data["Close"].pct_change(60)
data["sma_20_ratio"] = data["Close"] / data["Close"].rolling(20).mean()
data["sma_50_ratio"] = data["Close"] / data["Close"].rolling(50).mean()
data["volatility_20d"] = data["Close"].pct_change().rolling(20).std()

# Drop NaN rows from rolling calculations
feature_columns = ["return_5d", "return_20d", "return_60d", "sma_20_ratio", "sma_50_ratio", "volatility_20d"]
data = data.dropna(subset=feature_columns)

print(f"\nFinal dataset: {data.shape[0]} rows, features: {feature_columns}")
print(data[feature_columns].head())

Your data is now packed with meaningful features. Next up, you'll train a Random Forest model on this data using a time-based split that prevents future information from leaking into training.

Train the Model with a Time-Based Split

Your features are engineered and your dataset is clean. Now it's time to train a model that can actually predict whether the market goes up or down next week.

There's a critical trap with financial data that catches most beginners. If you randomly shuffle your data before splitting it into training and testing sets, the model "peeks" at future prices during training. This is called data leakage, and it produces results that look amazing in testing but fail completely in real life.

In this step, get ready to:
  • Split your data chronologically so the model only trains on past data.
  • Train a Random Forest classifier on the training set.
  • Evaluate the model's predictions on held-out test data.
Split the data chronologically into training and test sets

Unlike typical ML problems where you randomly shuffle data, time-series data must be split in order. The model trains on everything before a cutoff date, then gets tested on everything after it.

  • In sp500_predictor.py, add the following code at the bottom of your file:
# Step 4: Time-based train/test split
split_date = data.index[-504]  # Approximately 2 years of trading days
train = data[data.index < split_date]
test = data[data.index >= split_date]

print(f"\nTraining set: {train.shape[0]} rows ({train.index[0].date()} to {train.index[-1].date()})")
print(f"Test set: {test.shape[0]} rows ({test.index[0].date()} to {test.index[-1].date()})")

X_train = train[feature_columns]
y_train = train["Target"]
X_test = test[feature_columns]
y_test = test["Target"]

What does this code do?

The split point is data.index[-504], which grabs the date 504 rows from the end. Since there are roughly 252 trading days per year, this gives you approximately 2 years of test data.

Everything before that date becomes training data. Everything from that date onward becomes test data. The model never sees the test data during training, just like in real life you can't see next week's prices today.

Why not use a random train/test split?

Stock prices are autocorrelated. Today's price is heavily influenced by yesterday's price. If you randomly shuffle the data, the model gets training samples from 2025 and test samples from 2020. It has already "seen" the future when it makes its predictions.

This produces unrealistically high accuracy that completely falls apart on new data. A time-based split ensures your evaluation reflects real-world performance.

  • Save sp500_predictor.py.
  • Run the script with:
python3 sp500_predictor.py
  • You should see the training and test set sizes printed with date ranges. The training set should have roughly 4,000-5,000 rows and the test set around 500 rows.

Seeing a KeyError or empty DataFrame?

Make sure you placed this code after the data = data.dropna(subset=feature_columns) line. The split relies on the cleaned dataset with no NaN values.

Also confirm that feature_columns is defined earlier in your script as a list of all 6 feature names.

My time-based split isn't working correctly.

Train a RandomForestClassifier on the training data

A Random Forest builds hundreds of decision trees, each looking at a random subset of your features. It then combines their votes to make a final prediction. This "wisdom of the crowd" approach is robust against overfitting on noisy financial data.

  • Add the following code below the y_test line you just wrote:
# Train Random Forest
model = RandomForestClassifier(
    n_estimators=200,
    min_samples_split=50,
    random_state=42,
    class_weight="balanced"
)
model.fit(X_train, y_train)

What do these parameters do?

  • n_estimators=200 builds 200 decision trees. More trees generally means better predictions at the cost of training time.
  • min_samples_split=50 prevents any tree from making a decision based on fewer than 50 data points. This reduces overfitting on noisy financial data.
  • random_state=42 makes results reproducible. You'll get the same model each time you run the script.
  • class_weight="balanced" adjusts for the fact that ~57% of weeks are "up" weeks. Without this, the model might just predict "up" every time and get 57% accuracy for free.
  • Save sp500_predictor.py and run the script again:
python3 sp500_predictor.py
  • The script should complete without errors. Training 200 trees may take a few seconds. You won't see new output from this chunk yet, but the model is now trained and ready for evaluation.

Seeing a ValueError about input shapes?

Check that X_train uses feature_columns (the list of 6 feature names) and y_train uses "Target". A mismatch between feature columns and target causes shape errors.

My Random Forest model won't train on my S&P 500 data.

Evaluate the model on the test set

A trained model is useless until you measure how well it performs on data it has never seen. The predict_proba method returns the model's confidence that the market will rise, which is far more useful than a simple yes/no prediction.

  • Add the evaluation code below the model.fit(X_train, y_train) line:
# Evaluate
y_pred = model.predict(X_test)
y_prob = model.predict_proba(X_test)[:, 1]

accuracy = accuracy_score(y_test, y_pred)
roc_auc = roc_auc_score(y_test, y_prob)

print(f"\n--- Model Evaluation ---")
print(f"Accuracy: {accuracy:.3f}")
print(f"ROC-AUC: {roc_auc:.3f}")
print(f"\n{classification_report(y_test, y_pred, target_names=['Down', 'Up'])}")

What do these metrics tell you?

  • predict_proba(X_test)[:, 1] grabs the probability of class 1 ("up" week) for every test sample. This gives you a confidence score between 0 and 1.
  • Accuracy is the percentage of weeks the model correctly predicted as up or down.
  • ROC-AUC measures how well the model ranks weeks by likelihood of going up. A score of 0.50 means random guessing. Anything above 0.55 is meaningful for financial prediction.
  • The classification report shows precision, recall, and F1 for both "Down" and "Up" classes separately.
  • Save sp500_predictor.py and run the script:
python3 sp500_predictor.py
  • You should see the model evaluation output with accuracy and ROC-AUC scores. Your ROC-AUC should be between 0.55 and 0.72, confirming the model performs better than random guessing.

ROC-AUC below 0.50?

A score below 0.50 usually means the target labels got flipped somewhere. Double-check that your Target column uses 1 for "up" weeks and 0 for "down" weeks.

Also verify you're passing y_prob (probabilities) to roc_auc_score, not y_pred (binary predictions).

My model's ROC-AUC score seems wrong.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

Here's the complete sp500_predictor.py up to this point:

import yfinance as yf
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, roc_auc_score, classification_report

# Step 2: Download S&P 500 data
print("Downloading S&P 500 data...")
data = yf.download("^GSPC", start="2000-01-01", progress=False)
print(f"Downloaded {data.shape[0]} rows of daily data")
print(data.head())

# Create target: did the market rise over the next 5 trading days?
data["Forward_Return"] = data["Close"].shift(-5) / data["Close"] - 1
data["Target"] = (data["Forward_Return"] > 0).astype(int)
data = data.dropna(subset=["Target"])

print(f"\nClass balance: {data['Target'].value_counts(normalize=True).to_dict()}")

# Step 3: Engineer features
data["return_5d"] = data["Close"].pct_change(5)
data["return_20d"] = data["Close"].pct_change(20)
data["return_60d"] = data["Close"].pct_change(60)
data["sma_20_ratio"] = data["Close"] / data["Close"].rolling(20).mean()
data["sma_50_ratio"] = data["Close"] / data["Close"].rolling(50).mean()
data["volatility_20d"] = data["Close"].pct_change().rolling(20).std()

# Drop NaN rows from rolling calculations
feature_columns = ["return_5d", "return_20d", "return_60d", "sma_20_ratio", "sma_50_ratio", "volatility_20d"]
data = data.dropna(subset=feature_columns)

print(f"\nFinal dataset: {data.shape[0]} rows, features: {feature_columns}")
print(data[feature_columns].head())

# Step 4: Time-based train/test split
split_date = data.index[-504]  # Approximately 2 years of trading days
train = data[data.index < split_date]
test = data[data.index >= split_date]

print(f"\nTraining set: {train.shape[0]} rows ({train.index[0].date()} to {train.index[-1].date()})")
print(f"Test set: {test.shape[0]} rows ({test.index[0].date()} to {test.index[-1].date()})")

X_train = train[feature_columns]
y_train = train["Target"]
X_test = test[feature_columns]
y_test = test["Target"]

# Train Random Forest
model = RandomForestClassifier(
    n_estimators=200,
    min_samples_split=50,
    random_state=42,
    class_weight="balanced"
)
model.fit(X_train, y_train)

# Evaluate
y_pred = model.predict(X_test)
y_prob = model.predict_proba(X_test)[:, 1]

accuracy = accuracy_score(y_test, y_pred)
roc_auc = roc_auc_score(y_test, y_prob)

print(f"\n--- Model Evaluation ---")
print(f"Accuracy: {accuracy:.3f}")
print(f"ROC-AUC: {roc_auc:.3f}")
print(f"\n{classification_report(y_test, y_pred, target_names=['Down', 'Up'])}")

Your model is trained and evaluated. Next up, you'll use it to generate a real prediction for the upcoming week's market direction.

Generate a Prediction for Next Week

Your Random Forest is trained and evaluated. You know it performs better than random guessing at ranking which weeks are more likely to be positive.

Now it's time for the payoff. You'll feed the most recent market data into your model and get an actual probability prediction for what the S&P 500 will do over the next 5 trading days.

In this step, get ready to:
  • Extract the latest row of features and run it through the trained model.
  • Print the prediction alongside historical context so you can interpret the result.
Get the model's prediction for next week

The model's predict_proba() method returns a confidence score between 0 and 1 for each class. This is more useful than a binary yes/no answer because it tells you how confident the model is.

  • In sp500_predictor.py, add the following code at the bottom of the file (below the classification_report print statement):
# Step 5: Predict next week
latest_features = data[feature_columns].iloc[[-1]]
prediction_prob = model.predict_proba(latest_features)[0][1]
base_rate = data["Target"].mean()

What does this code do?

  • data[feature_columns].iloc[[-1]] grabs the most recent row of features as a DataFrame. The double brackets keep it as a DataFrame (not a Series), which is what scikit-learn expects as input.
  • predict_proba(latest_features)[0][1] gets the probability of class 1 (the market goes up). Index [0] selects the first (and only) row, and [1] selects the "up" class probability.
  • base_rate calculates the historical percentage of weeks that were positive. This gives you a baseline to compare your model's prediction against.
  • Save the file. You won't see output yet because these lines only calculate values without printing them.
Print the prediction with historical context

A raw probability by itself is hard to interpret. Printing it alongside the historical base rate lets you see whether the model is more bullish or more bearish than average.

  • Add these print statements directly below the base_rate line you just added:
print(f"\n--- Next Week Prediction ---")
print(f"Prediction: {prediction_prob:.0%} probability the S&P 500 rises over the next 5 trading days")
print(f"Historical base rate: {base_rate:.0%} of weeks are positive")

What does this code do?

  • The :.0% format specifier turns a decimal like 0.63 into "63%" with no decimal places.
  • Printing the historical base rate next to the prediction gives you context. If the base rate is 57% and the model predicts 72%, the model is notably more bullish than history would suggest.
  • Save sp500_predictor.py.
  • Run the full script by running this command in your terminal:
python3 sp500_predictor.py

At the very bottom of the output, you should see a "Next Week Prediction" section showing a percentage probability and the historical base rate.

Not seeing the prediction output?

  • If you see a NameError for prediction_prob or base_rate, make sure the print statements come after the lines that define those variables.
  • If you see an IndexError, check that .iloc[[-1]] uses double brackets (two opening and two closing).
  • If the script errors earlier (before reaching your new code), scroll up to check for issues in previous sections.

Help me debug my prediction output

✔️ Awesome, I've got everything!

Great. Double check you've saved your file, then move on to the validation below.

ⓧ I'd like to double check the full code

Here's the complete sp500_predictor.py file. Compare yours against this:

import yfinance as yf
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, roc_auc_score, classification_report

# Step 2: Download S&P 500 data
print("Downloading S&P 500 data...")
data = yf.download("^GSPC", start="2000-01-01", progress=False)
print(f"Downloaded {data.shape[0]} rows of daily data")
print(data.head())

# Create target: did the market rise over the next 5 trading days?
data["Forward_Return"] = data["Close"].shift(-5) / data["Close"] - 1
data["Target"] = (data["Forward_Return"] > 0).astype(int)
data = data.dropna(subset=["Target"])

print(f"\nClass balance: {data['Target'].value_counts(normalize=True).to_dict()}")

# Step 3: Engineer features
data["return_5d"] = data["Close"].pct_change(5)
data["return_20d"] = data["Close"].pct_change(20)
data["return_60d"] = data["Close"].pct_change(60)
data["sma_20_ratio"] = data["Close"] / data["Close"].rolling(20).mean()
data["sma_50_ratio"] = data["Close"] / data["Close"].rolling(50).mean()
data["volatility_20d"] = data["Close"].pct_change().rolling(20).std()

# Drop NaN rows from rolling calculations
feature_columns = ["return_5d", "return_20d", "return_60d", "sma_20_ratio", "sma_50_ratio", "volatility_20d"]
data = data.dropna(subset=feature_columns)

print(f"\nFinal dataset: {data.shape[0]} rows, features: {feature_columns}")
print(data[feature_columns].head())

# Step 4: Time-based train/test split
split_date = data.index[-504]  # Approximately 2 years of trading days
train = data[data.index < split_date]
test = data[data.index >= split_date]

print(f"\nTraining set: {train.shape[0]} rows ({train.index[0].date()} to {train.index[-1].date()})")
print(f"Test set: {test.shape[0]} rows ({test.index[0].date()} to {test.index[-1].date()})")

X_train = train[feature_columns]
y_train = train["Target"]
X_test = test[feature_columns]
y_test = test["Target"]

# Train Random Forest
model = RandomForestClassifier(
    n_estimators=200,
    min_samples_split=50,
    random_state=42,
    class_weight="balanced"
)
model.fit(X_train, y_train)

# Evaluate
y_pred = model.predict(X_test)
y_prob = model.predict_proba(X_test)[:, 1]

accuracy = accuracy_score(y_test, y_pred)
roc_auc = roc_auc_score(y_test, y_prob)

print(f"\n--- Model Evaluation ---")
print(f"Accuracy: {accuracy:.3f}")
print(f"ROC-AUC: {roc_auc:.3f}")
print(f"\n{classification_report(y_test, y_pred, target_names=['Down', 'Up'])}")

# Step 5: Predict next week
latest_features = data[feature_columns].iloc[[-1]]
prediction_prob = model.predict_proba(latest_features)[0][1]
base_rate = data["Target"].mean()

print(f"\n--- Next Week Prediction ---")
print(f"Prediction: {prediction_prob:.0%} probability the S&P 500 rises over the next 5 trading days")
print(f"Historical base rate: {base_rate:.0%} of weeks are positive")

Why print the base rate alongside the prediction?

A prediction of "63% chance the market rises" sounds impressive, but the S&P 500 historically rises about 57% of weeks. The model is only 6 percentage points more bullish than always guessing "up".

Showing both numbers side-by-side lets you instantly see whether the model's signal is meaningfully different from the historical average. That's what makes the prediction actionable rather than just a number.

Secret mission

Analyze Feature Importance

Your model makes predictions, but which features is it actually paying attention to? Discover which market signals drive your Random Forest's decisions by extracting and ranking its built-in feature importance scores.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. All data comes from Yahoo Finance's free public API via yfinance. Decide whether to keep your project files, pause for later, or delete everything.

Resources you used:

  • The sp500-predictor project folder on your Desktop (contains sp500_predictor.py and the venv virtual environment).
  • Python packages (yfinance, pandas, scikit-learn) installed inside the virtual environment.

Keep everything running

No action needed. Choose this if you want to keep experimenting with your predictor or tweak the features.

  • Your sp500-predictor folder stays on your Desktop.
  • Next time you want to run the script, open a terminal in the project folder, activate the virtual environment with source venv/bin/activate (macOS) or venv\Scripts\activate (Windows), and run python3 sp500_predictor.py.

Pause - I'll come back to this later

Deactivate the virtual environment to free up your terminal session. Your files stay intact for when you return.

  • In your terminal, deactivate the virtual environment by running:
deactivate
  • You can close VS Code. Your project folder remains on your Desktop ready for next time.

Delete - I don't want to use this again

Remove all project files from your machine. This deletes the script, the virtual environment, and all installed packages.

  • Deactivate the virtual environment first by running:
deactivate
  • Navigate to your Desktop and delete the project folder.

macOS

cd ~/Desktop
rm -rf sp500-predictor

Windows

cd ~\Desktop
rmdir /s /q sp500-predictor
  • Confirm the folder is gone by running ls (macOS) or dir (Windows). You should no longer see sp500-predictor in the output.

That's a Wrap!

That's a Wrap!

Nice work! You just built a complete machine learning pipeline that downloads real financial data, engineers predictive features, trains a model with proper methodology, and outputs a probability prediction for the S&P 500's weekly direction.

You've learned how to:

  • Download 20+ years of S&P 500 daily price data and engineer 6 predictive features capturing momentum, trend, and volatility from raw OHLCV data.
  • Train a Random Forest classifier using a proper time-based train/test split that prevents data leakage.
  • Generate a calibrated probability prediction for the upcoming week's market direction, evaluated with ROC-AUC to confirm it outperforms random guessing.
  • Secret Mission: Analyzed feature importance to understand which market signals drive the model's predictions.

Ready to quiz yourself?