Build an End-to-End MLOps Pipeline

Build a stock price prediction pipeline with MLflow, FastAPI, and Evidently.

Introduction

30 Second Summary

Stock prices change every second, and companies invest millions trying to predict where they'll go next. The real challenge isn't just building a model. It's managing the entire lifecycle from raw data to a live prediction service.

In this project, you will build an end-to-end MLOps pipeline that fetches live stock data, trains a machine learning model, serves predictions through an API, and monitors for data drift.

What You'll Build

A complete MLOps pipeline for predicting next-day stock closing prices, using XGBoost for modeling, MLflow for experiment tracking, FastAPI for serving, Docker for containerization, and Evidently for drift monitoring.

By the end of this project, you'll have:

  • A reproducible data pipeline with DVC that fetches stock data from Yahoo Finance and engineers technical indicator features.
  • An MLflow experiment tracker logging parameters, metrics, and model artifacts across training runs.
  • A FastAPI prediction API containerized with Docker and deployed live on Render's free tier.
  • Secret Mission: automated retraining triggers that detect data drift and retrain the model.

Are there any prerequisites?

This project assumes basic Python knowledge and familiarity with pandas and scikit-learn. You do not need prior experience with MLOps tools. The project walks you through every tool from scratch.

Before We Start

Machine learning models are only as useful as the systems that support them in production. Training a model is one piece of the puzzle.

Getting it into a reproducible pipeline with versioned data, tracked experiments, and a live API is the full picture. In this step, you will install the tools that power a professional MLOps workflow and set up your project environment.

In this step, get ready to:
  • Install Python, Docker, and Git.
  • Create a virtual environment and install core Python packages.
  • Initialize a Git repository and DVC for data versioning.
Install Python, Docker, and Git

This project requires Python 3.11 or higher, Git for version control, and Docker for containerization. Let's confirm each tool is ready on your machine.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or Command Prompt (Windows) and press Enter to open it.
  • Check your Python version by running:

macOS

python3 --version

Windows

python --version

✔️ I see version 3.11 or higher

Python is ready. Move on to the Git check below.

ⓧ I see an older version

Your system has Python installed, but the version is below 3.11. On macOS, the default python3 often points to an older system version even after you install a newer one.

  • Download and install the latest Python 3.11 from python.org/downloads.
  • After installing, run python3.11 --version to confirm the new version is accessible.
  • Use python3.11 instead of python3 for the rest of this project if your default still shows an older version.

ⓧ Command not found

Python is not installed on your system.

  • Download and install Python 3.11 or higher from python.org/downloads.
  • On Windows, make sure to check Add Python to PATH during the installation wizard.
  • Close and reopen your terminal after installation.
  • Run the version check command again to confirm it works.
  • Check your Git version by running:
git --version

✔️ I see a version number

Git is ready. Move on to the Docker check below.

ⓧ Command not found

Git is not installed on your system.

  • Download and install Git from git-scm.com/downloads.
  • On macOS, installing the Xcode Command Line Tools also installs Git. Run xcode-select --install if you prefer this method.
  • Close and reopen your terminal after installation, then run git --version again to confirm.
  • Check your Docker installation by running:
docker --version

✔️ I see a version number

Docker is ready. You are all set with the prerequisites.

ⓧ Command not found

Docker is not installed on your system.

  • Download and install Docker Desktop from docker.com/get-started.
  • After installation, open Docker Desktop to complete the setup.
  • Close and reopen your terminal after Docker Desktop is running, then run docker --version again to confirm.
Create your project and install dependencies

With the prerequisites in place, you need a dedicated project folder and an isolated Python environment. A virtual environment keeps this project's packages separate from your system Python.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type VS Code and press Enter to open it.
  • Open a terminal inside VS Code by pressing Ctrl+` (macOS and Windows).
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir mlops-stock-prediction
cd mlops-stock-prediction
  • Create a Python virtual environment by running:

macOS

python3 -m venv .venv

Windows

python -m venv .venv
  • Activate the virtual environment by running:

macOS

source .venv/bin/activate

Windows

.venv\Scripts\activate

You should see (.venv) appear at the beginning of your terminal prompt. This confirms the virtual environment is active.

What is a virtual environment?

A virtual environment is an isolated copy of Python that keeps your project's packages separate from your system-wide installation. This way, different projects can use different library versions without conflicts.

  • Open the mlops-stock-prediction folder in VS Code's editor by clicking File in the top menu bar.
  • Click Open Folder.
  • Navigate to your Desktop, select the mlops-stock-prediction folder, and click Open.
  • If VS Code asks whether you trust the authors, click Yes, I trust the authors.
  • Reopen the integrated terminal by pressing Ctrl+` (macOS and Windows).
  • Reactivate your virtual environment in this new terminal by running:

macOS

source .venv/bin/activate

Windows

.venv\Scripts\activate
  • Create a new file by clicking the new file icon next to MLOPS-STOCK-PREDICTION in the Explorer sidebar.
  • Name the file requirements.txt and press Enter.
  • Paste the following content into requirements.txt:
yfinance==1.4.1
pandas>=2.0.0
numpy>=1.24.0
scikit-learn>=1.3.0
xgboost>=2.0.0
mlflow>=2.15.0
dvc>=3.50.0
evidently==0.7.21
fastapi[standard]>=0.113.0

What does this file do?

A requirements.txt file pins exact or minimum versions for every package your project needs. Anyone who clones your project can run a single install command to get the exact same environment.

  • yfinance fetches live stock market data from Yahoo Finance.
  • xgboost is the gradient-boosted tree model you will train.
  • mlflow tracks your experiments, including parameters, metrics, and model artifacts.
  • dvc versions your data files alongside Git.
  • evidently detects data drift in production.
  • fastapi serves predictions through a REST API.
  • Save requirements.txt by pressing Cmd+S (macOS) or Ctrl+S (Windows).
  • Install all the dependencies by running this command in your terminal:
pip install -r requirements.txt

This takes a minute or two as it downloads and installs all packages. You should see a final line that says Successfully installed followed by a list of package names.

Seeing installation errors?

Make sure your virtual environment is active (you see (.venv) in your terminal prompt). If a specific package fails, try upgrading pip first and then rerun the install command.

pip install --upgrade pip
pip install -r requirements.txt

help me troubleshoot package installation errors in my MLOps project

Initialize Git and DVC

Your project needs two version control systems working together. Git tracks your code and configuration files. DVC (Data Version Control) tracks your data files and ML pipeline stages alongside Git.

  • Initialize a Git repository in your project folder by running:
git init
  • Initialize DVC inside the Git repository by running:
dvc init

You should see a message confirming that DVC has been initialized. DVC creates a .dvc/ directory with its configuration files and automatically stages them for Git.

Why use DVC alongside Git?

Git is great for code, but large data files and model artifacts slow it down. DVC tracks those heavy files with lightweight pointer files in Git while storing the actual data separately.

This means your repository stays fast and your data remains reproducible.

  • Make your first commit to save the initial project state by running:
git add .
git commit -m "Initialize project with DVC"
  • Confirm your project files are set up by running:
ls

You should see requirements.txt listed in the output alongside a .dvc directory (visible if you run ls -a).

DVC init failed?

Make sure you ran git init first. DVC requires an existing Git repository to initialize properly. If you see a permission error, confirm your virtual environment is still active and that dvc was installed in it.

help me fix my DVC initialization error

✔️ Awesome, I've got everything!

Great. Double check that you have saved requirements.txt and that your Git commit went through successfully.

ⓧ I'd like to double check the full code

Here is the complete requirements.txt file for comparison:

yfinance==1.4.1
pandas>=2.0.0
numpy>=1.24.0
scikit-learn>=1.3.0
xgboost>=2.0.0
mlflow>=2.15.0
dvc>=3.50.0
evidently==0.7.21
fastapi[standard]>=0.113.0

Your environment is ready. Next up, you will build the data pipeline that fetches real stock market data and engineers features for your model.

Set Up Your MLOps Environment

An MLOps pipeline has many moving parts: data fetching scripts, feature engineering, training code, a serving application, and monitoring reports. Without clear project structure, these pieces become tangled and impossible to reproduce.

Your tools and packages are ready to go. In this step, you'll organize your project into a clean directory structure and define a DVC pipeline that connects your scripts into reproducible stages.

In this step, get ready to:
  • Create the project directory structure for scripts, data, and reports.
  • Define the DVC pipeline stages that will orchestrate your ML workflow.
Create the project directory structure

Each part of the ML lifecycle gets its own folder. Scripts go in src/, the API app goes in app/, datasets go in data/, and drift reports go in reports/.

  • In your terminal, create the project subdirectories by running this command:
mkdir -p src app data/raw data/processed reports
  • Confirm the directories exist by running:
ls

You should see src, app, data, reports, and your existing requirements.txt in the output.

What does this structure do?

  • src/ holds your pipeline scripts (data collection, feature engineering, training, monitoring).
  • app/ holds your FastAPI prediction service.
  • data/raw/ stores the original stock data fetched from Yahoo Finance.
  • data/processed/ stores the engineered features ready for training.
  • reports/ stores the drift monitoring HTML reports.

Not seeing all the folders?

Make sure you ran the command from your project folder (the same directory where requirements.txt lives). If you're in the wrong location, cd back to your project folder and try again.

help me find my project directory and create the subdirectories

Define your DVC pipeline

DVC pipelines let you define each stage of your ML workflow as a reproducible step. Each stage declares what command it runs, what files it depends on, and what files it outputs.

  • Create a new file called dvc.yaml in your project folder (where requirements.txt lives) and paste the following pipeline definition:
stages:
  fetch_data:
    cmd: python src/data_collection.py
    outs:
      - data/raw/stock_data.csv

  engineer_features:
    cmd: python src/feature_engineering.py
    deps:
      - data/raw/stock_data.csv
      - src/feature_engineering.py
    outs:
      - data/processed/features.csv

  train:
    cmd: python src/train.py
    deps:
      - data/processed/features.csv
      - src/train.py
    outs:
      - data/processed/reference_data.csv
    metrics:
      - metrics.json:
          cache: false

What does this pipeline do?

This file defines three stages that DVC will run in sequence:

  • fetch_data runs your data collection script and outputs the raw stock CSV.
  • engineer_features depends on the raw data and outputs processed features. DVC knows to rerun this stage only when its dependencies change.
  • train depends on the processed features and outputs reference data for monitoring. It also tracks a metrics.json file that DVC can compare across runs.

Later, running dvc repro will execute only the stages whose dependencies have changed. This makes your entire pipeline reproducible.

  • Save dvc.yaml.
  • Confirm DVC recognizes your pipeline by running:
dvc status

You should see output mentioning the pipeline stages. DVC will report them as changed or missing since you haven't run the scripts yet.

Seeing a "not a DVC repository" error?

This means your terminal isn't in the project folder where you initialized DVC. Navigate back to the folder containing requirements.txt and .dvc/ and try again.

help me fix the DVC repository not found error

✔️ Awesome, I've got everything!

Great. Double check you've saved dvc.yaml and you're ready for the next step.

ⓧ I'd like to double check the full code

Here's the complete dvc.yaml file. Make sure yours matches exactly:

stages:
  fetch_data:
    cmd: python src/data_collection.py
    outs:
      - data/raw/stock_data.csv

  engineer_features:
    cmd: python src/feature_engineering.py
    deps:
      - data/raw/stock_data.csv
      - src/feature_engineering.py
    outs:
      - data/processed/features.csv

  train:
    cmd: python src/train.py
    deps:
      - data/processed/features.csv
      - src/train.py
    outs:
      - data/processed/reference_data.csv
    metrics:
      - metrics.json:
          cache: false

Your MLOps project skeleton is in place. Next up, you'll write the data pipeline scripts that fetch live stock data and engineer features for your model.

Build the Data Pipeline

Your MLOps environment is ready, but a machine learning model is only as good as the data flowing into it. Before you can train anything, you need a reliable way to fetch real market data and transform it into features the model can learn from.

In this step, you'll build the first two stages of your DVC-managed pipeline: a data collection script that pulls live stock prices from Yahoo Finance, and a feature engineering script that computes technical indicators like moving averages and RSI.

In this step, get ready to:
  • Write a data collection script that fetches 5 years of AAPL stock data.
  • Engineer technical indicator features for the ML model.
  • Run the DVC pipeline to generate and track your datasets.
Write the data collection script

The yfinance library gives you free access to historical stock data from Yahoo Finance. Your script will fetch 5 years of daily OHLCV (Open, High, Low, Close, Volume) data for Apple (AAPL) and save it as a CSV file.

  • Create a new file called src/data_collection.py in VS Code and paste the following code:
import yfinance as yf
import pandas as pd
from pathlib import Path


def fetch_stock_data(ticker: str = "AAPL", period: str = "5y") -> pd.DataFrame:
    """Fetch historical stock data from Yahoo Finance."""
    stock = yf.Ticker(ticker)
    df = stock.history(period=period)
    df = df.reset_index()
    df = df[["Date", "Open", "High", "Low", "Close", "Volume"]]
    return df


def main():
    data_dir = Path("data/raw")
    data_dir.mkdir(parents=True, exist_ok=True)

    df = fetch_stock_data(ticker="AAPL", period="5y")
    df.to_csv(data_dir / "stock_data.csv", index=False)
    print(f"Fetched {len(df)} rows of AAPL stock data")


if __name__ == "__main__":
    main()

What does this code do?

The fetch_stock_data() function creates a yfinance Ticker object and calls .history(period="5y") to pull 5 years of daily price data. It then selects only the columns the model needs: Date, Open, High, Low, Close, and Volume.

The main() function ensures the data/raw directory exists, fetches the data, and saves it as stock_data.csv.

  • Save src/data_collection.py.
  • Test the script by running this command in your terminal:
python src/data_collection.py

You should see a message like Fetched 1258 rows of AAPL stock data (the exact number depends on trading days available).

Seeing a connection error?

  • Make sure you have an active internet connection. The script fetches live data from Yahoo Finance.
  • If you see a JSONDecodeError or timeout, wait a moment and try again. Yahoo Finance occasionally rate-limits requests.
  • Confirm your virtual environment is activated and yfinance is installed by running pip show yfinance.

Help me debug a yfinance connection error when fetching AAPL stock data.

✔️ Awesome, I've got everything!

Great. Make sure you've saved src/data_collection.py before moving on.

ⓧ I'd like to double check the full code

import yfinance as yf
import pandas as pd
from pathlib import Path


def fetch_stock_data(ticker: str = "AAPL", period: str = "5y") -> pd.DataFrame:
    """Fetch historical stock data from Yahoo Finance."""
    stock = yf.Ticker(ticker)
    df = stock.history(period=period)
    df = df.reset_index()
    df = df[["Date", "Open", "High", "Low", "Close", "Volume"]]
    return df


def main():
    data_dir = Path("data/raw")
    data_dir.mkdir(parents=True, exist_ok=True)

    df = fetch_stock_data(ticker="AAPL", period="5y")
    df.to_csv(data_dir / "stock_data.csv", index=False)
    print(f"Fetched {len(df)} rows of AAPL stock data")


if __name__ == "__main__":
    main()
Engineer features for the model

Raw stock prices alone aren't enough for a model to learn meaningful patterns. Technical indicators like moving averages, RSI, and MACD capture the momentum and trend signals that traders use to predict price movements.

What are technical indicators?

Technical indicators are mathematical calculations based on price and volume data. Traders use them to identify trends and reversals in the market.

  • Moving Averages (MA10, MA50) smooth out price noise over 10 and 50 days.
  • RSI (Relative Strength Index) measures if a stock is overbought or oversold on a 0-100 scale.
  • MACD tracks the difference between short-term and long-term momentum.
  • Lag features give the model access to recent historical closing prices.
  • Create a new file called src/feature_engineering.py and paste the following helper functions at the top:
import pandas as pd
import numpy as np
from pathlib import Path


def compute_rsi(series: pd.Series, window: int = 14) -> pd.Series:
    """Compute Relative Strength Index."""
    delta = series.diff()
    gain = delta.where(delta > 0, 0.0)
    loss = -delta.where(delta < 0, 0.0)
    avg_gain = gain.rolling(window=window).mean()
    avg_loss = loss.rolling(window=window).mean()
    rs = avg_gain / avg_loss
    rsi = 100 - (100 / (1 + rs))
    return rsi


def compute_macd(series: pd.Series) -> pd.Series:
    """Compute MACD (12-day EMA minus 26-day EMA)."""
    ema12 = series.ewm(span=12, adjust=False).mean()
    ema26 = series.ewm(span=26, adjust=False).mean()
    return ema12 - ema26

What do these helper functions do?

  • compute_rsi() calculates the Relative Strength Index using a 14-day rolling window. It compares average gains to average losses to produce a 0-100 score.
  • compute_macd() subtracts the 26-day exponential moving average from the 12-day EMA. A positive MACD signals upward momentum.

You won't see a change yet. The next function uses these helpers to build the full feature set.

  • Add the engineer_features() function below compute_macd():
def engineer_features(df: pd.DataFrame) -> pd.DataFrame:
    """Add technical indicators and lag features."""
    df = df.copy()

    # Moving averages
    df["MA10"] = df["Close"].rolling(window=10).mean()
    df["MA50"] = df["Close"].rolling(window=50).mean()

    # RSI
    df["RSI"] = compute_rsi(df["Close"])

    # MACD
    df["MACD"] = compute_macd(df["Close"])

    # Lag features
    df["Lag_1"] = df["Close"].shift(1)
    df["Lag_2"] = df["Close"].shift(2)
    df["Lag_3"] = df["Close"].shift(3)

    # Daily volatility
    df["Volatility"] = df["High"] - df["Low"]

    # Target: next day close
    df["Target"] = df["Close"].shift(-1)

    # Drop rows with NaN values
    df = df.dropna().reset_index(drop=True)

    return df

What does this function do?

This function takes the raw stock DataFrame and adds all the features the model will train on:

  • MA10 and MA50 are rolling averages that smooth out daily noise.
  • Lag_1, Lag_2, Lag_3 give the model the previous 3 days of closing prices.
  • Target is the next day's closing price, shifted by -1. This is what the model will learn to predict.
  • dropna() removes rows where rolling calculations or shifts produced NaN values.

You won't see a change yet. The entry point below ties everything together.

  • Add the main() function and entry point below engineer_features():
def main():
    raw_path = Path("data/raw/stock_data.csv")
    processed_dir = Path("data/processed")
    processed_dir.mkdir(parents=True, exist_ok=True)

    df = pd.read_csv(raw_path, parse_dates=["Date"])
    df = engineer_features(df)
    df.to_csv(processed_dir / "features.csv", index=False)
    print(f"Engineered features: {len(df)} rows, {len(df.columns)} columns")


if __name__ == "__main__":
    main()

What does the main function do?

  • It reads the raw CSV you fetched in the previous script from data/raw/stock_data.csv.
  • It runs engineer_features() to add all the technical indicators.
  • It saves the enriched DataFrame to data/processed/features.csv for the training step to consume.
  • Save src/feature_engineering.py.
  • Test the script by running:
python src/feature_engineering.py

You should see output like Engineered features: 1197 rows, 15 columns (the exact numbers depend on the data fetched).

Seeing a FileNotFoundError?

  • Make sure you ran python src/data_collection.py first. The feature engineering script reads from data/raw/stock_data.csv which must already exist.
  • Check that you are running the command from your project's root directory (the folder containing src/ and data/).

Help me fix a FileNotFoundError in my feature engineering script.

✔️ Awesome, I've got everything!

Great. Make sure you've saved src/feature_engineering.py before moving on.

ⓧ I'd like to double check the full code

import pandas as pd
import numpy as np
from pathlib import Path


def compute_rsi(series: pd.Series, window: int = 14) -> pd.Series:
    """Compute Relative Strength Index."""
    delta = series.diff()
    gain = delta.where(delta > 0, 0.0)
    loss = -delta.where(delta < 0, 0.0)
    avg_gain = gain.rolling(window=window).mean()
    avg_loss = loss.rolling(window=window).mean()
    rs = avg_gain / avg_loss
    rsi = 100 - (100 / (1 + rs))
    return rsi


def compute_macd(series: pd.Series) -> pd.Series:
    """Compute MACD (12-day EMA minus 26-day EMA)."""
    ema12 = series.ewm(span=12, adjust=False).mean()
    ema26 = series.ewm(span=26, adjust=False).mean()
    return ema12 - ema26


def engineer_features(df: pd.DataFrame) -> pd.DataFrame:
    """Add technical indicators and lag features."""
    df = df.copy()

    # Moving averages
    df["MA10"] = df["Close"].rolling(window=10).mean()
    df["MA50"] = df["Close"].rolling(window=50).mean()

    # RSI
    df["RSI"] = compute_rsi(df["Close"])

    # MACD
    df["MACD"] = compute_macd(df["Close"])

    # Lag features
    df["Lag_1"] = df["Close"].shift(1)
    df["Lag_2"] = df["Close"].shift(2)
    df["Lag_3"] = df["Close"].shift(3)

    # Daily volatility
    df["Volatility"] = df["High"] - df["Low"]

    # Target: next day close
    df["Target"] = df["Close"].shift(-1)

    # Drop rows with NaN values
    df = df.dropna().reset_index(drop=True)

    return df


def main():
    raw_path = Path("data/raw/stock_data.csv")
    processed_dir = Path("data/processed")
    processed_dir.mkdir(parents=True, exist_ok=True)

    df = pd.read_csv(raw_path, parse_dates=["Date"])
    df = engineer_features(df)
    df.to_csv(processed_dir / "features.csv", index=False)
    print(f"Engineered features: {len(df)} rows, {len(df.columns)} columns")


if __name__ == "__main__":
    main()
Run the DVC pipeline to track your data

You've verified both scripts work individually. Now let's use DVC to run them as an orchestrated pipeline so that every dataset is versioned and reproducible.

Why use DVC instead of running scripts directly?

DVC tracks which inputs produce which outputs. If your raw data changes, dvc repro automatically re-runs only the stages that are affected. This means you always know exactly which data produced your model.

Your dvc.yaml already defines the fetch_data and engineer_features stages from the previous step. The train stage also exists in the file, but you haven't written src/train.py yet. You'll run only the data stages for now.

  • First, delete the data files your manual test runs created, so DVC can regenerate them from scratch:
rm -rf data/raw/stock_data.csv data/processed/features.csv
  • Run the DVC pipeline up to the engineer_features stage by running:
dvc repro engineer_features

What does this command do?

The dvc repro engineer_features command tells DVC to reproduce the engineer_features stage and any stages it depends on. DVC reads dvc.yaml, sees that engineer_features depends on data/raw/stock_data.csv (produced by fetch_data), and runs both stages in the correct order.

  • Confirm both output files were created by running:
ls data/raw/stock_data.csv data/processed/features.csv

You should see both file paths listed without any errors. DVC has now generated .gitignore entries and dvc.lock to track exactly which code and parameters produced these files.

DVC says the stage failed?

  • If DVC reports a failure on fetch_data, it's likely a network issue fetching from Yahoo Finance. Check your internet connection and try again.
  • If you see an error about src/train.py not existing, make sure you specified the stage name: dvc repro engineer_features (not plain dvc repro which tries to run all stages).
  • Verify your dvc.yaml stage names match exactly. The commands reference python src/data_collection.py and python src/feature_engineering.py.

Help me troubleshoot a DVC pipeline failure.

Your data pipeline is fully operational. Next up, you'll train an XGBoost model on these features and track the experiment with MLflow.

Train and Track with MLflow

In the previous step, you built a data pipeline that fetches stock data and engineers features like moving averages and RSI. Now you have clean, feature-rich data ready for modeling.

Training a model once is easy. The hard part is tracking what you tried, what worked, and being able to reproduce your results. In this step, you will train an XGBoost model and log every detail to MLflow so you can compare experiments and register the best model for serving.

In this step, get ready to:
  • Configure MLflow to use a local SQLite backend for experiment tracking.
  • Train an XGBoost regressor and log parameters, metrics, and the model artifact.
  • Explore experiments in the MLflow UI and confirm your registered model.
Configure MLflow with a SQLite backend

MLflow needs a place to store experiment data (parameters, metrics, model artifacts). The simplest backend is a local SQLite database file, which means zero external services needed.

  • Create a new file called src/train.py in VS Code.
  • Add the imports, MLflow configuration, data loading, and time-series split by pasting this code:
import pandas as pd
import numpy as np
import mlflow
import mlflow.sklearn
from xgboost import XGBRegressor
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
from pathlib import Path
import json


def train_model():
    """Train XGBoost model and log to MLflow."""
    mlflow.set_tracking_uri("sqlite:///mlflow.db")
    mlflow.set_experiment("stock-price-prediction")

    # Load data
    df = pd.read_csv("data/processed/features.csv", parse_dates=["Date"])

    # Define features and target
    feature_cols = ["Open", "High", "Low", "Close", "Volume",
                    "MA10", "MA50", "RSI", "MACD",
                    "Lag_1", "Lag_2", "Lag_3", "Volatility"]
    target_col = "Target"

    # Time-series split (80/20)
    split_idx = int(len(df) * 0.8)
    train_df = df.iloc[:split_idx]
    test_df = df.iloc[split_idx:]

    X_train = train_df[feature_cols]
    y_train = train_df[target_col]
    X_test = test_df[feature_cols]
    y_test = test_df[target_col]

What does this code do?

  • mlflow.set_tracking_uri("sqlite:///mlflow.db") tells MLflow to store all experiment data in a local SQLite file called mlflow.db in your project root.
  • mlflow.set_experiment("stock-price-prediction") creates a named experiment to group related runs together.
  • The time-series split uses the first 80% of data for training and the last 20% for testing. This respects chronological order, which is critical for financial data where you cannot train on future prices.

Why a time-series split instead of random split?

With stock data, using a random train/test split would leak future information into the training set. The model would appear accurate in testing but fail on real predictions. A chronological split mimics how the model will be used in production: trained on past data, predicting the future.

Train the XGBoost model and log to MLflow

Now you will add the training loop inside an MLflow run context. Everything logged inside this block gets tracked as a single experiment run.

  • Add the following code directly below the last line you pasted (after y_test = test_df[target_col]):

    # Model parameters
    params = {
        "n_estimators": 200,
        "max_depth": 6,
        "learning_rate": 0.05,
        "subsample": 0.8,
        "colsample_bytree": 0.8,
        "random_state": 42,
    }

    with mlflow.start_run():
        # Log parameters
        mlflow.log_params(params)
        mlflow.log_param("ticker", "AAPL")
        mlflow.log_param("features", feature_cols)
        mlflow.log_param("train_size", len(train_df))
        mlflow.log_param("test_size", len(test_df))

        # Train model
        model = XGBRegressor(**params)
        model.fit(X_train, y_train)

        # Predict and evaluate
        y_pred = model.predict(X_test)
        rmse = np.sqrt(mean_squared_error(y_test, y_pred))
        mae = mean_absolute_error(y_test, y_pred)
        r2 = r2_score(y_test, y_pred)

        # Log metrics
        mlflow.log_metric("rmse", rmse)
        mlflow.log_metric("mae", mae)
        mlflow.log_metric("r2", r2)

        # Log model
        mlflow.sklearn.log_model(
            model,
            "model",
            registered_model_name="stock-price-xgboost"
        )

        print(f"RMSE: {rmse:.4f}")
        print(f"MAE: {mae:.4f}")
        print(f"R2: {r2:.4f}")

        # Save test data reference for monitoring
        test_df[feature_cols].to_csv("data/processed/reference_data.csv", index=False)

    return model


if __name__ == "__main__":
    train_model()

What does this code do?

  • mlflow.start_run() opens a tracking context. Everything inside this block (parameters, metrics, artifacts) gets recorded as a single experiment run.
  • mlflow.log_params(params) records the hyperparameters so you can reproduce or compare runs later.
  • Three metrics are computed: RMSE (root mean squared error), MAE (mean absolute error), and R2 (coefficient of determination). These tell you how close predictions are to actual prices.
  • mlflow.sklearn.log_model() saves the trained model as an artifact AND registers it in the MLflow Model Registry under the name stock-price-xgboost.
  • The reference data CSV is saved for drift monitoring in a later step.
  • Save src/train.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved src/train.py before moving on.

ⓧ I'd like to double check the full code

import pandas as pd
import numpy as np
import mlflow
import mlflow.sklearn
from xgboost import XGBRegressor
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
from pathlib import Path
import json


def train_model():
    """Train XGBoost model and log to MLflow."""
    mlflow.set_tracking_uri("sqlite:///mlflow.db")
    mlflow.set_experiment("stock-price-prediction")

    # Load data
    df = pd.read_csv("data/processed/features.csv", parse_dates=["Date"])

    # Define features and target
    feature_cols = ["Open", "High", "Low", "Close", "Volume",
                    "MA10", "MA50", "RSI", "MACD",
                    "Lag_1", "Lag_2", "Lag_3", "Volatility"]
    target_col = "Target"

    # Time-series split (80/20)
    split_idx = int(len(df) * 0.8)
    train_df = df.iloc[:split_idx]
    test_df = df.iloc[split_idx:]

    X_train = train_df[feature_cols]
    y_train = train_df[target_col]
    X_test = test_df[feature_cols]
    y_test = test_df[target_col]

    # Model parameters
    params = {
        "n_estimators": 200,
        "max_depth": 6,
        "learning_rate": 0.05,
        "subsample": 0.8,
        "colsample_bytree": 0.8,
        "random_state": 42,
    }

    with mlflow.start_run():
        # Log parameters
        mlflow.log_params(params)
        mlflow.log_param("ticker", "AAPL")
        mlflow.log_param("features", feature_cols)
        mlflow.log_param("train_size", len(train_df))
        mlflow.log_param("test_size", len(test_df))

        # Train model
        model = XGBRegressor(**params)
        model.fit(X_train, y_train)

        # Predict and evaluate
        y_pred = model.predict(X_test)
        rmse = np.sqrt(mean_squared_error(y_test, y_pred))
        mae = mean_absolute_error(y_test, y_pred)
        r2 = r2_score(y_test, y_pred)

        # Log metrics
        mlflow.log_metric("rmse", rmse)
        mlflow.log_metric("mae", mae)
        mlflow.log_metric("r2", r2)

        # Log model
        mlflow.sklearn.log_model(
            model,
            "model",
            registered_model_name="stock-price-xgboost"
        )

        print(f"RMSE: {rmse:.4f}")
        print(f"MAE: {mae:.4f}")
        print(f"R2: {r2:.4f}")

        # Save test data reference for monitoring
        test_df[feature_cols].to_csv("data/processed/reference_data.csv", index=False)

    return model


if __name__ == "__main__":
    train_model()
Run the training and explore the MLflow UI

Your training script is ready. Now you will run it to train the model and then launch the MLflow UI to explore the results.

  • Run the training script by executing this command in your terminal:
python src/train.py

You should see output showing RMSE, MAE, and R2 values, followed by a message confirming the model was registered. The exact numbers will vary depending on the stock data fetched.

What should I see?

Your terminal will print three metrics (RMSE, MAE, R2) and then show a registration confirmation like:

RMSE: 3.XXXX
MAE: 2.XXXX
R2: 0.9XXX
Successfully registered model 'stock-price-xgboost'.
Created version '1' of model 'stock-price-xgboost'.

The exact metric values depend on the market data that was downloaded. An R2 above 0.9 indicates the model explains most of the variance in next-day closing prices.

Seeing a FileNotFoundError?

  • Make sure you ran python src/data_collection.py and python src/feature_engineering.py in the previous step. The training script needs data/processed/features.csv to exist.
  • Confirm you are running the command from your project root folder (the one containing the src/ directory).

Having trouble with the training script?

  • Now launch the MLflow UI to explore your experiment by running this command in a new terminal:
mlflow server --backend-store-uri sqlite:///mlflow.db --port 5000
  • Open your browser and navigate to http://localhost:5000.
  • Click on the stock-price-prediction experiment in the left sidebar.
  • Click on the run to see the logged parameters, metrics, and model artifact.

What is the MLflow Model Registry?

The Model Registry is a central hub for managing model versions. When you passed registered_model_name="stock-price-xgboost" in your training script, MLflow automatically created a registered model and assigned it version 1.

Every time you retrain and log a new model under the same name, MLflow increments the version number. This makes it easy to roll back to a previous version if a new model performs worse.

MLflow UI not loading?

  • Make sure the mlflow server command is still running in your terminal. It must stay active for the UI to work.
  • If port 5000 is already in use, stop the other process or change --port 5000 to another port like 5001.
  • Confirm that mlflow.db exists in your project root. It is created when python src/train.py runs successfully.

Still stuck with the MLflow UI?

Your model is trained, tracked, and registered. Next up, you will serve this model as a live REST API using FastAPI.

Serve the Model with FastAPI

You have a trained model sitting in MLflow's registry, but it can only make predictions if you load it manually in a script. To make it useful for other applications (or users), you need to expose it as an API that accepts feature values and returns a prediction.

In this step, you will build a FastAPI application that loads your registered model at startup and serves predictions over HTTP. This is how production ML systems work: the model lives behind a REST endpoint that any client can call.

In this step, get ready to:
  • Build a FastAPI application with request and response schemas.
  • Use the lifespan pattern to load your MLflow model at startup.
  • Test the prediction API locally with sample requests.
Create the app directory and define the data schemas

Your API needs to know the exact shape of incoming prediction requests and outgoing responses. In FastAPI, you define these shapes using Pydantic models.

  • Create a new app/ directory in your project root by running this command:
mkdir app
  • Create a new file called main.py inside the app/ directory by running:
touch app/main.py
  • Confirm both exist by running:
ls app/

You should see main.py listed in the output.

  • Open app/main.py in VS Code and add the imports and Pydantic schema models by pasting the following code:
import pickle
from contextlib import asynccontextmanager
from pathlib import Path

import mlflow
import pandas as pd
from fastapi import FastAPI
from pydantic import BaseModel


class PredictionRequest(BaseModel):
    open: float
    high: float
    low: float
    close: float
    volume: float
    ma10: float
    ma50: float
    rsi: float
    macd: float
    lag_1: float
    lag_2: float
    lag_3: float
    volatility: float


class PredictionResponse(BaseModel):
    predicted_close: float
    ticker: str

What does this code do?

The imports bring in everything you need: mlflow to load your registered model, pandas to create the feature DataFrame, and FastAPI plus BaseModel for the web framework and data validation.

The PredictionRequest model defines the 13 features your XGBoost model expects. FastAPI will automatically validate that incoming requests contain all these fields with the right types. PredictionResponse defines what the API returns: a predicted closing price and the ticker symbol.

  • Save app/main.py.
Use the lifespan pattern to load the model at startup

Loading an ML model takes time. You only want to do it once when the server starts, not on every request. FastAPI's lifespan pattern lets you run setup code before the app begins serving traffic.

  • In app/main.py, add the lifespan context manager and FastAPI app initialization below the PredictionResponse class:
ml_models = {}


@asynccontextmanager
async def lifespan(app: FastAPI):
    # Load the ML model at startup
    mlflow.set_tracking_uri("sqlite:///mlflow.db")
    model_uri = "models:/stock-price-xgboost/1"
    ml_models["model"] = mlflow.sklearn.load_model(model_uri)
    yield
    # Clean up
    ml_models.clear()


app = FastAPI(
    title="Stock Price Prediction API",
    description="Predict next-day stock closing price using XGBoost",
    version="1.0.0",
    lifespan=lifespan,
)

What is the lifespan pattern?

The @asynccontextmanager decorator turns your function into a context manager. Everything before yield runs when the app starts (loading the model from MLflow). Everything after yield runs when the app shuts down (clearing the models dictionary).

The ml_models dictionary acts as a global store. Your endpoint functions access the loaded model through this dictionary instead of loading it on every request.

  • Save app/main.py.

Seeing import errors in VS Code?

Make sure your virtual environment is selected as the Python interpreter in VS Code. Press Cmd+Shift+P (macOS) or Ctrl+Shift+P (Windows), type Python: Select Interpreter, and choose the one inside your project's virtual environment.

Help me fix import errors in my FastAPI file.

Add the prediction endpoints and test the API

With the model loading handled, you need two endpoints: a health check to confirm the server is running, and the prediction endpoint itself.

  • In app/main.py, add both endpoint functions below the app = FastAPI(...) block:
@app.get("/health")
def health_check():
    return {"status": "healthy", "model_loaded": "model" in ml_models}


@app.post("/predict", response_model=PredictionResponse)
def predict(request: PredictionRequest):
    feature_names = [
        "Open", "High", "Low", "Close", "Volume",
        "MA10", "MA50", "RSI", "MACD",
        "Lag_1", "Lag_2", "Lag_3", "Volatility",
    ]
    features = pd.DataFrame(
        [[
            request.open, request.high, request.low, request.close,
            request.volume, request.ma10, request.ma50, request.rsi,
            request.macd, request.lag_1, request.lag_2, request.lag_3,
            request.volatility,
        ]],
        columns=feature_names,
    )

    model = ml_models["model"]
    prediction = model.predict(features)[0]

    return PredictionResponse(predicted_close=round(float(prediction), 2), ticker="AAPL")

What do these endpoints do?

  • `/health` is a GET endpoint that returns whether the model is loaded. Load balancers and monitoring tools use health checks like this.
  • `/predict` is a POST endpoint that takes the 13 features as JSON, builds a pandas DataFrame matching the column names your model was trained on, runs model.predict(), and returns the predicted next-day close price.
  • Save app/main.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved app/main.py with all three sections: imports and schemas, lifespan and app, and both endpoints.

ⓧ I'd like to double check the full code

import pickle
from contextlib import asynccontextmanager
from pathlib import Path

import mlflow
import pandas as pd
from fastapi import FastAPI
from pydantic import BaseModel


class PredictionRequest(BaseModel):
    open: float
    high: float
    low: float
    close: float
    volume: float
    ma10: float
    ma50: float
    rsi: float
    macd: float
    lag_1: float
    lag_2: float
    lag_3: float
    volatility: float


class PredictionResponse(BaseModel):
    predicted_close: float
    ticker: str


ml_models = {}


@asynccontextmanager
async def lifespan(app: FastAPI):
    # Load the ML model at startup
    mlflow.set_tracking_uri("sqlite:///mlflow.db")
    model_uri = "models:/stock-price-xgboost/1"
    ml_models["model"] = mlflow.sklearn.load_model(model_uri)
    yield
    # Clean up
    ml_models.clear()


app = FastAPI(
    title="Stock Price Prediction API",
    description="Predict next-day stock closing price using XGBoost",
    version="1.0.0",
    lifespan=lifespan,
)


@app.get("/health")
def health_check():
    return {"status": "healthy", "model_loaded": "model" in ml_models}


@app.post("/predict", response_model=PredictionResponse)
def predict(request: PredictionRequest):
    feature_names = [
        "Open", "High", "Low", "Close", "Volume",
        "MA10", "MA50", "RSI", "MACD",
        "Lag_1", "Lag_2", "Lag_3", "Volatility",
    ]
    features = pd.DataFrame(
        [[
            request.open, request.high, request.low, request.close,
            request.volume, request.ma10, request.ma50, request.rsi,
            request.macd, request.lag_1, request.lag_2, request.lag_3,
            request.volatility,
        ]],
        columns=feature_names,
    )

    model = ml_models["model"]
    prediction = model.predict(features)[0]

    return PredictionResponse(predicted_close=round(float(prediction), 2), ticker="AAPL")

Now start the FastAPI development server to test your API.

  • Launch the server by running this command:
fastapi dev app/main.py

What does this command do?

The fastapi dev command starts a local development server with auto-reload enabled. It detects your app object in app/main.py and serves it at http://127.0.0.1:8000. When the server starts, it triggers the lifespan function, which loads your MLflow model into memory.

You should see output showing the server started at http://127.0.0.1:8000 with a message about application startup complete.

  • Open a second terminal window (keep the server running in the first one).
  • Test the health endpoint by running:
curl http://127.0.0.1:8000/health

You should see a JSON response like {"status":"healthy","model_loaded":true}.

  • Test the prediction endpoint with sample feature values by running:
curl -X POST http://127.0.0.1:8000/predict \
  -H "Content-Type: application/json" \
  -d '{"open": 175.50, "high": 177.30, "low": 174.80, "close": 176.90, "volume": 52000000, "ma10": 175.20, "ma50": 172.80, "rsi": 58.5, "macd": 1.23, "lag_1": 176.10, "lag_2": 175.80, "lag_3": 174.50, "volatility": 2.50}'

You should see a JSON response containing a predicted_close value and "ticker": "AAPL". The exact prediction number depends on your trained model.

Explore the interactive docs

FastAPI auto-generates interactive API documentation. Open http://127.0.0.1:8000/docs in your browser to see a Swagger UI where you can try out both endpoints without using curl.

Server won't start or model fails to load?

  • Make sure you are running the command from your project root directory (the folder containing mlflow.db). The lifespan function uses a relative path to find the database.
  • If you see an error about the model not being found, verify you registered a model named stock-price-xgboost in the previous step. You can check by running mlflow models list or checking the MLflow UI.
  • If port 8000 is already in use, stop any other process running on that port or close the previous MLflow server first.

Help me debug why my FastAPI server won't start or the model fails to load.

  • Stop the server by pressing Ctrl+C in the terminal where it is running.

Your ML model is now serving predictions over HTTP. Next up, you will containerize this application with Docker and deploy it to a live endpoint on Render.

Containerize and Deploy

Your prediction API works locally. Now it is time to package it into a portable container and ship it to a live URL that anyone can access.

In this step, you will write a Dockerfile that bundles your FastAPI app with its trained model, test the container on your machine, and then deploy the whole thing to Render's free tier.

In this step, get ready to:
  • Write a Dockerfile that packages the API and model artifacts.
  • Build and test the Docker image locally.
  • Deploy the containerized app to Render's free tier.
Write the Dockerfile

A Dockerfile is a recipe that tells Docker how to assemble your application into a self-contained image. The image includes the Python runtime, your dependencies, your application code, and the trained model files.

  • Create a new file called Dockerfile in the root of your project folder (the same level as requirements.txt).
  • Paste the following into Dockerfile:
FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app/ ./app/
COPY mlflow.db .
COPY mlruns/ ./mlruns/

EXPOSE 8000

CMD ["fastapi", "run", "app/main.py", "--port", "8000"]

What does this Dockerfile do?

  • FROM python:3.11-slim starts from a lightweight Python 3.11 base image.
  • COPY requirements.txt and RUN pip install install all your Python dependencies first. This layer gets cached so rebuilds are fast.
  • The next COPY instructions bring in your FastAPI app code, the MLflow database, and the model artifacts stored in mlruns/.
  • CMD tells Docker to start FastAPI on port 8000 when the container launches.
  • Save Dockerfile.

Seeing a file extension issue?

Make sure the file is named exactly Dockerfile with no extension (not Dockerfile.txt). Some editors add extensions automatically.

help me fix my Dockerfile naming

Build and test the Docker image locally

Before deploying anywhere, you should verify the container runs correctly on your own machine. This catches missing files and dependency issues early.

  • Build the Docker image by running this command:
docker build -t stock-predictor .

What does this command do?

docker build reads your Dockerfile and assembles the image layer by layer. The -t stock-predictor flag gives the image a name you can reference later. The . at the end tells Docker to use the current directory as the build context.

  • The build may take a few minutes as it downloads the base image and installs packages.
  • Once the build finishes, start a container from your new image by running:
docker run -p 8000:8000 stock-predictor
  • Open a new terminal tab and test the health endpoint by running:
curl http://localhost:8000/health
  • You should see a JSON response: {"status":"healthy","model_loaded":true}.
  • Press Ctrl+C in the terminal running the container to stop it.

Container exits immediately or model fails to load?

  • Make sure mlflow.db and the mlruns/ directory exist in your project root. These were created in Step 3 when you trained the model.
  • Check that your model is registered as version 1. Run mlflow models list to confirm.
  • If the image builds but the container crashes, check the logs that Docker prints to the terminal for a traceback.

help me debug my Docker container crash

Deploy to Render

Render is a cloud platform that can build and run your Docker image directly from a Git repository. You will push your project to GitHub, then connect that repo to Render for automatic deployment.

  • First, push your project to a GitHub repository. Add all files and commit by running:
git add .
git commit -m "Add Dockerfile for deployment"
  • Create a new repository on GitHub (public or private) and push your code by running:
git remote add origin [[YOUR_REPO_URL="https://github.com/your-username/stock-predictor.git"]]
git branch -M main
git push -u origin main

Why push to Git first?

Render deploys by pulling your source code from a linked Git repository. Every time you push to your main branch, Render automatically rebuilds and redeploys your service.

  • Open https://render.com in your browser and sign up for a free account (or sign in if you already have one).
  • In the Render Dashboard, click + New in the top-right corner.
  • Select Web Service.
  • Select Git Provider and connect your GitHub account if you have not already.
  • Select the repository you just pushed and click Connect.
  • In the service creation form, set the Language field to Docker.
  • Choose the Free instance type.
  • Click Deploy at the bottom of the form.

What is Render doing now?

Render clones your repository, detects the Dockerfile, builds the image using BuildKit, and starts a container from it. The free instance gives you 512 MB RAM and 0.1 CPU. It spins down after 15 minutes of inactivity to save resources.

  • Wait for the deploy status to show Live in the Render Dashboard. This may take 5-10 minutes for the first build.
  • Once live, find your service's unique .onrender.com URL on your service's page in the Dashboard.
  • Test your live API by running:
curl [[RENDER_URL="https://your-service-name.onrender.com"]]/health
  • You should see the same healthy response: {"status":"healthy","model_loaded":true}.

Deploy failed or health check not passing?

  • Check the deploy logs in the Render Dashboard for error messages. Common issues include missing files that were not committed to Git.
  • Make sure mlflow.db and the mlruns/ directory are committed. Run git status to check for untracked files.
  • If you see memory errors, the model may be too large for the free tier. Try reducing the number of estimators in your training parameters.

help me debug my Render deployment

✔️ Awesome, I've got everything!

Great. Double check you have saved your Dockerfile and that your Render service is showing Live.

ⓧ I'd like to double check the full code

Here is the complete Dockerfile for this step:

FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app/ ./app/
COPY mlflow.db .
COPY mlruns/ ./mlruns/

EXPOSE 8000

CMD ["fastapi", "run", "app/main.py", "--port", "8000"]

Your prediction API is now live on the internet. Next up, you will add monitoring to detect when the data your model sees starts drifting from what it was trained on.

Monitor for Data Drift with Evidently

Your prediction API is live on Render. Anyone can hit the endpoint and get a stock price forecast.

But how do you know the model is still accurate? Financial markets shift constantly. The data your model sees in production might look very different from the data it trained on. This is called data drift, and it silently degrades model performance. In this step, you will build a monitoring script that detects when your data has drifted.

In this step, get ready to:
  • Write a drift detection script using Evidently's DataDriftPreset.
  • Simulate drift by fetching recent stock data.
  • Generate an HTML drift report comparing training data vs. current data.
Write the drift monitoring script

Your model was trained on historical data that reflects certain patterns. Over time, the real-world data feeding into your API might shift in distribution. Evidently is an open-source library that statistically compares two datasets and flags when features have drifted.

What is data drift?

Data drift means the statistical properties of your model's input features have changed since training. For example, if AAPL's daily volume suddenly doubles compared to the training period, Volume has drifted.

Evidently uses the Population Stability Index (PSI) method to quantify this. PSI below 0.1 means no significant drift. PSI above 0.25 signals a major shift that likely requires retraining.

  • Create a new file called src/monitor.py inside your src folder.
  • Add the imports and the run_drift_detection() function by pasting this code:
import pandas as pd
from evidently import Report
from evidently.presets import DataDriftPreset
from pathlib import Path


def run_drift_detection(reference_path: str, current_path: str, report_path: str):
    """Run data drift detection comparing reference and current data."""
    reference_df = pd.read_csv(reference_path)
    current_df = pd.read_csv(current_path)

    report = Report([DataDriftPreset(method="psi")], include_tests=True)
    my_eval = report.run(reference_df, current_df)
    my_eval.save_html(report_path)

    print(f"Drift report saved to {report_path}")
    return my_eval

What does this code do?

The run_drift_detection() function loads two CSV files: the reference data (what the model trained on) and the current data (what it sees now in production).

It creates an Evidently Report configured with DataDriftPreset(method="psi") to use the Population Stability Index for comparison. The include_tests=True flag adds pass/fail drift tests for each feature. Finally, my_eval.save_html() generates a visual HTML report.

Why PSI instead of other methods?

PSI works well for continuous numerical features like stock prices and volumes. It bins the data into ranges and measures how much the distribution has shifted between reference and current datasets. It is a common choice in financial ML because it gives interpretable, threshold-based results.

  • Below the run_drift_detection() function, add the main() function and entry point:
def main():
    reports_dir = Path("reports")
    reports_dir.mkdir(parents=True, exist_ok=True)

    run_drift_detection(
        reference_path="data/processed/reference_data.csv",
        current_path="data/processed/current_data.csv",
        report_path="reports/data_drift_report.html",
    )


if __name__ == "__main__":
    main()

What does this code do?

The main() function creates a reports/ directory if it does not exist, then calls run_drift_detection() with three paths.

The reference path points to data/processed/reference_data.csv which was saved during training in Step 3. This is the baseline distribution your model expects. The current path points to data/processed/current_data.csv which you will generate in the next substep.

  • Save src/monitor.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved your file before moving on.

ⓧ I'd like to double check the full code

import pandas as pd
from evidently import Report
from evidently.presets import DataDriftPreset
from pathlib import Path


def run_drift_detection(reference_path: str, current_path: str, report_path: str):
    """Run data drift detection comparing reference and current data."""
    reference_df = pd.read_csv(reference_path)
    current_df = pd.read_csv(current_path)

    report = Report([DataDriftPreset(method="psi")], include_tests=True)
    my_eval = report.run(reference_df, current_df)
    my_eval.save_html(report_path)

    print(f"Drift report saved to {report_path}")
    return my_eval


def main():
    reports_dir = Path("reports")
    reports_dir.mkdir(parents=True, exist_ok=True)

    run_drift_detection(
        reference_path="data/processed/reference_data.csv",
        current_path="data/processed/current_data.csv",
        report_path="reports/data_drift_report.html",
    )


if __name__ == "__main__":
    main()
Simulate drift with recent data

Your monitoring script compares reference data against current data. The reference data already exists from training. Now you need to generate a current_data.csv file representing what the model might see in production today.

  • Generate current data by fetching the last month of AAPL stock data and engineering features. Run this command in your terminal:
python -c "
from src.data_collection import fetch_stock_data
from src.feature_engineering import engineer_features
df = fetch_stock_data(ticker='AAPL', period='1mo')
df = engineer_features(df)
feature_cols = ['Open', 'High', 'Low', 'Close', 'Volume', 'MA10', 'MA50', 'RSI', 'MACD', 'Lag_1', 'Lag_2', 'Lag_3', 'Volatility']
df[feature_cols].to_csv('data/processed/current_data.csv', index=False)
print(f'Generated {len(df)} rows of current data')
"

What does this command do?

This reuses your existing fetch_stock_data() and engineer_features() functions to grab the most recent month of AAPL data. It extracts only the feature columns (the same 13 features your model uses) and saves them to data/processed/current_data.csv.

Because this data is from a different time window than the training data, its distributions may differ. This is exactly what drift detection is designed to catch.

  • Confirm the file was created by running:
ls data/processed/current_data.csv

You should see the file path printed, confirming current_data.csv exists.

Seeing an import error?

If Python cannot find src.data_collection, make sure you are running the command from your project root directory (the folder containing the src/ folder). Also confirm your virtual environment is still activated.

help me fix this import error when generating current data

Run the drift detection

You now have both datasets ready. Time to compare them and see if the recent market data has drifted from what your model was trained on.

  • Run the monitoring script by executing:
python src/monitor.py
  • You should see a message like Drift report saved to reports/data_drift_report.html in your terminal.
  • Open the generated report by running:
open reports/data_drift_report.html

On Windows, use start reports/data_drift_report.html instead.

The report opens in your browser showing a dashboard with drift results per feature. Each feature gets a PSI score and a pass/fail verdict.

How do I read the drift report?

The report shows each of your 13 features with its PSI value. A green checkmark means no significant drift was detected for that feature. A red X means the feature's distribution has shifted beyond the threshold.

It is normal to see some drift between a 5-year training window and a 1-month recent window. Volume and volatility tend to drift the most because market conditions change frequently.

Script failing with a file not found error?

Make sure data/processed/reference_data.csv exists from Step 3 (training). Also confirm data/processed/current_data.csv was created in the previous substep. Run ls data/processed/ to see both files listed.

help me debug why my drift detection script cannot find the data files

You now have a complete monitoring system that can detect when your model's inputs have shifted. In the Secret Mission, you will wire this into an automated retraining pipeline that kicks off when drift is detected.

Secret mission

Automated Retraining on Drift

Your monitoring script detects drift, but what happens next? In production, teams don't retrain models by hand. Build an automated retraining pipeline that detects when data has shifted beyond a threshold, re-fetches fresh data, retrains the model, and logs the new run to MLflow. Then redeploy the updated model to your live endpoint.

Clean Up Your Resources

Clean Up Your Resources

This project runs on Render's free tier, which costs $0/month and auto-sleeps after 15 minutes of inactivity. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Render free web service (auto-sleeps after 15 minutes of inactivity).
  • Local Docker images built during the project.
  • Local project folder (includes mlflow.db, mlruns/, data/, src/, app/, and reports/).

Keep everything running

No action needed. Choose this if you are still actively building or experimenting with your pipeline.

  • Your Render service stays deployed at $0/month. It spins down automatically after 15 minutes of inactivity and wakes up on the next request.
  • Your local MLflow data (mlflow.db and mlruns/) stays on your machine for future experiments.
  • Docker images remain cached locally for faster rebuilds.

Pause - I'll come back to this later

Render's free tier auto-sleeps after 15 minutes of inactivity, so there is nothing to pause manually. Your service is effectively paused whenever it is not receiving requests.

  • Stop any running MLflow server by pressing Ctrl+C in its terminal window.
  • Close VS Code and any open terminal sessions to free up memory.
  • Your project files, Docker images, and Render service will all still be there when you come back.

Delete - I don't want to use this again

Remove all project resources and start fresh. Follow each step below to fully clean up.

  • Delete your Render web service:
  • Open the Render Dashboard and click on your web service.
  • Click Settings in the left sidebar.
  • Scroll to the bottom of the page and click Delete Web Service.
  • Type the service name to confirm deletion.
  • Remove local Docker images by running this command in your terminal:
docker image prune -a
  • Type y and press Enter when prompted to confirm.
  • Delete the local project folder. Navigate to your Desktop and remove the project directory:

macOS

cd ~/Desktop
rm -rf stock-mlops-pipeline

Windows

cd ~\Desktop
rmdir /s /q stock-mlops-pipeline

What about Python, Git, and Docker?

These are general-purpose tools useful beyond this project. You do not need to uninstall them unless you are certain you will not use them again.

Nice Work!

Nice Work!

You just built a production-grade MLOps pipeline from scratch. You took a machine learning model from raw stock data all the way to a live prediction API with proper versioning, experiment tracking, and drift monitoring.

Here's what you accomplished:

  • Built a reproducible data pipeline with DVC and trained an XGBoost model with full experiment tracking in MLflow (parameters, metrics, and model registry).
  • Served predictions via a FastAPI REST API, containerized the application with Docker, and deployed it to a live endpoint on Render.
  • Monitored for data drift using Evidently to detect when incoming data distributions shift from your training baseline.
  • Secret Mission: Added automated retraining triggers that detect drift and retrain the model, logging fresh experiments to MLflow.

Ready to quiz yourself?