Build a Production MLOps Pipeline

Build a churn prediction pipeline with MLflow, FastAPI, Docker, and CI/CD.

Introduction

30 Second Summary

Most machine learning models never make it past the Jupyter notebook. The gap between a trained model and a production system that reliably serves predictions is where most ML projects stall.

In this project, you will build a complete MLOps pipeline that takes a customer churn prediction model from training all the way through to containerized deployment with CI/CD automation and drift monitoring.

What You'll Build

A production-grade MLOps pipeline for customer churn prediction, featuring experiment tracking, a REST API, containerized deployment, automated testing, and data drift detection.

By the end of this project, you'll have:

  • A trained churn prediction model with full experiment tracking and a model registry in MLflow.
  • A FastAPI REST API serving predictions, containerized with Docker and tested with pytest.
  • A GitHub Actions CI/CD pipeline that trains the model, runs tests, detects data drift with Evidently AI, and builds the container on every push.
  • Secret Mission: An automated model quality gate that blocks degraded models from reaching production.

Are there any prerequisites?

You should have basic Python familiarity and be comfortable running Docker commands. You also need a GitHub account.

No prior MLOps or machine learning experience is required. The project walks you through every step.

Before We Start

Before We Start...

Set Up Your Project Environment

Before you can train models and build APIs, you need a clean, isolated workspace where all your tools play nicely together.

In this step, you'll create a project folder, set up a Python virtual environment, install your dependencies, and connect everything to GitHub so your code is version-controlled from the start.

In this step, get ready to:
  • Create a Python virtual environment and install core dependencies.
  • Initialize a Git repository and connect it to GitHub.
  • Verify all tools are working with quick version checks.
Create a Python virtual environment and install dependencies

A virtual environment keeps this project's packages separate from any other Python projects on your machine.

  • Open your terminal. Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal and press Enter to open it.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir mlops-churn-pipeline
cd mlops-churn-pipeline
  • Confirm the folder exists by running:
ls

You should see an empty directory listing since the folder is brand new.

  • Create a Python virtual environment by running:
python3 -m venv venv

What does this command do?

python3 -m venv venv tells Python to create a virtual environment in a folder called venv. This isolates your project's packages from the rest of your system.

  • Activate the virtual environment by running:

macOS

source venv/bin/activate

Windows

.\venv\Scripts\Activate.ps1
  • Confirm the virtual environment is active. You should see (venv) at the beginning of your terminal prompt.

Don't see (venv) in your prompt?

On Windows, if you get a permissions error, you may need to run Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned first, then try the activation again.

On macOS/Linux, make sure you typed source before the path. Without it, the command won't modify your current shell session.

Help me activate my Python virtual environment.

Now you'll create a requirements.txt file that lists every package your project needs with exact version numbers.

  • Open your project in Cursor. Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Cursor and press Enter to open it.
  • Click File in the top menu bar.
  • Click Open Folder.
  • Select the mlops-churn-pipeline folder on your Desktop, then click Open.
  • Create a new file called requirements.txt by clicking the new file icon in the Explorer sidebar.
  • Paste the following into requirements.txt:
scikit-learn==1.8.0
mlflow==3.12.0
fastapi[standard]==0.136.1
uvicorn==0.34.0
pandas==2.2.3
evidently==0.7.21
pytest==8.3.4
httpx==0.28.1

What are these packages?

  • scikit-learn is your machine learning library for training the churn model.
  • mlflow handles experiment tracking and model registry.
  • fastapi and uvicorn power your prediction API.
  • evidently detects data drift in production.
  • pytest and httpx run your automated test suite.
  • Save requirements.txt.

✔️ Awesome, I've got everything!

Great. Make sure you've saved the file before moving on.

ⓧ I'd like to double check the full code

scikit-learn==1.8.0
mlflow==3.12.0
fastapi[standard]==0.136.1
uvicorn==0.34.0
pandas==2.2.3
evidently==0.7.21
pytest==8.3.4
httpx==0.28.1
  • Back in your terminal, install all the packages by running:
pip install -r requirements.txt
  • Wait for the installation to finish. This may take 2-3 minutes depending on your internet connection.
  • You should see a final line that says Successfully installed followed by a list of packages.

Seeing installation errors?

If you see a version conflict or a "no matching distribution" error, make sure your (venv) prefix is showing in your prompt. Installing without an active virtual environment can cause version conflicts with system packages.

If the download is timing out, check your internet connection and try running the install command again.

Help me fix my pip install errors.

Initialize a Git repository and connect to GitHub

Version control lets you track every change to your pipeline. It also enables the CI/CD automation you'll build in a later step.

  • Initialize a Git repository in your project folder by running:
git init

Next, you'll set the MLflow tracking URI. MLflow's Model Registry requires a database-backed store.

You'll use a local SQLite database so the registry works without any cloud setup.

  • Set the MLFLOW_TRACKING_URI environment variable by running:

macOS

export MLFLOW_TRACKING_URI=sqlite:///mlflow.db

Windows

$env:MLFLOW_TRACKING_URI = "sqlite:///mlflow.db"

Why SQLite for MLflow?

MLflow's file-based backend does not support the Model Registry. By pointing MLflow at a SQLite database, you get full registry support (registering models, loading by version) without needing a remote server.

The database file (mlflow.db) is created automatically the first time MLflow writes to it.

Now you'll create a repository on GitHub and connect it as your remote.

  • Open your browser and go to github.com.
  • Click the + icon in the top-right corner.
  • Select New repository.
  • Enter mlops-churn-pipeline as the repository name.
  • Set visibility to Public.
  • Do NOT check any of the initialization boxes (no README, no .gitignore, no license).
  • Click Create repository.

GitHub will show you a quick setup page with commands. You'll use those now.

  • Back in your terminal, connect your local repo to GitHub by running:
git remote add origin [[YOUR_REPO_URL="https://github.com/YOUR_USERNAME/mlops-churn-pipeline.git"]]

Where do I find my repo URL?

Copy the HTTPS URL from the quick setup page that GitHub just showed you. It looks like https://github.com/your-username/mlops-churn-pipeline.git.

  • Verify the remote is connected by running:
git remote -v
  • You should see two lines showing origin pointing to your GitHub repository URL (one for fetch, one for push).

Seeing 'remote origin already exists'?

This means a remote named origin was already added. Remove it first with git remote remove origin, then re-run the git remote add command.

Help me connect my local Git repo to GitHub.

Verify all tools are working

A quick set of version checks confirms that everything installed correctly and your virtual environment is active.

  • Run these version checks to confirm your setup:
python3 --version
pip show scikit-learn | grep Version
pip show mlflow | grep Version
pip show fastapi | grep Version
  • You should see output showing Python 3.11 or higher, scikit-learn at 1.8.0, mlflow at 3.12.0, and fastapi at 0.136.1.

Seeing wrong versions or 'not found'?

If pip show returns nothing, the package did not install. Make sure your (venv) is active and re-run pip install -r requirements.txt.

If python3 --version shows a version below 3.11, you need to install Python 3.11 or higher before continuing. scikit-learn 1.8.0 requires Python 3.11 as a minimum.

Help me verify my Python packages are installed correctly.

Your environment is locked and loaded. Next up, you'll train your first churn prediction model and track every experiment with MLflow.

Train Your Churn Model with MLflow Experiment Tracking

Your environment is ready. Now it's time to build the core of any MLOps pipeline: a trained model with full experiment tracking.

In a real ML team, you train dozens of model variations before finding one worth deploying. Without tracking, you lose which parameters produced which results. MLflow solves this by recording every experiment run, so you can compare models and promote the best one to production.

In this step, get ready to:
  • Load and preprocess the Telco Customer Churn dataset.
  • Train two models with full MLflow experiment tracking.
  • Compare experiments in the MLflow UI and register the best model.
Load the Telco Customer Churn dataset and preprocess it

Before training any model, you need clean, numerical data. The IBM Telco Customer Churn dataset contains a mix of categorical and numerical features that describe customer demographics, services, and billing.

  • Create a new file called train.py in your project folder.
  • Add the following imports and MLflow configuration at the top of train.py:
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, roc_auc_score
import mlflow
import mlflow.sklearn

# Use SQLite backend so Model Registry works
mlflow.set_tracking_uri("sqlite:///mlflow.db")

What does this code do?

The imports bring in pandas for data handling, scikit-learn for model training and evaluation, and mlflow for experiment tracking.

The set_tracking_uri("sqlite:///mlflow.db") line tells MLflow to store all experiment data in a local SQLite database. This is required because MLflow's Model Registry only works with a database-backed store.

  • Below the configuration, add the load_and_preprocess_data() function:

def load_and_preprocess_data():
    url = "https://raw.githubusercontent.com/IBM/telco-customer-churn-on-icp4d/master/data/Telco-Customer-Churn.csv"
    df = pd.read_csv(url)

    df["TotalCharges"] = pd.to_numeric(df["TotalCharges"], errors="coerce")
    df["TotalCharges"].fillna(0, inplace=True)

    df.drop("customerID", axis=1, inplace=True)

    le = LabelEncoder()
    categorical_cols = df.select_dtypes(include=["object"]).columns
    for col in categorical_cols:
        df[col] = le.fit_transform(df[col])

    X = df.drop("Churn", axis=1)
    y = df["Churn"]

    return train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)

What does this function do?

This function downloads the dataset directly from IBM's GitHub repository, so there's nothing to install or unzip.

The preprocessing handles three issues: TotalCharges has blank strings that need converting to numbers, customerID is irrelevant to predictions, and ML models need numbers instead of text labels.

The LabelEncoder converts every text column (like "Male"/"Female") into integers (0/1). Finally, the data is split 80/20 for training and testing, with stratify=y ensuring both sets have the same proportion of churned customers.

  • Save train.py.

Seeing a syntax error?

Make sure the load_and_preprocess_data() function is outside and below the set_tracking_uri line, not indented under it. Python functions need to start at the top indentation level.

help me fix a syntax error in my train.py file

Train multiple models with MLflow experiment tracking

Now you'll create the training loop that runs multiple model experiments. Each model gets its own MLflow run where parameters, metrics, and the trained model artifact are all logged automatically.

  • Below load_and_preprocess_data() in train.py, add the evaluate_model() helper function:

def evaluate_model(model, X_test, y_test):
    y_pred = model.predict(X_test)
    y_proba = model.predict_proba(X_test)[:, 1]

    metrics = {
        "accuracy": accuracy_score(y_test, y_pred),
        "precision": precision_score(y_test, y_pred),
        "recall": recall_score(y_test, y_pred),
        "f1": f1_score(y_test, y_pred),
        "roc_auc": roc_auc_score(y_test, y_proba),
    }
    return metrics

What does this function do?

This helper takes a trained model and test data, then calculates five classification metrics. y_pred gives hard predictions (0 or 1), while y_proba gives the probability of churn, which is needed for ROC-AUC.

Returning a dictionary makes it easy to log all five metrics to MLflow in one call.

  • Below evaluate_model(), add the main train_models() function and the entry point:

def train_models():
    X_train, X_test, y_train, y_test = load_and_preprocess_data()

    mlflow.set_experiment("churn-prediction")

    models = {
        "LogisticRegression": LogisticRegression(max_iter=1000, random_state=42),
        "RandomForest": RandomForestClassifier(n_estimators=100, random_state=42),
    }

    best_f1 = 0
    best_run_id = None

    for name, model in models.items():
        with mlflow.start_run(run_name=name):
            model.fit(X_train, y_train)
            metrics = evaluate_model(model, X_test, y_test)

            mlflow.log_params({"model_type": name})
            mlflow.log_metrics(metrics)
            mlflow.sklearn.log_model(model, artifact_path="model")

            print(f"{name}: F1={metrics['f1']:.4f}, ROC-AUC={metrics['roc_auc']:.4f}")

            if metrics["f1"] > best_f1:
                best_f1 = metrics["f1"]
                best_run_id = mlflow.active_run().info.run_id

    # Register the best model
    best_model_uri = f"runs:/{best_run_id}/model"
    mlflow.register_model(best_model_uri, "churn-prediction-model")
    print(f"\nBest model registered with run_id: {best_run_id}, F1: {best_f1:.4f}")


if __name__ == "__main__":
    train_models()

What does this code do?

The train_models() function is the heart of your training pipeline. It loads data, then loops through two models: Logistic Regression and Random Forest.

Each iteration opens an MLflow run with mlflow.start_run(). Inside that run, it trains the model, evaluates it, and logs everything: the model type as a parameter, all five metrics, and the serialized model artifact.

After both models run, it finds the one with the highest F1 score and registers it in the MLflow Model Registry under the name churn-prediction-model. This gives you a versioned, named model that other parts of your pipeline can load by name.

  • Save train.py.
  • Run the training script by executing this command in your terminal:
python train.py

You should see output showing both models' F1 and ROC-AUC scores, followed by a message confirming the best model was registered.

Seeing a connection error when loading data?

The script downloads the dataset from GitHub. Make sure you have an active internet connection. If you're behind a corporate firewall, try using a personal network.

help me fix a connection error when running train.py

Compare experiments in the MLflow UI

Your training data is now stored in the SQLite database. MLflow includes a web UI that lets you visually compare runs, inspect metrics, and browse registered models.

  • Launch the MLflow UI by running this command:
mlflow ui
  • Open http://localhost:5000 in your browser.
  • Click on the churn-prediction experiment in the left sidebar.
  • You should see two runs listed: LogisticRegression and RandomForest. Compare their F1 and ROC-AUC metrics.
  • Click Models in the top navigation to confirm churn-prediction-model is registered.
  • Press Ctrl+C in your terminal to stop the MLflow UI when you're done.

Why track experiments?

Without experiment tracking, you'd lose which hyperparameters produced which results after a few iterations. MLflow stores parameters, metrics, and model artifacts together so you can always reproduce or roll back to any previous run.

The Model Registry adds a promotion workflow on top. You can tag models as "staging" or "production" and load them by name instead of hunting for file paths.

MLflow UI not loading?

Make sure you ran python train.py first. The UI needs the mlflow.db file to exist before it can display experiments.

If port 5000 is already in use, stop the other process or run mlflow ui --port 5001 instead.

help me get the MLflow UI working

✔️ Awesome, I've got everything!

Great. Double check you've saved train.py and confirmed both runs appear in the MLflow UI.

ⓧ I'd like to double check the full code

Here is the complete train.py file:

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, roc_auc_score
import mlflow
import mlflow.sklearn

# Use SQLite backend so Model Registry works
mlflow.set_tracking_uri("sqlite:///mlflow.db")


def load_and_preprocess_data():
    url = "https://raw.githubusercontent.com/IBM/telco-customer-churn-on-icp4d/master/data/Telco-Customer-Churn.csv"
    df = pd.read_csv(url)

    df["TotalCharges"] = pd.to_numeric(df["TotalCharges"], errors="coerce")
    df["TotalCharges"].fillna(0, inplace=True)

    df.drop("customerID", axis=1, inplace=True)

    le = LabelEncoder()
    categorical_cols = df.select_dtypes(include=["object"]).columns
    for col in categorical_cols:
        df[col] = le.fit_transform(df[col])

    X = df.drop("Churn", axis=1)
    y = df["Churn"]

    return train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)


def evaluate_model(model, X_test, y_test):
    y_pred = model.predict(X_test)
    y_proba = model.predict_proba(X_test)[:, 1]

    metrics = {
        "accuracy": accuracy_score(y_test, y_pred),
        "precision": precision_score(y_test, y_pred),
        "recall": recall_score(y_test, y_pred),
        "f1": f1_score(y_test, y_pred),
        "roc_auc": roc_auc_score(y_test, y_proba),
    }
    return metrics


def train_models():
    X_train, X_test, y_train, y_test = load_and_preprocess_data()

    mlflow.set_experiment("churn-prediction")

    models = {
        "LogisticRegression": LogisticRegression(max_iter=1000, random_state=42),
        "RandomForest": RandomForestClassifier(n_estimators=100, random_state=42),
    }

    best_f1 = 0
    best_run_id = None

    for name, model in models.items():
        with mlflow.start_run(run_name=name):
            model.fit(X_train, y_train)
            metrics = evaluate_model(model, X_test, y_test)

            mlflow.log_params({"model_type": name})
            mlflow.log_metrics(metrics)
            mlflow.sklearn.log_model(model, artifact_path="model")

            print(f"{name}: F1={metrics['f1']:.4f}, ROC-AUC={metrics['roc_auc']:.4f}")

            if metrics["f1"] > best_f1:
                best_f1 = metrics["f1"]
                best_run_id = mlflow.active_run().info.run_id

    # Register the best model
    best_model_uri = f"runs:/{best_run_id}/model"
    mlflow.register_model(best_model_uri, "churn-prediction-model")
    print(f"\nBest model registered with run_id: {best_run_id}, F1: {best_f1:.4f}")


if __name__ == "__main__":
    train_models()

You now have a fully tracked training pipeline with a registered model. Next up, you'll build a REST API that loads this model and serves predictions in real time.

Build a Prediction API with FastAPI

Your churn model is trained and registered in MLflow. That's the research side of machine learning done.

Now you need to make that model accessible to other systems. In production, models serve predictions through REST APIs. In this step, you'll build a FastAPI application that loads your registered model and exposes prediction endpoints.

In this step, get ready to:
  • Create a FastAPI application that loads the churn model from MLflow on startup.
  • Define Pydantic request and response schemas for input validation.
  • Add health and predict endpoints, then test them locally with curl.
Create the FastAPI app and load your model

The API needs to load your trained model once when the server starts, then reuse it for every prediction request. FastAPI's startup event handler is perfect for this.

  • Create a new file called app.py in your project folder.
  • Add the following code to set up the FastAPI app and model loading:
from fastapi import FastAPI
from pydantic import BaseModel
import mlflow.sklearn
import pandas as pd
import os

app = FastAPI(title="Churn Prediction API", version="1.0.0")

# Configure MLflow tracking URI (must match training)
MLFLOW_TRACKING_URI = os.getenv("MLFLOW_TRACKING_URI", "sqlite:///mlflow.db")
mlflow.set_tracking_uri(MLFLOW_TRACKING_URI)

MODEL_URI = os.getenv("MODEL_URI", "models:/churn-prediction-model/latest")

model = None


@app.on_event("startup")
def load_model():
    global model
    model = mlflow.sklearn.load_model(MODEL_URI)

What does this code do?

The app reads MLFLOW_TRACKING_URI from the environment so it knows where your MLflow database lives. It defaults to sqlite:///mlflow.db which matches what train.py used.

The @app.on_event("startup") decorator tells FastAPI to run load_model() once when the server starts. This loads the registered model from MLflow into memory so every prediction request is fast.

  • Save app.py.
  • You won't see a change yet. The next chunk adds the request schema the endpoints need.
Define the request and response schemas

FastAPI uses Pydantic models for automatic input validation. If someone sends a request with missing fields or wrong types, FastAPI rejects it before your code ever runs.

  • In app.py, add the following classes below the load_model() function:
class PredictionRequest(BaseModel):
    gender: int
    SeniorCitizen: int
    Partner: int
    Dependents: int
    tenure: int
    PhoneService: int
    MultipleLines: int
    InternetService: int
    OnlineSecurity: int
    OnlineBackup: int
    DeviceProtection: int
    TechSupport: int
    StreamingTV: int
    StreamingMovies: int
    Contract: int
    PaperlessBilling: int
    PaymentMethod: int
    MonthlyCharges: float
    TotalCharges: float


class PredictionResponse(BaseModel):
    churn_prediction: int
    churn_probability: float

What does this code do?

PredictionRequest defines exactly what fields a prediction request must include. These 19 fields match the features your trained model expects (after encoding).

PredictionResponse defines the shape of the API's output. Callers get both the binary prediction (0 = stays, 1 = churns) and the probability score.

  • Save app.py.
  • You won't see a change yet. The next chunk adds the endpoints that complete the API.
Add the health and predict endpoints

Every production API needs a health check endpoint so monitoring tools can verify the service is running. The predict endpoint does the actual work of returning churn predictions.

  • In app.py, add the following endpoints below the PredictionResponse class:
@app.get("/health")
def health():
    return {"status": "healthy", "model_loaded": model is not None}


@app.post("/predict", response_model=PredictionResponse)
def predict(request: PredictionRequest):
    input_df = pd.DataFrame([request.model_dump()])
    prediction = model.predict(input_df)[0]
    probability = model.predict_proba(input_df)[0][1]

    return PredictionResponse(
        churn_prediction=int(prediction),
        churn_probability=float(probability),
    )

What does this code do?

GET /health returns the server status and whether the model loaded successfully. Monitoring tools and load balancers poll this endpoint.

POST /predict takes a validated PredictionRequest, converts it to a DataFrame, and runs the model. It returns both the binary prediction and the probability that the customer will churn.

  • Save app.py.
  • Start the API server by running this command:
uvicorn app:app --port 8000

You should see output indicating the server is running on http://127.0.0.1:8000.

FastAPI auto-generates API docs

Visit http://localhost:8000/docs while the server is running to see interactive API documentation. You can test endpoints directly from the browser.

  • Open a second terminal window. Make sure you're in your project folder with the virtual environment activated.
  • Test the health endpoint by running:
curl http://localhost:8000/health

You should see:

{"status":"healthy","model_loaded":true}
  • Test the predict endpoint by running:
curl -X POST http://localhost:8000/predict \
  -H "Content-Type: application/json" \
  -d '{"gender": 1, "SeniorCitizen": 0, "Partner": 1, "Dependents": 0, "tenure": 12, "PhoneService": 1, "MultipleLines": 0, "InternetService": 1, "OnlineSecurity": 0, "OnlineBackup": 1, "DeviceProtection": 0, "TechSupport": 0, "StreamingTV": 0, "StreamingMovies": 0, "Contract": 0, "PaperlessBilling": 1, "PaymentMethod": 2, "MonthlyCharges": 70.35, "TotalCharges": 844.2}'

You should see a JSON response with churn_prediction (0 or 1) and churn_probability (a decimal between 0 and 1).

What do these results mean?

A churn_prediction of 1 means the model predicts this customer will leave. A value of 0 means they'll likely stay.

The churn_probability shows how confident the model is. A value of 0.85 means 85% confidence the customer will churn.

  • Stop the uvicorn server by pressing Ctrl+C in the first terminal window.

Not getting a response from curl?

Make sure the uvicorn server is still running in your first terminal. You need two terminal windows open at the same time.

If you see a connection refused error, check that you're using port 8000 in both the server command and the curl URL.

If the predict endpoint returns a 422 error, make sure your JSON payload includes all 19 fields with the correct types (integers and floats).

Help me debug my FastAPI connection

✔️ Awesome, I've got everything!

Great. Double check you've saved app.py with all three sections (imports/config, schemas, endpoints).

ⓧ I'd like to double check the full code

from fastapi import FastAPI
from pydantic import BaseModel
import mlflow.sklearn
import pandas as pd
import os

app = FastAPI(title="Churn Prediction API", version="1.0.0")

# Configure MLflow tracking URI (must match training)
MLFLOW_TRACKING_URI = os.getenv("MLFLOW_TRACKING_URI", "sqlite:///mlflow.db")
mlflow.set_tracking_uri(MLFLOW_TRACKING_URI)

MODEL_URI = os.getenv("MODEL_URI", "models:/churn-prediction-model/latest")

model = None


@app.on_event("startup")
def load_model():
    global model
    model = mlflow.sklearn.load_model(MODEL_URI)


class PredictionRequest(BaseModel):
    gender: int
    SeniorCitizen: int
    Partner: int
    Dependents: int
    tenure: int
    PhoneService: int
    MultipleLines: int
    InternetService: int
    OnlineSecurity: int
    OnlineBackup: int
    DeviceProtection: int
    TechSupport: int
    StreamingTV: int
    StreamingMovies: int
    Contract: int
    PaperlessBilling: int
    PaymentMethod: int
    MonthlyCharges: float
    TotalCharges: float


class PredictionResponse(BaseModel):
    churn_prediction: int
    churn_probability: float


@app.get("/health")
def health():
    return {"status": "healthy", "model_loaded": model is not None}


@app.post("/predict", response_model=PredictionResponse)
def predict(request: PredictionRequest):
    input_df = pd.DataFrame([request.model_dump()])
    prediction = model.predict(input_df)[0]
    probability = model.predict_proba(input_df)[0][1]

    return PredictionResponse(
        churn_prediction=int(prediction),
        churn_probability=float(probability),
    )

Your prediction API is live and serving responses. Next up, you'll package it into a Docker container so it runs the same way anywhere.

Containerize Your API with Docker

Your API works locally when you run uvicorn directly. But deploying to any server means packaging your code, dependencies, model artifacts, and database into a single portable unit.

Docker solves this by wrapping everything into a container image that runs identically on any machine. In this step, you'll write a multi-stage Dockerfile, build the image, and verify predictions work through the container.

In this step, get ready to:
  • Write a multi-stage Dockerfile that installs dependencies and runs the FastAPI server.
  • Build and run the container locally, verifying predictions work through Docker.
  • Create a .dockerignore to keep the image lean.
Write a multi-stage Dockerfile

A multi-stage build uses two separate stages inside a single Dockerfile. The first stage installs all dependencies, and the second stage copies only the installed packages into a clean image.

This keeps the final image small because build tools and caches from the installation stage never make it into production.

  • Create a new file called Dockerfile in the root of your project folder (the same directory that contains app.py).
  • Paste the following into your Dockerfile:
FROM python:3.11-slim AS builder

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

FROM python:3.11-slim

WORKDIR /app

COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages
COPY --from=builder /usr/local/bin /usr/local/bin

COPY app.py .
COPY mlflow.db .
COPY mlartifacts ./mlartifacts

ENV MODEL_URI="models:/churn-prediction-model/latest"
ENV MLFLOW_TRACKING_URI="sqlite:///mlflow.db"

EXPOSE 8000

CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

What does this Dockerfile do?

  • Stage 1 (builder) starts from python:3.11-slim and installs all Python packages from requirements.txt. This stage exists purely to compile and cache packages.
  • Stage 2 (final) starts from a fresh python:3.11-slim image and copies only the installed packages from the builder. No pip caches, no build artefacts.
  • The final stage copies app.py, mlflow.db, and mlartifacts/ so the container has the trained model baked in.
  • ENV lines configure MLflow's tracking URI and model URI so the app loads the registered model on startup.
  • CMD starts uvicorn on port 8000, making the API accessible from outside the container.
  • Save Dockerfile.

Why bundle the model inside the image?

Copying mlflow.db and mlartifacts/ directly into the image means the container is fully self-contained. It can load and serve predictions without reaching out to an external model store.

In a production environment, you would typically pull models from a remote registry (like S3 or a managed MLflow server). For this project, baking them in keeps deployment simple and reproducible.

Build and run the container locally

With the Dockerfile in place, you can now build the image and run it to confirm predictions work through Docker.

  • Make sure you are not still running uvicorn from the previous step. If it is running, press Ctrl+C to stop it (port 8000 must be free).
  • Build the Docker image by running this command:
docker build -t churn-prediction-api:latest .

What does this command do?

The -t flag tags the image with the name churn-prediction-api:latest so you can reference it easily. The . tells Docker to use the current directory as the build context.

  • Wait for the build to finish. You should see a line containing Successfully tagged churn-prediction-api:latest or similar confirmation at the end.

Build failing?

  • Make sure Docker Desktop is running. You can check by running docker info in your terminal.
  • Confirm your Dockerfile is in the same directory as requirements.txt, app.py, and mlflow.db.
  • Check that the file is named exactly Dockerfile with a capital D and no file extension.

Help me debug my Docker build error

  • Start the container by running this command:
docker run -d --name churn-api -p 8000:8000 churn-prediction-api:latest

What does this command do?

  • -d runs the container in detached mode (in the background), freeing your terminal.
  • --name churn-api gives the container a friendly name so you can stop it easily later.
  • -p 8000:8000 maps port 8000 on your machine to port 8000 inside the container.
  • Verify the health endpoint works by running this command:
curl http://localhost:8000/health
  • You should see a JSON response containing {"status":"healthy","model_loaded":true}.
  • Test the prediction endpoint by running this command:
curl -X POST http://localhost:8000/predict \
  -H "Content-Type: application/json" \
  -d '{"gender": 1, "SeniorCitizen": 0, "Partner": 1, "Dependents": 0, "tenure": 12, "PhoneService": 1, "MultipleLines": 0, "InternetService": 1, "OnlineSecurity": 0, "OnlineBackup": 1, "DeviceProtection": 0, "TechSupport": 0, "StreamingTV": 0, "StreamingMovies": 0, "Contract": 0, "PaperlessBilling": 1, "PaymentMethod": 2, "MonthlyCharges": 70.35, "TotalCharges": 844.2}'
  • You should see a JSON response with churn_prediction (0 or 1) and churn_probability (a decimal between 0 and 1). Your model is now serving predictions from inside a Docker container.

Getting connection refused?

  • The container might still be starting up. Wait a few seconds and try again.
  • Check that the container is running by running docker ps. You should see churn-api listed.
  • If the container exited immediately, check its logs by running docker logs churn-api to see the error.

Help me debug my Docker container issue

  • Stop and remove the container now that you have verified it works by running this command:
docker stop churn-api && docker rm churn-api
Create a .dockerignore to keep the image lean

When Docker builds an image, it sends your entire project folder to the Docker daemon as the build context. Files like .git, venv/, and test files add bulk without contributing anything to the running container.

  • Create a new file called .dockerignore in the same directory as your Dockerfile.
  • Paste the following into .dockerignore:
__pycache__
*.pyc
.git
.github
venv
.env
*.egg-info
drift_report.html
tests/

What does this file do?

Each line tells Docker to skip that file or folder when building the image. This speeds up the build and keeps the image smaller.

  • __pycache__ and *.pyc are compiled bytecode that gets regenerated inside the container anyway.
  • venv is your local virtual environment. The container installs its own packages.
  • .env prevents secrets from accidentally leaking into the image.
  • tests/ are not needed at runtime. Tests run in CI, not inside the production container.
  • Save .dockerignore.
  • Verify the file is in your project by running:
ls -a
  • You should see .dockerignore and Dockerfile listed alongside your other project files. Future builds will now exclude all the files you listed.

Don't see .dockerignore in the listing?

  • Files starting with a dot are hidden by default. Make sure you used ls -a (the -a flag shows hidden files).
  • Check that the file name starts with a dot. It should be .dockerignore, not dockerignore.

Help me find my .dockerignore file

✔️ Awesome, I've got everything!

Great. Double check you have saved both Dockerfile and .dockerignore.

ⓧ I'd like to double check the full code

Here is the complete Dockerfile:

FROM python:3.11-slim AS builder

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

FROM python:3.11-slim

WORKDIR /app

COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages
COPY --from=builder /usr/local/bin /usr/local/bin

COPY app.py .
COPY mlflow.db .
COPY mlartifacts ./mlartifacts

ENV MODEL_URI="models:/churn-prediction-model/latest"
ENV MLFLOW_TRACKING_URI="sqlite:///mlflow.db"

EXPOSE 8000

CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

Here is the complete .dockerignore:

__pycache__
*.pyc
.git
.github
venv
.env
*.egg-info
drift_report.html
tests/

Your API is now containerized and portable. Anyone with Docker can pull this image and get a running prediction service in seconds. Next up, you'll write automated tests to make sure the model and API behave correctly before every deployment.

Add Automated Tests with pytest

You have a trained model, a serving API, and a Docker container. But how do you know the model still meets quality standards after a code change?

Automated tests catch regressions before they reach production. In this step, you'll write tests that verify both model quality thresholds and API behaviour using pytest.

In this step, get ready to:
  • Write unit tests for model loading and prediction accuracy thresholds.
  • Write integration tests for the FastAPI endpoints using TestClient.
  • Run the test suite locally and confirm all tests pass.
Write unit tests for model quality

A model quality test loads the registered model from MLflow and asserts that its performance metrics stay above minimum thresholds. If someone accidentally degrades the model, the test fails before anything ships.

  • Create the tests/ directory inside your project folder by running this command:
mkdir tests
  • Confirm the directory exists by running:
ls

You should see tests listed alongside your other project files like train.py and app.py.

  • Create a new file called test_model.py inside the tests/ directory and paste the following code:
import mlflow
import mlflow.sklearn
from train import load_and_preprocess_data, evaluate_model

mlflow.set_tracking_uri("sqlite:///mlflow.db")


def test_model_quality():
    _, X_test, _, y_test = load_and_preprocess_data()

    model = mlflow.sklearn.load_model("models:/churn-prediction-model/latest")
    metrics = evaluate_model(model, X_test, y_test)

    assert metrics["f1"] > 0.5, f"F1 score too low: {metrics['f1']}"
    assert metrics["roc_auc"] > 0.7, f"ROC-AUC too low: {metrics['roc_auc']}"

What does this code do?

This test reuses the load_and_preprocess_data() and evaluate_model() functions you already wrote in train.py. It loads the registered champion model from MLflow and evaluates it on the test split.

The two assert statements act as quality gates. If F1 drops below 0.5 or ROC-AUC drops below 0.7, the test fails with a clear error message showing the actual value.

  • Save tests/test_model.py.

Why these specific thresholds?

An F1 of 0.5 means the model is at least better than random guessing on the minority class. A ROC-AUC of 0.7 means the model can meaningfully distinguish churners from non-churners. These are deliberately conservative floors, not targets.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

import mlflow
import mlflow.sklearn
from train import load_and_preprocess_data, evaluate_model

mlflow.set_tracking_uri("sqlite:///mlflow.db")


def test_model_quality():
    _, X_test, _, y_test = load_and_preprocess_data()

    model = mlflow.sklearn.load_model("models:/churn-prediction-model/latest")
    metrics = evaluate_model(model, X_test, y_test)

    assert metrics["f1"] > 0.5, f"F1 score too low: {metrics['f1']}"
    assert metrics["roc_auc"] > 0.7, f"ROC-AUC too low: {metrics['roc_auc']}"
Write integration tests for the FastAPI endpoints

Unit tests check the model in isolation. Integration tests verify that the full API handles requests correctly, validates input, and returns properly structured responses.

  • Create a new file called test_api.py inside the tests/ directory.
  • Paste the following code for the imports, test client setup, and health endpoint test:
from fastapi.testclient import TestClient
from app import app

client = TestClient(app)


def test_health_endpoint():
    response = client.get("/health")
    assert response.status_code == 200
    data = response.json()
    assert data["status"] == "healthy"
    assert data["model_loaded"] is True

What does this code do?

FastAPI's TestClient simulates HTTP requests against your app without starting a real server. It triggers the startup event (which loads the model) and then lets you call endpoints directly in your test.

The health test confirms the API responds with a 200 status, reports itself as healthy, and has a loaded model.

  • Add the prediction and invalid input tests below the test_health_endpoint function:

def test_predict_endpoint():
    payload = {
        "gender": 1,
        "SeniorCitizen": 0,
        "Partner": 1,
        "Dependents": 0,
        "tenure": 12,
        "PhoneService": 1,
        "MultipleLines": 0,
        "InternetService": 1,
        "OnlineSecurity": 0,
        "OnlineBackup": 1,
        "DeviceProtection": 0,
        "TechSupport": 0,
        "StreamingTV": 0,
        "StreamingMovies": 0,
        "Contract": 0,
        "PaperlessBilling": 1,
        "PaymentMethod": 2,
        "MonthlyCharges": 70.35,
        "TotalCharges": 844.2,
    }
    response = client.post("/predict", json=payload)
    assert response.status_code == 200
    data = response.json()
    assert "churn_prediction" in data
    assert "churn_probability" in data
    assert data["churn_prediction"] in [0, 1]
    assert 0.0 <= data["churn_probability"] <= 1.0


def test_predict_invalid_input():
    response = client.post("/predict", json={"invalid": "data"})
    assert response.status_code == 422

What does this code do?

The test_predict_endpoint function sends a valid 19-field customer payload and confirms the response contains a binary prediction (0 or 1) and a probability between 0.0 and 1.0.

The test_predict_invalid_input function sends garbage data and asserts that Pydantic's validation returns a 422 status code. This confirms your API rejects malformed requests.

  • Save tests/test_api.py.

Seeing an import error for httpx?

FastAPI's TestClient requires httpx to be installed. This is already in your requirements.txt. If you see an import error, confirm you installed all dependencies by re-running pip install -r requirements.txt.

Help me fix my TestClient import error.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

from fastapi.testclient import TestClient
from app import app

client = TestClient(app)


def test_health_endpoint():
    response = client.get("/health")
    assert response.status_code == 200
    data = response.json()
    assert data["status"] == "healthy"
    assert data["model_loaded"] is True


def test_predict_endpoint():
    payload = {
        "gender": 1,
        "SeniorCitizen": 0,
        "Partner": 1,
        "Dependents": 0,
        "tenure": 12,
        "PhoneService": 1,
        "MultipleLines": 0,
        "InternetService": 1,
        "OnlineSecurity": 0,
        "OnlineBackup": 1,
        "DeviceProtection": 0,
        "TechSupport": 0,
        "StreamingTV": 0,
        "StreamingMovies": 0,
        "Contract": 0,
        "PaperlessBilling": 1,
        "PaymentMethod": 2,
        "MonthlyCharges": 70.35,
        "TotalCharges": 844.2,
    }
    response = client.post("/predict", json=payload)
    assert response.status_code == 200
    data = response.json()
    assert "churn_prediction" in data
    assert "churn_probability" in data
    assert data["churn_prediction"] in [0, 1]
    assert 0.0 <= data["churn_probability"] <= 1.0


def test_predict_invalid_input():
    response = client.post("/predict", json={"invalid": "data"})
    assert response.status_code == 422
Run the test suite locally

Both test files are ready. Time to run them and confirm everything passes.

  • Run the full test suite by running this command:
pytest tests/ -v

You should see output showing each test function with a PASSED label. All 4 tests should pass (1 from test_model.py and 3 from test_api.py).

Tests failing?

  • If test_model_quality fails, make sure you ran train.py in Step 2 and that mlflow.db exists in your project folder.
  • If test_health_endpoint or test_predict_endpoint fails with a model loading error, confirm your MLFLOW_TRACKING_URI environment variable is set to sqlite:///mlflow.db.
  • Make sure you're running pytest from your project's root folder (the one containing train.py and app.py), not from inside the tests/ directory.

Help me debug my failing pytest tests.

Your test suite is locked in. Every time you change the model or the API, these tests act as a safety net. Next up, you'll automate them with a CI/CD pipeline so they run on every push to GitHub.

Set Up CI/CD with GitHub Actions

Your tests pass locally, and your Docker image builds without issues. But what happens when you push new code next week?

Without automation, someone has to remember to run the tests, retrain the model, and rebuild the container every single time. GitHub Actions eliminates that manual step by running your entire pipeline automatically on every push.

In this step, get ready to:
  • Create a GitHub Actions workflow file that triggers on pushes to main.
  • Configure automated testing, model training, and Docker builds.
  • Push your code to GitHub and verify the pipeline runs.
Create the GitHub Actions workflow file

GitHub Actions workflows live in a special directory called .github/workflows/ inside your repository. Each YAML file in that directory defines a separate automated pipeline.

  • Create the workflow directory by running these commands:
mkdir -p .github/workflows
  • Confirm the directory exists by running:
ls .github/workflows

You should see an empty directory listed with no errors.

  • Create the workflow file ci.yml inside .github/workflows/ and paste in the following configuration:
name: MLOps CI Pipeline

on:
  push:
    branches: ["main"]
  pull_request:
    branches: ["main"]

permissions:
  contents: read

env:
  MLFLOW_TRACKING_URI: "sqlite:///mlflow.db"

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v6

      - name: Set up Python 3.11
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

      - name: Train model
        run: python train.py

      - name: Run tests
        run: pytest tests/ -v

      - name: Run drift detection
        run: python drift_detection.py

What does this configuration do?

The workflow header defines when and how the pipeline runs:

  • on: push/pull_request triggers the workflow whenever code is pushed to main or a pull request targets main.
  • permissions: contents: read follows the principle of least privilege. The workflow can read your repo but not write to it.
  • env: MLFLOW_TRACKING_URI sets the tracking URI for all jobs so MLflow uses the SQLite backend and the Model Registry works.
  • The test job checks out the code, installs Python 3.11, installs your packages from requirements.txt, trains the model fresh, runs your test suite, and runs drift detection.

Why train the model inside CI?

Your tests depend on a registered model in the MLflow Model Registry. Since CI starts from a clean environment, the model must be trained first so the registry is populated before pytest runs.

Add the Docker build job

The test job validates your code. Now you need a second job that builds the production container only after tests pass.

  • In ci.yml, add the following docker job below the test job (at the same indentation level as test:, inside jobs:):
  docker:
    runs-on: ubuntu-latest
    needs: test

    steps:
      - uses: actions/checkout@v6

      - name: Set up Python 3.11
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install dependencies and train model
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
          python train.py
        env:
          MLFLOW_TRACKING_URI: "sqlite:///mlflow.db"

      - name: Build Docker image
        run: docker build -t churn-prediction-api:latest .

What does the Docker job do?

  • needs: test means this job only runs after the test job succeeds. If tests fail, the Docker image is never built.
  • The job trains the model again so that mlflow.db and mlartifacts/ exist before the Dockerfile copies them into the container.
  • docker build -t churn-prediction-api:latest . builds the same image you tested locally, proving it builds in a clean environment.
  • Save .github/workflows/ci.yml.

✔️ Awesome, I've got everything!

Great. Double check you have saved the file before moving on.

ⓧ I'd like to double check the full code

name: MLOps CI Pipeline

on:
  push:
    branches: ["main"]
  pull_request:
    branches: ["main"]

permissions:
  contents: read

env:
  MLFLOW_TRACKING_URI: "sqlite:///mlflow.db"

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v6

      - name: Set up Python 3.11
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

      - name: Train model
        run: python train.py

      - name: Run tests
        run: pytest tests/ -v

      - name: Run drift detection
        run: python drift_detection.py

  docker:
    runs-on: ubuntu-latest
    needs: test

    steps:
      - uses: actions/checkout@v6

      - name: Set up Python 3.11
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install dependencies and train model
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
          python train.py
        env:
          MLFLOW_TRACKING_URI: "sqlite:///mlflow.db"

      - name: Build Docker image
        run: docker build -t churn-prediction-api:latest .

YAML indentation errors?

YAML is whitespace-sensitive. Make sure the docker: key is at the same indentation level as test: (two spaces under jobs:). Each steps: entry uses a - prefix at six spaces of indentation.

help me fix YAML indentation in my GitHub Actions workflow

Push your code and verify the pipeline

Your workflow is ready. Pushing the code to GitHub triggers the pipeline automatically because the workflow listens for push events on main.

  • Stage all your files and push to GitHub by running these commands:
git add .
git commit -m "Add CI/CD pipeline with GitHub Actions"
git push origin main

What triggers the workflow?

Pushing to main fires a push event. GitHub detects the .github/workflows/ci.yml file and starts executing the workflow immediately.

  • Open your repository on GitHub in your browser.
  • Click the Actions tab at the top of the repository page.
  • Click on the workflow run named Add CI/CD pipeline with GitHub Actions (your commit message).
  • Watch the test job execute each step. Once it passes, the docker job starts automatically.

What should I see?

Each job shows a green checkmark when it completes successfully. The test job runs first (training, tests, drift detection). The docker job runs second (installs, trains, builds the image). Both jobs finishing green means your full pipeline is working.

The pipeline takes a few minutes to complete because it installs all dependencies and trains the model from scratch.

Pipeline failing on the drift detection step?

If you have not yet created drift_detection.py (which you will build in the next step), the Run drift detection step will fail. This is expected. You will fix this in Step 7 when you create the drift detection script.

help me troubleshoot my GitHub Actions pipeline failure

Seeing 'permission denied' on push?

Make sure your GitHub remote is configured correctly. Run git remote -v to check. If you are using HTTPS, GitHub may prompt for authentication. If using SSH, ensure your SSH key is added to your GitHub account.

help me fix git push authentication issues

Your CI/CD pipeline is live. Every push to main now automatically trains, tests, and builds your containerized API. Next up, you will add drift detection to catch silent model degradation before it reaches production.

Detect Data Drift with Evidently AI

Your CI/CD pipeline tests the model and builds the container on every push. But there is a silent failure mode that tests alone cannot catch: the real-world data changing shape after deployment.

This is called data drift. If customers start behaving differently than your training data expected, your model's predictions quietly degrade. In this step, you will build a drift detection script using Evidently AI that catches this before it reaches production.

In this step, get ready to:
  • Create a drift detection script that simulates production data and compares it against training data.
  • Generate an HTML drift report showing which features have shifted.
  • Verify drift detection runs automatically in your CI pipeline.
Create the drift simulation script

In production, you would compare your training data against real incoming requests. For this project, you will simulate drift by artificially shifting values in the test set to mimic what happens when customer behaviour changes over time.

  • Create a new file called drift_detection.py in your project root (the same folder as train.py).
  • Add the imports and the drift simulation function by pasting this code:
import pandas as pd
import numpy as np
from evidently import Report
from evidently.presets import DataDriftPreset
from evidently.core import DataDefinition
from evidently import Dataset
from train import load_and_preprocess_data


def simulate_production_data(X_reference, drift_fraction=0.3, seed=42):
    """Simulate production data with some drift."""
    rng = np.random.default_rng(seed)
    X_production = X_reference.copy()
    n_drift = int(len(X_production) * drift_fraction)

    # Simulate drift in MonthlyCharges (increase by 20-50%)
    drift_indices = rng.choice(len(X_production), n_drift, replace=False)
    X_production.loc[X_production.index[drift_indices], "MonthlyCharges"] *= rng.uniform(1.2, 1.5, n_drift)

    # Simulate drift in tenure (new customers)
    X_production.loc[X_production.index[drift_indices], "tenure"] = rng.integers(0, 6, n_drift)

    return X_production

What does this code do?

The imports pull in Evidently's Report, DataDriftPreset, Dataset, and DataDefinition. These are the building blocks Evidently 0.7.21 requires for drift analysis.

The simulate_production_data function takes a reference DataFrame and introduces realistic drift: it bumps MonthlyCharges up by 20-50% and sets tenure to very low values (simulating an influx of new customers). A fixed seed of 42 makes results deterministic so CI runs are consistent.

Why simulate drift instead of using real production data?

In a real deployment, you would log incoming prediction requests and compare them against training data on a schedule. For this project, simulated drift lets you prove the detection pipeline works end-to-end without needing a live production system.

Generate the drift report

The simulation function creates drifted data. Now you need the detection logic that compares it against the training distribution and produces a report.

  • In drift_detection.py, add the run_drift_detection function and main block below the simulate_production_data function:
def run_drift_detection():
    X_train, X_test, _, _ = load_and_preprocess_data()

    X_production = simulate_production_data(X_test)

    # Create Evidently Dataset objects (required in 0.7+)
    reference_dataset = Dataset.from_pandas(
        pd.DataFrame(X_train),
        data_definition=DataDefinition()
    )
    current_dataset = Dataset.from_pandas(
        pd.DataFrame(X_production),
        data_definition=DataDefinition()
    )

    report = Report([DataDriftPreset()])
    my_eval = report.run(current_dataset, reference_dataset)

    my_eval.save_html("drift_report.html")
    print("Drift report saved to drift_report.html")

    # Get results as dictionary and inspect drift metrics
    results = my_eval.dict()

    # Use DriftedColumnsCount metric results
    # Look for the share of drifted columns in the results
    drifted_count_metric = None
    for metric_result in results.get("metrics", []):
        if "drifted_columns_count" in str(metric_result).lower() or "share" in str(metric_result.get("metric", "")):
            drifted_count_metric = metric_result
            break

    # Fallback: print full results for debugging and use a simple heuristic
    if drifted_count_metric is None:
        print("Full results keys for debugging:")
        print(results.keys() if isinstance(results, dict) else type(results))
        print("Drift detection completed. Check drift_report.html for details.")
        return True

    drift_share = drifted_count_metric.get("result", {}).get("share", 0)
    n_drifted = drifted_count_metric.get("result", {}).get("count", 0)
    print(f"Drifted columns: {n_drifted} (share: {drift_share:.1%})")

    if drift_share > 0.5:
        print("WARNING: Significant data drift detected!")
        return False
    return True


if __name__ == "__main__":
    is_stable = run_drift_detection()
    if not is_stable:
        exit(1)

What does this code do?

  • Dataset.from_pandas() wraps your DataFrames into Evidently Dataset objects with a DataDefinition(). This is required in Evidently 0.7+.
  • Report([DataDriftPreset()]) creates a report configured to detect statistical drift across all features using sensible defaults.
  • report.run(current_dataset, reference_dataset) runs the drift analysis, comparing simulated production data against training data.
  • The script searches the results dictionary for the DriftedColumnsCount metric. If more than 50% of columns have drifted, it returns False and exits with code 1 (signalling failure to CI).
  • Save drift_detection.py.
  • Run the drift detection script locally by running this command:
python drift_detection.py

You should see output showing the drifted column count and a message confirming the report was saved to drift_report.html.

  • Open drift_report.html in your browser to see the full interactive drift report.

What should I see in the drift report?

The HTML report shows a per-feature breakdown with statistical tests for each column. You should see MonthlyCharges and tenure flagged as drifted (since those are the ones you artificially shifted). Other features should remain stable.

Seeing an import error?

If you see an error about Dataset or DataDefinition not being found, make sure your virtual environment is active and you have evidently==0.7.21 installed. Older versions of Evidently used a different API.

help me debug an Evidently import error in my drift detection script

✔️ Awesome, I've got everything!

Great. Double check you have saved drift_detection.py and confirmed the report generates successfully.

ⓧ I'd like to double check the full code

import pandas as pd
import numpy as np
from evidently import Report
from evidently.presets import DataDriftPreset
from evidently.core import DataDefinition
from evidently import Dataset
from train import load_and_preprocess_data


def simulate_production_data(X_reference, drift_fraction=0.3, seed=42):
    """Simulate production data with some drift."""
    rng = np.random.default_rng(seed)
    X_production = X_reference.copy()
    n_drift = int(len(X_production) * drift_fraction)

    # Simulate drift in MonthlyCharges (increase by 20-50%)
    drift_indices = rng.choice(len(X_production), n_drift, replace=False)
    X_production.loc[X_production.index[drift_indices], "MonthlyCharges"] *= rng.uniform(1.2, 1.5, n_drift)

    # Simulate drift in tenure (new customers)
    X_production.loc[X_production.index[drift_indices], "tenure"] = rng.integers(0, 6, n_drift)

    return X_production


def run_drift_detection():
    X_train, X_test, _, _ = load_and_preprocess_data()

    X_production = simulate_production_data(X_test)

    # Create Evidently Dataset objects (required in 0.7+)
    reference_dataset = Dataset.from_pandas(
        pd.DataFrame(X_train),
        data_definition=DataDefinition()
    )
    current_dataset = Dataset.from_pandas(
        pd.DataFrame(X_production),
        data_definition=DataDefinition()
    )

    report = Report([DataDriftPreset()])
    my_eval = report.run(current_dataset, reference_dataset)

    my_eval.save_html("drift_report.html")
    print("Drift report saved to drift_report.html")

    # Get results as dictionary and inspect drift metrics
    results = my_eval.dict()

    # Use DriftedColumnsCount metric results
    # Look for the share of drifted columns in the results
    drifted_count_metric = None
    for metric_result in results.get("metrics", []):
        if "drifted_columns_count" in str(metric_result).lower() or "share" in str(metric_result.get("metric", "")):
            drifted_count_metric = metric_result
            break

    # Fallback: print full results for debugging and use a simple heuristic
    if drifted_count_metric is None:
        print("Full results keys for debugging:")
        print(results.keys() if isinstance(results, dict) else type(results))
        print("Drift detection completed. Check drift_report.html for details.")
        return True

    drift_share = drifted_count_metric.get("result", {}).get("share", 0)
    n_drifted = drifted_count_metric.get("result", {}).get("count", 0)
    print(f"Drifted columns: {n_drifted} (share: {drift_share:.1%})")

    if drift_share > 0.5:
        print("WARNING: Significant data drift detected!")
        return False
    return True


if __name__ == "__main__":
    is_stable = run_drift_detection()
    if not is_stable:
        exit(1)
Integrate drift detection into your CI pipeline

Your GitHub Actions workflow already includes a drift detection step. When you set up CI in the previous step, the workflow file included a "Run drift detection" step that executes python drift_detection.py after the tests pass.

  • Open .github/workflows/ci.yml and confirm you can see this step inside the test job:
      - name: Run drift detection
        run: python drift_detection.py

Why drift detection belongs in CI

By running drift detection as a CI step, your pipeline will fail if the data distribution has shifted too much. This catches silent model degradation before a broken model ever gets packaged into a Docker image.

  • Push your new drift detection script to GitHub by running these commands:
git add drift_detection.py
git commit -m "Add drift detection with Evidently AI"
git push origin main
  • Open your repository on GitHub and click the **Actions** tab.
  • Click into the latest workflow run and expand the **Run drift detection** step inside the test job.

You should see the drift summary output and a green checkmark confirming the pipeline passed.

Pipeline failing on the drift detection step?

Make sure drift_detection.py is committed at the root of your repository (same level as train.py). The CI workflow runs python drift_detection.py from the repo root, so the file must be there.

Also verify that evidently==0.7.21 is in your requirements.txt. The CI installs from that file before running the script.

help me debug my GitHub Actions drift detection step

Your pipeline now has three layers of protection: unit and integration tests, model quality checks, and data drift detection. Next up, you will tackle the Secret Mission and add an automated quality gate that blocks deployment if a new model performs worse than the current champion.

Secret mission

Add a Model Quality Gate

Your pipeline runs tests and checks for drift. But what happens if a new model is technically valid yet performs worse than the current champion? Build an automated quality gate that blocks deployment when the candidate model fails to beat the champion on key metrics.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing cloud costs. Decide whether to keep your resources for your portfolio, pause Docker to free up system resources, or delete everything entirely.

Resources you used:

  • Project folder on your Desktop (contains all code, mlflow.db, mlartifacts/, and your virtual environment).
  • Docker image (churn-prediction-api:latest).
  • GitHub repository with your MLOps pipeline code.

Keep everything running

No action needed. Choose this if you're still actively building or want to keep the project as a portfolio piece.

  • Your MLflow tracking data (mlflow.db and mlartifacts/) stays in your project folder, ready for the MLflow UI anytime.
  • Your Docker image remains available locally for running the prediction API.
  • Your GitHub repository stays public, showcasing your MLOps skills to potential employers.

Pause - I'll come back to this later

Shut down running processes to free up system memory and CPU. Your files and data stay intact.

  • Stop any running Docker containers by running:
docker stop $(docker ps -q)
  • Stop any running uvicorn server by pressing Ctrl+C in the terminal where it's running.
  • Quit Docker Desktop to free up background resources.
  • Your project folder, MLflow data, and GitHub repository remain untouched for when you return.

Delete - I don't want to use this again

Remove all project resources and start fresh. This is irreversible.

  • Delete your project folder (which includes your virtual environment, MLflow database, and all artifacts) by running:

macOS

rm -rf ~/Desktop/mlops-churn-prediction

Windows

rmdir /s /q %USERPROFILE%\Desktop\mlops-churn-prediction
  • Remove the Docker image by running:
docker image rm churn-prediction-api:latest
  • Reclaim any additional Docker disk space by running:
docker image prune
  • Delete your GitHub repository by navigating to your repository page on GitHub.
  • Click Settings in the top navigation bar.
  • Scroll down to the Danger Zone section.
  • Click Delete this repository and confirm by typing the repository name when prompted.

Nice Work!

Nice Work!

Congratulations! You just built a complete production pipeline from model training to monitored deployment. This is the full MLOps lifecycle that most ML teams struggle to achieve.

You've learned how to:

  • Trained a churn prediction model with full experiment tracking and model registry using MLflow.
  • Built and containerized a production REST API with input validation using FastAPI and Docker.
  • Automated testing, builds, and data drift monitoring with GitHub Actions, pytest, and Evidently AI.
  • Added an automated model quality gate that blocks bad models from reaching production.

Ready to quiz yourself?