Track Customer Churn Experiments

Compare, track, and reload local churn models with MLflow.

Introduction

30 Second Summary

A result can look convincing today yet become impossible to repeat next week. Missing records leave you guessing which choices produced it.

In this project, you will build a local customer churn experiment lab using Python with MLflow. The finished workflow trains two models before reloading the strongest one for a fresh prediction.

What You'll Build

Picture running a single script that fills a browser dashboard with two competing churn models before the winner predicts whether a held-out customer will stay or churn.

By the end of this project, you'll have:

  • A reproducible comparison that trains logistic regression beside a random forest on the same churn data. The terminal prints accuracy plus F1 scores for both candidates.
  • A local MLflow dashboard where both runs remain available after the terminal closes. Each run preserves its settings beside the scores that support your comparison.
  • A champion model that reloads from a saved model artifact. The script prints a fresh stay or churn prediction for one held-out customer.
  • Secret Mission: Replace manual parameter guessing with a tracked four-combination GridSearchCV sweep.

Are there any prerequisites?

You need basic Python knowledge. Familiarity with machine learning helps you understand the model comparison. A Windows computer needs internet access for the one-time package installation.

Before We Start

Let’s lock in the goal before the hands-on work begins. You’re defining what the model predicts and why preserving experiment history makes each comparison reproducible.

Set Up the Local ML Lab

Your churn lab depends on Python packages that must work together. A version-pinned virtual environment keeps those packages isolated from other projects.

You will prepare the project in Visual Studio Code. You will install MLflow and scikit-learn at the exact versions used throughout this project.

In this step, get ready to:
  • Verify that Python 3.11 or newer is available.
  • Create the project files inside customer-churn-mlops.
  • Activate an isolated environment with the pinned MLflow and scikit-learn packages.
Verify Python and create the project folder

Scikit-learn 1.9.1 requires Python 3.11 or newer. Checking the interpreter first prevents an unclear installation failure later.

  • Press the Windows key to open Windows search.
  • Type Visual Studio Code and press Enter.
  • Click Terminal in the top menu.
  • Select New Terminal.
  • Click the profile arrow beside the plus button in the terminal panel.
  • Select Command Prompt.
  • Check your installed Python version by running:
python --version

What does this command do?

The --version option prints the Python interpreter version used by this Command Prompt. The result determines whether your environment can install the pinned packages.

✔️ I see version 3.11 or higher

Your Python interpreter is ready. You have cleared the compatibility check for both pinned packages.

ⓧ I see an older version

The installed interpreter is below the version required by scikit-learn 1.9.1. Install a supported 64-bit Python release before creating the environment.

  • Visit the official Python releases for Windows page.
  • Download a Windows installer (64-bit) for Python 3.11 or newer.
  • Run the downloaded installer.
  • Select Add Python to PATH on the installer screen.
  • Complete the installation.
  • Restart Visual Studio Code so it reads the updated PATH.
  • Open a new Command Prompt terminal from the Terminal menu.
  • Check the updated version by running:
python --version

What should I see?

The command should now report Python 3.11 or newer. If it still reports the older interpreter, rerun the installer and enable Add Python to PATH.

ⓧ Command not found

Windows cannot currently locate Python from Command Prompt. Install a supported 64-bit release with command-line access enabled.

  • Visit the official Python releases for Windows page.
  • Download a Windows installer (64-bit) for Python 3.11 or newer.
  • Run the downloaded installer.
  • Select Add Python to PATH on the installer screen.
  • Complete the installation.
  • Restart Visual Studio Code so it reads the updated PATH.
  • Open a new Command Prompt terminal from the Terminal menu.
  • Confirm that Command Prompt can find Python by running:
python --version

What should I see?

The command should print Python 3.11 or newer. That output confirms the installer added Python to the command path.

That compatibility check is complete. Your interpreter can now support the local ML lab.

  • Click File in the Visual Studio Code menu.
  • Select Open Folder.
  • Select your Desktop in the folder dialog.
  • Click the New Folder button.
  • Type customer-churn-mlops and press Enter.
  • Select the new customer-churn-mlops folder.
  • Click Select Folder.
  • Confirm that you trust the folder if Visual Studio Code displays a workspace trust prompt.

You should see customer-churn-mlops at the top of the Explorer sidebar. This folder now anchors every local project resource.

Create the project files and virtual environment

The dependency file makes the package versions reproducible. The ignore file keeps generated environments and experiment data out of Git history.

  • Hover over customer-churn-mlops in the Explorer sidebar.
  • Click the new file icon.
  • Name the file requirements.txt and press Enter.
  • Define the pinned dependencies by pasting this content into requirements.txt:
mlflow==3.16.0
scikit-learn==1.9.1

What does this file control?

  • The mlflow==3.16.0 line pins the experiment tracking package.
  • The scikit-learn==1.9.1 line pins the machine learning package.
  • The exact pins give each fresh environment the same dependency foundation.
  • Save requirements.txt by pressing Ctrl+S.

The unsaved dot on the requirements.txt tab should disappear. The saved file remains visible in the Explorer sidebar.

Dependency file looks different?

Check that the file is named requirements.txt with no extra file extension. Confirm that each dependency appears on its own line.

Help me check my dependency file.

  • Hover over customer-churn-mlops in the Explorer sidebar.
  • Click the new file icon.
  • Name the file .gitignore and press Enter.
  • Define the generated resources Git should ignore by pasting this content into .gitignore:
.venv/
__pycache__/
*.pyc
mlflow.db
mlartifacts/

What does this file exclude?

  • The .venv/ entry excludes the local virtual environment.
  • The __pycache__/ entry excludes Python cache folders.
  • The *.pyc entry excludes compiled Python files.
  • The mlflow.db entry excludes the local experiment database.
  • The mlartifacts/ entry excludes logged model artifacts.
  • Save .gitignore by pressing Ctrl+S.

The unsaved dot on the .gitignore tab should disappear. Both project files should now appear in the Explorer sidebar.

Missing the ignore file?

Confirm that the filename begins with a dot. Check that Visual Studio Code did not append another file extension.

Help me find my ignore file.

✔️ Awesome, I've got everything!

Your dependency pins and ignore rules are saved. Both files are ready for the isolated environment.

ⓧ I'd like to double check the full code

Compare your complete requirements.txt file with this reference:

mlflow==3.16.0
scikit-learn==1.9.1

What should match?

Your file should contain these two dependency pins in this order. Each pin should be on its own line.

Compare your complete .gitignore file with this reference:

.venv/
__pycache__/
*.pyc
mlflow.db
mlartifacts/

What should match?

Your file should contain all five ignore patterns in this order. These entries cover the generated local resources used by the project.

A project-local environment gives customer-churn-mlops its own package installation. This boundary prevents another Python project from changing the versions used here.

  • Open a new Command Prompt terminal from the Terminal menu.
  • Create the .venv environment by running:
python -m venv .venv

What does this command do?

The venv module creates an isolated Python environment inside .venv. The environment contains its own package installation location.

  • Confirm that the Explorer sidebar now lists .venv.
  • Activate the environment by running:
.venv\Scripts\activate.bat

What does activation change?

Activation points the current Command Prompt session at the interpreter inside .venv. Package installation commands now target this project environment.

The terminal prompt should now begin with the virtual environment name. That prompt confirms the environment is active.

Environment not activating?

  • Confirm that your terminal profile says Command Prompt.
  • Confirm that .venv appears inside customer-churn-mlops.
  • Confirm that the activation command uses backslashes.

Help me activate my virtual environment.

Install and verify the pinned packages

The active environment is currently empty of the project dependencies. Installing from requirements.txt turns the saved pins into the packages used by every later run.

  • Install the pinned project dependencies by running:
python -m pip install -r requirements.txt

What does this command do?

The command asks Python to run pip inside the active environment. The -r option tells it to install every dependency listed in requirements.txt.

This installation can take several minutes because MLflow brings in supporting packages. The scrolling download and installation messages show that the process is still working.

When the installation finishes, Command Prompt returns to the active environment prompt. Your local lab now has both core packages available.

Package installation failed?

  • Confirm that the terminal still shows the active virtual environment.
  • Confirm that requirements.txt is saved inside customer-churn-mlops.
  • Check that your internet connection can download Python packages.

Help me troubleshoot the installation.

If the installation still fails, share the non-sensitive terminal output in the NextWork community for environment-specific help.

Before you check, which two pinned package lines do you expect to find in the installed package list?

  • Verify the installed packages by running:
python -m pip freeze

What does this command show?

The freeze command prints the packages installed in the active environment. It includes direct dependencies and supporting packages.

You should see many package lines. Confirm that the output includes mlflow==3.16.0 and scikit-learn==1.9.1.

Pinned packages missing?

  • Confirm that the current terminal still uses the activated environment.
  • Compare requirements.txt with the double-check tab above.
  • Rerun the installation command after correcting the file.

Help me verify my installed packages.

Your local ML lab now has an isolated interpreter and reproducible dependency pins. Next, you will train two churn models and discover what terminal-only results fail to preserve.

Train Baseline Models

Your isolated environment is ready. Now you can turn that setup into a repeatable machine learning baseline.

A quick experiment often ends with scores printed to a terminal. You will build that terminal-only version first so its limits become visible.

In this step, get ready to:
  • Generate reproducible synthetic churn data with a stratified train/test split.
  • Define a scaled logistic regression pipeline plus a random forest candidate.
  • Fit both candidates before printing accuracy plus binary F1 scores.
Generate reproducible churn data

Synthetic classification data gives both models the same controlled churn problem. A fixed random state makes each run reproducible.

  • In the Explorer sidebar, hover over the customer-churn-mlops folder.
  • Click the new file icon beside the folder name.
  • Enter train.py as the file name.
  • Press Enter to create the file.
  • Add the data imports plus reproducible dataset by pasting this code into train.py:
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, f1_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler


def main():
    X, y = make_classification(
        n_samples=1000,
        n_features=8,
        n_informative=5,
        n_redundant=1,
        weights=[0.7, 0.3],
        class_sep=1.0,
        random_state=42,
    )


if __name__ == "__main__":
    main()

What does this code do?

  • The imports provide the dataset generator, candidate models, evaluation metrics, splitting helper, pipeline helper, and scaler.
  • The make_classification() call creates 1,000 synthetic customer records with eight features.
  • The weights value gives class 0 more examples than class 1. This resembles a churn problem where fewer customers leave.
  • The random_state value keeps the generated X features plus y labels reproducible.
  • Save train.py by pressing Ctrl+S.
  • Check that the dataset code executes by running this command in the activated terminal:
python train.py

What should I see?

You should return to the active command prompt without a traceback. This confirms that Python can import the installed packages and generate the synthetic dataset.

Your first runnable slice is working. The same synthetic customer records can now anchor every model comparison.

Script not returning to the prompt?

Confirm that train.py is saved inside the customer-churn-mlops folder. Check that the terminal prompt still shows the activated .venv environment.

Compare the parentheses plus indentation in your dataset block with the code above.

Help me debug the first version of train.py.

Define the candidate models

A fair comparison starts with two candidates trained on identical data. The preprocessing pipeline scales inputs for logistic regression while keeping the transformation attached to the model.

  • In train.py, find the line containing def main():.
  • Place this candidate factory directly above def main()::
def create_candidates():
    return [
        (
            "logistic-regression",
            make_pipeline(
                StandardScaler(),
                LogisticRegression(max_iter=1000),
            ),
        ),
        (
            "random-forest",
            RandomForestClassifier(
                n_estimators=200,
                max_depth=5,
                random_state=42,
            ),
        ),
    ]

How are the candidates different?

  • The logistic-regression candidate scales every feature before fitting the classifier.
  • The random-forest candidate uses 200 trees with a maximum depth of 5.
  • The fixed random state keeps the random forest reproducible across repeated runs.
  • The names in each tuple give the terminal output a stable label for each candidate.
  • Save train.py by pressing Ctrl+S.
  • Confirm that Python accepts the candidate definition by running:
python train.py

What should I see?

You should return to the active command prompt without a traceback. This confirms that both candidate model definitions are valid.

Both candidates are now defined. They are ready to receive the same training records.

Seeing a syntax problem?

Check that create_candidates() starts at the left edge of the file. Confirm that every tuple plus model constructor closes before the final list bracket.

Help me fix the candidate model definition.

Train and compare the models

A stratified train/test split preserves the class balance across the training data plus held-out test data. Both models then face the same evaluation set.

  • In main(), find the closing parenthesis for the make_classification() block.
  • Place this split directly below that block:
    X_train, X_test, y_train, y_test = train_test_split(
        X,
        y,
        test_size=0.2,
        random_state=42,
        stratify=y,
    )

What does the split control?

  • The test_size value reserves 20 percent of the records for evaluation.
  • The fixed random state reproduces the same split on every run.
  • The stratify argument preserves the class proportions in both subsets.
  • Place this training loop directly below the train_test_split() block:
    for run_name, model in create_candidates():
        model.fit(X_train, y_train)
        predictions = model.predict(X_test)
        accuracy = accuracy_score(y_test, predictions)
        f1 = f1_score(y_test, predictions, average="binary")
        print(f"{run_name}: accuracy={accuracy:.3f}, f1={f1:.3f}")

How does the comparison work?

  • The loop fits each candidate on the same X_train features plus y_train labels.
  • Each fitted model predicts labels for the untouched X_test records.
  • Accuracy measures the share of all predictions that are correct.
  • F1 score balances precision with recall for the binary churn class.
  • Save train.py by pressing Ctrl+S.

✔️ Awesome, I've got everything!

Your baseline script is ready. Double check that you saved train.py before running it.

ⓧ I'd like to double check the full code

Compare your complete train.py file with this reference:

from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, f1_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler


def create_candidates():
    return [
        (
            "logistic-regression",
            make_pipeline(
                StandardScaler(),
                LogisticRegression(max_iter=1000),
            ),
        ),
        (
            "random-forest",
            RandomForestClassifier(
                n_estimators=200,
                max_depth=5,
                random_state=42,
            ),
        ),
    ]


def main():
    X, y = make_classification(
        n_samples=1000,
        n_features=8,
        n_informative=5,
        n_redundant=1,
        weights=[0.7, 0.3],
        class_sep=1.0,
        random_state=42,
    )

    X_train, X_test, y_train, y_test = train_test_split(
        X,
        y,
        test_size=0.2,
        random_state=42,
        stratify=y,
    )

    for run_name, model in create_candidates():
        model.fit(X_train, y_train)
        predictions = model.predict(X_test)
        accuracy = accuracy_score(y_test, predictions)
        f1 = f1_score(y_test, predictions, average="binary")
        print(f"{run_name}: accuracy={accuracy:.3f}, f1={f1:.3f}")


if __name__ == "__main__":
    main()

Before you run the finished script, which candidate do you expect to have the higher F1 score?

  • Train both baseline models by running:
python train.py

What should I see?

You should see one metric line labeled logistic-regression. You should see a second metric line labeled random-forest.

Each line contains accuracy plus F1 rounded to three decimal places. The command finishes by returning you to the active prompt.

The comparison has no structured experiment history or reusable model artifact. This intended shortfall shows why terminal output becomes difficult to manage across repeated experiments.

You have your first visible model comparison. Both churn candidates now train against the same reproducible split.

Missing one of the metric lines?

Check that the training loop stays inside main() with four spaces of indentation. Confirm that the loop calls create_candidates() with parentheses.

If model fitting stops, compare the split variable names with X_train, X_test, y_train, and y_test in the full-code tab.

Help me debug my baseline model run.

Your baseline works, but its evidence disappears into terminal scrollback. Next, you will give every run a durable home with MLflow.

Log Experiments with MLflow

Your baseline now trains two churn models in train.py. However, closing the terminal removes the only visible comparison between them.

MLflow gives each candidate a named run in durable experiment tracking history. Every result stays tied to the model that produced it.

In this step, get ready to:
  • Start a local MLflow tracking server.
  • Add named run logging to both model candidates.
  • Verify the dashboard contains two reusable model runs.
Start the local tracking server

A tracking server receives experiment records from your training script. It stores the records in local files so they survive beyond one terminal session.

  • Return to the activated Command Prompt terminal from earlier.
  • Start the local MLflow server by running:
mlflow server

What does this command do?

The command starts the Tracking UI plus its local API. It stores run metadata in mlflow.db.

Logged model files go into mlartifacts. Both resources remain inside your customer-churn-mlops folder.

You should see the server report http://127.0.0.1:5000. That local service is now ready to preserve every training run.

Why MLflow for this lab?

MLflow keeps this project focused on comparing models through a local client and server. DVC focuses on versioned data pipelines, so it fits a separate pipeline-versioning workflow.

  • Leave the first terminal running so the tracking server stays available.
  • Click the plus button in the Visual Studio Code terminal panel to create a second Command Prompt terminal.
  • Activate the existing virtual environment in the second terminal by running:
.venv\Scripts\activate.bat

Why use a second terminal?

The first terminal remains occupied by the tracking server. The second terminal runs train.py inside the same virtual environment.

Your second terminal prompt should now show that the virtual environment is active. You have one terminal serving MLflow plus one terminal ready to train models.

Server or environment not ready?

  • Confirm the first terminal still uses the activated virtual environment if the server does not start.
  • Confirm the second terminal is inside the customer-churn-mlops folder if activation does not update its prompt.

Help me check both local terminals.

Add the MLflow tracking lifecycle

A useful run preserves the evidence needed to understand a result. MLflow groups that evidence under one run name.

What belongs in a tracked run?

  • Parameters record the configuration used to train a candidate.
  • Metrics record the measured accuracy plus F1 score.
  • A model signature records the expected model inputs plus outputs.
  • A model artifact packages the fitted estimator for reuse.
  • Select train.py from the file list on the left.
  • Replace the import section at the top of train.py with this import group:
import mlflow
import mlflow.sklearn
from mlflow.models import infer_signature
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, f1_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

What do these imports provide?

  • The MLflow imports provide run tracking plus scikit-learn model logging.
  • The infer_signature import describes the model input plus prediction shapes.
  • The existing scikit-learn imports keep the reproducible training workflow intact.
  • Add the tracking configuration below the imports:
TRACKING_URI = "http://localhost:5000"
EXPERIMENT_NAME = "customer-churn-models"
LABELS = {0: "stay", 1: "churn"}

What does this configuration control?

  • TRACKING_URI points the script at your local tracking server.
  • EXPERIMENT_NAME groups both model candidates under one experiment.
  • LABELS keeps the numeric churn outcomes mapped to readable class names.
  • Save train.py.
  • Verify the new imports load inside the virtual environment by running this command in the second terminal:
python train.py

What does this check prove?

The script should still print one metric line for each candidate. A successful run confirms the new MLflow imports are available inside .venv.

You should see the existing logistic-regression plus random-forest metric lines without a Python import error.

Imports not loading?

Confirm the second terminal still uses the activated .venv environment. The package installation belongs to that environment.

Help me diagnose the MLflow import problem.

  • Add the MLflow client setup as the first two lines inside main().
    mlflow.set_tracking_uri(TRACKING_URI)
    mlflow.set_experiment(EXPERIMENT_NAME)

How does the script find its experiment?

mlflow.set_tracking_uri() connects the client to the local server. mlflow.set_experiment() selects the named experiment for subsequent runs.

  • Save train.py.
  • Confirm the client reaches the local server by running:
python train.py

What changed during this run?

The script now connects to the local tracking service before training. It also selects customer-churn-models as the active experiment.

The two terminal metric lines should still print successfully. The local server now has the experiment ready to receive named runs.

  • Add the first section of train_and_log() below create_candidates().
def train_and_log(run_name, model, params, X_train, X_test, y_train, y_test):
    with mlflow.start_run(run_name=run_name):
        model.fit(X_train, y_train)
        predictions = model.predict(X_test)
        metrics = {
            "accuracy": float(accuracy_score(y_test, predictions)),
            "f1": float(f1_score(y_test, predictions, average="binary")),
        }

        mlflow.log_params(params)
        mlflow.log_metrics(metrics)

        signature = infer_signature(X_test, predictions)
        model_info = mlflow.sklearn.log_model(
            sk_model=model,
            name="model",
            signature=signature,
            input_example=X_test[:5],
        )

How does each run capture its evidence?

  • mlflow.start_run() opens one named record for the current candidate.
  • metrics keeps accuracy plus binary F1 together.
  • mlflow.log_params() records the model configuration.
  • mlflow.sklearn.log_model() saves the fitted estimator with its signature plus input example.
  • Complete train_and_log() by adding this section directly below the model logging call:
        print(
            f"{run_name}: accuracy={metrics['accuracy']:.3f}, "
            f"f1={metrics['f1']:.3f}"
        )

        return {
            "run_name": run_name,
            "accuracy": metrics["accuracy"],
            "f1": metrics["f1"],
            "model_uri": model_info.model_uri,
        }

Why return the model URI?

The printed line preserves the familiar terminal feedback. The returned record keeps each score tied to the logged model location.

  • Save train.py.
  • Check that the new helper is valid Python by running:
python train.py

What does this check confirm?

The existing training loop should still print both candidate results. A successful run confirms that train_and_log() is defined without breaking the script.

Seeing a Python syntax problem?

Check the indentation inside train_and_log(). Every statement inside the run context needs consistent spacing.

Help me inspect the helper function.

Each candidate now needs a parameter record. The training loop can pass that record into train_and_log() with the fitted model.

  • Attach this parameter dictionary to the logistic-regression tuple after its make_pipeline() value:
            {
                "model_type": "logistic_regression",
                "scaler": "standard",
                "max_iter": 1000,
            },

What does this parameter record describe?

The dictionary records the estimator type plus its scaling choice. It also preserves the maximum iteration setting used for this run.

  • Attach this parameter dictionary to the random-forest tuple after its RandomForestClassifier() value:
            {
                "model_type": "random_forest",
                "n_estimators": 200,
                "max_depth": 5,
                "random_state": 42,
            },

What does the forest record preserve?

This dictionary preserves the tree count plus maximum depth. It also records the fixed random state behind the reproducible candidate.

  • Delete the existing loop that starts with for run_name, model in create_candidates():.
  • Add the tracked training loop in the same location:
    [
        train_and_log(
            run_name,
            model,
            params,
            X_train,
            X_test,
            y_train,
            y_test,
        )
        for run_name, model, params in create_candidates()
    ]

How does this create two runs?

The comprehension passes each candidate plus its parameter dictionary into train_and_log(). Each call opens a separate named MLflow run.

  • Save the completed train.py file.

✔️ Awesome, I've got everything!

Your tracking code is ready. Double check that train.py is saved before running it.

ⓧ I'd like to double check the full code

Compare your complete train.py file with this version:

import mlflow
import mlflow.sklearn
from mlflow.models import infer_signature
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, f1_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

TRACKING_URI = "http://localhost:5000"
EXPERIMENT_NAME = "customer-churn-models"
LABELS = {0: "stay", 1: "churn"}


def create_candidates():
    return [
        (
            "logistic-regression",
            make_pipeline(
                StandardScaler(),
                LogisticRegression(max_iter=1000),
            ),
            {
                "model_type": "logistic_regression",
                "scaler": "standard",
                "max_iter": 1000,
            },
        ),
        (
            "random-forest",
            RandomForestClassifier(
                n_estimators=200,
                max_depth=5,
                random_state=42,
            ),
            {
                "model_type": "random_forest",
                "n_estimators": 200,
                "max_depth": 5,
                "random_state": 42,
            },
        ),
    ]


def train_and_log(run_name, model, params, X_train, X_test, y_train, y_test):
    with mlflow.start_run(run_name=run_name):
        model.fit(X_train, y_train)
        predictions = model.predict(X_test)
        metrics = {
            "accuracy": float(accuracy_score(y_test, predictions)),
            "f1": float(f1_score(y_test, predictions, average="binary")),
        }

        mlflow.log_params(params)
        mlflow.log_metrics(metrics)

        signature = infer_signature(X_test, predictions)
        model_info = mlflow.sklearn.log_model(
            sk_model=model,
            name="model",
            signature=signature,
            input_example=X_test[:5],
        )

        print(
            f"{run_name}: accuracy={metrics['accuracy']:.3f}, "
            f"f1={metrics['f1']:.3f}"
        )

        return {
            "run_name": run_name,
            "accuracy": metrics["accuracy"],
            "f1": metrics["f1"],
            "model_uri": model_info.model_uri,
        }


def main():
    mlflow.set_tracking_uri(TRACKING_URI)
    mlflow.set_experiment(EXPERIMENT_NAME)

    X, y = make_classification(
        n_samples=1000,
        n_features=8,
        n_informative=5,
        n_redundant=1,
        weights=[0.7, 0.3],
        class_sep=1.0,
        random_state=42,
    )

    X_train, X_test, y_train, y_test = train_test_split(
        X,
        y,
        test_size=0.2,
        random_state=42,
        stratify=y,
    )

    [
        train_and_log(
            run_name,
            model,
            params,
            X_train,
            X_test,
            y_train,
            y_test,
        )
        for run_name, model, params in create_candidates()
    ]


if __name__ == "__main__":
    main()
Run and inspect the tracked experiments

The script now sends a complete record for each candidate to the local server. The dashboard turns those records into a comparison you can revisit.

Before you run the script, take a moment to predict how many named runs the dashboard will contain.

  • Run the tracked training workflow from the second terminal:
python train.py

What happens during this run?

The script trains both candidates against the same split. Each candidate creates its own run with parameters plus metrics.

Each run also receives an inferred signature plus an input example. The fitted model is stored as a reusable artifact.

You should see one accuracy plus F1 line for logistic-regression. You should also see one for random-forest.

Training run not reaching MLflow?

  • Confirm the first terminal still runs the local MLflow server.
  • Confirm TRACKING_URI matches the local dashboard address in train.py.
  • Confirm the second terminal still uses the activated virtual environment.

Help me debug the tracked run.

  • Press the Windows key to open the search bar.
  • Type the name of your web browser.
  • Press Enter to open the browser.
  • Enter http://localhost:5000 in the address bar.
  • Select the customer-churn-models experiment from the experiment list.
  • Confirm the run list contains logistic-regression plus random-forest.
  • Open each named run to confirm its recorded parameters plus metrics.
  • Inspect each logged model to confirm its signature plus input example.

You should see two separate runs under customer-churn-models. Each run preserves its configuration, accuracy, F1 score, signature, input example, and model artifact.

Dashboard missing a run?

Refresh the browser after python train.py finishes. Confirm you selected customer-churn-models instead of another experiment.

Help me find the missing MLflow run.

You have closed the experiment-history gap. Both candidate models now carry the evidence plus artifacts needed for a durable comparison.

Your model runs are tracked and reusable. Next, you will turn their F1 scores into a champion selection plus a visible churn prediction.

Select and Reload the Champion

Your MLflow dashboard now preserves both candidate runs. That experiment history gives you evidence you can use after the terminal closes.

A comparison dashboard becomes valuable when its evidence leads to a reusable model. You will rank the candidates by F1 before using accuracy as a secondary signal.

The winning run already points to a logged model artifact. This step reloads that artifact for one held-out churn prediction.

In this step, get ready to:
  • Compare tracked metrics to identify the strongest run.
  • Select the champion from the retained run results.
  • Reload the logged model to produce a held-out prediction.
Compare the tracked candidates

Both runs used the same training split. Their recorded metrics support a direct comparison.

  • Return to the browser dashboard at http://localhost:5000.
  • Select the customer-churn-models experiment from the experiment list.
  • Compare the f1 values across the two run rows to identify the higher score.
  • Check the accuracy values as a secondary signal.
  • Open the run with the higher f1 value.
  • Inspect its recorded parameters to confirm which configuration produced the score.
  • Inspect its model details to confirm that a signature is recorded.
  • Confirm that the logged model artifact is available for reuse.

Why use F1 first?

The synthetic churn data uses a 70/30 class balance. F1 captures how well the model balances precision with recall for the churn class.

Accuracy measures the overall proportion of correct predictions. It provides a secondary check on the model's performance.

That comparison gives you an evidence-backed winner. The next edit turns that decision into working inference.

Select and reload the champion

The train_and_log() function already returns each run's name, metrics, and model location. The script currently runs the list comprehension without retaining those returned dictionaries.

  • In train.py, locate the list comprehension immediately after the train_test_split() block.
  • Use this existing list as your reference:
[
        train_and_log(
            run_name,
            model,
            params,
            X_train,
            X_test,
            y_train,
            y_test,
        )
        for run_name, model, params in create_candidates()
    ]

Why capture the returned values?

The list comprehension trains both candidates. Its returned dictionaries contain the evidence needed to choose a champion.

  • Replace the opening bracket with results = [ so the block looks like this:
    results = [
        train_and_log(
            run_name,
            model,
            params,
            X_train,
            X_test,
            y_train,
            y_test,
        )
        for run_name, model, params in create_candidates()
    ]

What changed?

The results list retains one returned dictionary for each candidate. Every dictionary includes the run's f1 value and model_uri.

  • Save train.py by pressing Ctrl+S.
  • Confirm the updated list still trains both candidates by running:
python train.py

What should I see?

You will see one accuracy and F1 line for logistic-regression. You will also see one for random-forest.

Those lines confirm that assigning the returned dictionaries to results preserves the existing training workflow.

Script stopped after the edit?

  • Check that results = [ remains indented inside main().
  • Compare the closing bracket with the reference block above.

Help me fix the results list in my training script.

Good, your candidate results are now retained without changing the training workflow.

The stored model_uri identifies the artifact attached to each run. The champion's URI lets the script load the fitted estimator without training that selected object again.

  • In train.py, add the champion selection and prediction code below the closing bracket of results:
    champion = max(results, key=lambda result: result["f1"])
    loaded_model = mlflow.sklearn.load_model(champion["model_uri"])

    predicted_index = int(loaded_model.predict(X_test[:1])[0])
    actual_index = int(y_test[0])

    print(f"Champion: {champion['run_name']}")
    print(f"Predicted: {LABELS[predicted_index]}")
    print(f"Actual: {LABELS[actual_index]}")

What does this code do?

  • The max() call selects the result with the highest f1 value.
  • The load_model() call reloads the fitted estimator from the champion's model_uri.
  • The loaded model predicts the first record in X_test.
  • The matching value from y_test provides the actual class for comparison.
  • The LABELS dictionary turns each numeric class into stay or churn.
  • Save train.py by pressing Ctrl+S.

✔️ Awesome, I've got everything!

Your champion selection and prediction code is saved in train.py.

ⓧ I'd like to double check the full code

Here is the complete train.py file for comparison.

import mlflow
import mlflow.sklearn
from mlflow.models import infer_signature
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, f1_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

TRACKING_URI = "http://localhost:5000"
EXPERIMENT_NAME = "customer-churn-models"
LABELS = {0: "stay", 1: "churn"}


def create_candidates():
    return [
        (
            "logistic-regression",
            make_pipeline(
                StandardScaler(),
                LogisticRegression(max_iter=1000),
            ),
            {
                "model_type": "logistic_regression",
                "scaler": "standard",
                "max_iter": 1000,
            },
        ),
        (
            "random-forest",
            RandomForestClassifier(
                n_estimators=200,
                max_depth=5,
                random_state=42,
            ),
            {
                "model_type": "random_forest",
                "n_estimators": 200,
                "max_depth": 5,
                "random_state": 42,
            },
        ),
    ]


def train_and_log(run_name, model, params, X_train, X_test, y_train, y_test):
    with mlflow.start_run(run_name=run_name):
        model.fit(X_train, y_train)
        predictions = model.predict(X_test)
        metrics = {
            "accuracy": float(accuracy_score(y_test, predictions)),
            "f1": float(f1_score(y_test, predictions, average="binary")),
        }

        mlflow.log_params(params)
        mlflow.log_metrics(metrics)

        signature = infer_signature(X_test, predictions)
        model_info = mlflow.sklearn.log_model(
            sk_model=model,
            name="model",
            signature=signature,
            input_example=X_test[:5],
        )

        print(
            f"{run_name}: accuracy={metrics['accuracy']:.3f}, "
            f"f1={metrics['f1']:.3f}"
        )

        return {
            "run_name": run_name,
            "accuracy": metrics["accuracy"],
            "f1": metrics["f1"],
            "model_uri": model_info.model_uri,
        }


def main():
    mlflow.set_tracking_uri(TRACKING_URI)
    mlflow.set_experiment(EXPERIMENT_NAME)

    X, y = make_classification(
        n_samples=1000,
        n_features=8,
        n_informative=5,
        n_redundant=1,
        weights=[0.7, 0.3],
        class_sep=1.0,
        random_state=42,
    )

    X_train, X_test, y_train, y_test = train_test_split(
        X,
        y,
        test_size=0.2,
        random_state=42,
        stratify=y,
    )

    results = [
        train_and_log(
            run_name,
            model,
            params,
            X_train,
            X_test,
            y_train,
            y_test,
        )
        for run_name, model, params in create_candidates()
    ]

    champion = max(results, key=lambda result: result["f1"])
    loaded_model = mlflow.sklearn.load_model(champion["model_uri"])

    predicted_index = int(loaded_model.predict(X_test[:1])[0])
    actual_index = int(y_test[0])

    print(f"Champion: {champion['run_name']}")
    print(f"Predicted: {LABELS[predicted_index]}")
    print(f"Actual: {LABELS[actual_index]}")


if __name__ == "__main__":
    main()
Run a prediction with the loaded model

The final check proves that the tracked evidence leads to a reusable prediction artifact. Your server terminal must remain running while the script logs and reloads the model.

Before you run the final script, do you expect the loaded champion's prediction to match the actual label? Hold that prediction while the model runs.

  • Run the completed training workflow in the second activated terminal by running:
python train.py

What should I see?

  • The script first prints fresh accuracy and F1 results for both baseline candidates.
  • The Champion: line names the candidate with the highest F1 score.
  • The Predicted: line displays stay or churn for the first held-out customer.
  • The Actual: line displays the known label for that same customer.

Missing the champion output?

  • Confirm that the first terminal still shows the running MLflow server process.
  • Check that the champion code is indented inside main().
  • Compare your train.py file with the complete reference above.

Help me debug why my champion model does not load or print a prediction.

That is the full lifecycle working. Your script now chooses a tracked winner and reloads its saved artifact for a fresh prediction.

Secret mission

Track a Hyperparameter Search

Replace manual parameter choices with a four-combination GridSearchCV sweep. Track the trials in MLflow, then compare the tuned result with your baseline.

Clean Up Your Resources

Clean Up Your Resources

Choose whether to keep your local MLflow lab available, pause its server, or remove it completely. Everything runs on your computer, so these resources have no ongoing service cost.

Resources you used:

  • The local MLflow server that keeps the dashboard available at http://localhost:5000.
  • The .venv virtual environment containing the pinned project packages.
  • The mlflow.db database containing your experiments and runs.
  • The mlartifacts/ folder containing your logged model artifacts.
  • The project files requirements.txt, .gitignore, and train.py.

Keep everything running

No action is needed. Choose this option while you are still comparing runs or working on the project.

  • Leave the terminal running mlflow server open to keep the dashboard available.
  • Keep the customer-churn-mlops folder in place so your environment and experiment history remain available.
  • Use http://localhost:5000 whenever you want to compare the baseline runs with the tracked grid search.

Pause - I'll come back to this later

Shut down the running server to free its terminal. Your customer-churn-mlops folder keeps the complete lab for your next session.

  • Switch back to the terminal running mlflow server.
  • Press Ctrl+C to stop the server.
  • Leave the customer-churn-mlops folder in place for your next session.

Delete - I don't want to use this again

Remove all project resources to start fresh.

Deleting the folder is final. Its scope is limited to this local lab.

  • Switch back to the terminal running mlflow server.
  • Press Ctrl+C to stop the server.
  • Close Visual Studio Code.
  • Delete the customer-churn-mlops folder from the location where you created it.
  • Confirm that customer-churn-mlops no longer appears in that location.

Nice Work!

Nice Work!

You did it! Your local customer churn experiment lab now turns repeatable training runs into durable MLflow evidence.

What you learned:

  • Built a reproducible churn benchmark from synthetic classification data. Trained logistic regression plus a random forest on one stratified train/test split.
  • Logged each candidate's parameters in the MLflow dashboard. Preserved its accuracy plus F1 score beside the run. Attached a model signature. Kept a reusable model artifact ready for loading.
  • Selected the champion model using its highest F1 score. Reloaded the saved artifact to print a stay-or-churn prediction for one held-out customer.
  • Secret Mission: completed a tracked four-combination GridSearchCV sweep. MLflow organizes its trials beneath the random-forest-grid-search parent run. Measured the tuned candidate against the manual random forest baseline.

Ready to quiz yourself?