Build an ML Deployment Risk Checker

Build a tested ML risk checker with leakage detection and GitHub Actions CI.

Introduction

30 Second Summary

Software releases often carry warning signs before they fail. Under deadline pressure, those clues can disappear inside spreadsheets and checklists.

In this project, you will build a Python deployment risk checker powered by an interpretable decision tree. A deliberately perfect result exposes data leakage before GitHub Actions runs automated tests against the corrected model on every push.

What You'll Build

You'll push your finished project to a public repository and watch a passing CI workflow confirm that its leakage-safe risk predictions still meet the tested contract.

By the end of this project, you'll have:

  • A deliberately naive binary classifier you can run to expose a suspicious 100.0% score and identify rollback_occurred as the leaked feature.
  • A leakage-safe deployment risk model that reports its holdout accuracy and returns a visible LOW or HIGH prediction for a sample release.
  • A public repository with defensive unit tests that protect the model's feature contract and a passing CI workflow that reruns those checks on every push.
  • Secret Mission: A four-fold cross-validation experiment that compares decision-tree depths and helps you justify the most stable choice.

Are there any prerequisites?

You need a Windows computer plus a GitHub account. Basic familiarity with Python files and terminal commands is helpful.

The outlined path costs $0 with a public repository and standard GitHub-hosted runners.

Before We Start

Before any hands-on work, connect this deployment risk checker to the role you want to explore. This gives the applied machine learning, testing, and CI work ahead an employer-facing purpose.

Set Up the Windows Project

Your machine learning checker needs a predictable Python environment. An isolated virtual environment keeps its packages separate from your other projects.

This project requires Python 3.11 or newer because scikit-learn 1.9.1 supports that baseline.

In this step, you'll create the Windows workspace in Visual Studio Code. You'll pin the exact dependency before installing it inside the project.

In this step, get ready to:
  • Create a Windows workspace for the project.
  • Pin the project dependency while excluding generated files.
  • Install scikit-learn in an isolated virtual environment.
Create the project workspace

A folder opened in Visual Studio Code becomes a workspace. Its integrated PowerShell terminal starts inside that folder.

  • Press the Windows key to open Windows search.
  • Type Visual Studio Code into the search bar.
  • Press Enter to open Visual Studio Code.
  • Select File from the top menu.
  • Select Open Folder....
  • Choose your Desktop in the folder picker.
  • Create a folder named deployment-risk-checker inside your Desktop.
  • Confirm deployment-risk-checker as the folder to open.

You should see deployment-risk-checker at the top of the Explorer sidebar.

  • Confirm that you trust the workspace if Visual Studio Code asks.
  • Select Terminal from the top menu.
  • Select New Terminal.

The PowerShell prompt should end with deployment-risk-checker. This confirms that commands run inside the correct folder.

  • Check your installed Python version by running this command:
python -c "import sys; print(sys.version)"

What does this command do?

  • The python command starts your installed Python interpreter.
  • The -c option runs the quoted Python statement directly.
  • The sys.version value prints the interpreter version used by this terminal.

✔️ I see version 3.11 or higher

Python is ready for scikit-learn 1.9.1. Keep this Visual Studio Code window open.

ⓧ I see an older version

Your installed Python version is below 3.11. Install a current stable Windows release before creating the virtual environment.

  • Visit the official Python Windows releases page.
  • Choose a current stable Python release.
  • Download its Windows installer.
  • Run the downloaded installer.
  • Complete the installation using the default options.
  • Restart Visual Studio Code after the installation finishes.
  • Return to the deployment-risk-checker workspace.
  • Start a new integrated PowerShell terminal.
  • Check the new Python version by running:
python -c "import sys; print(sys.version)"

Why run the check again?

Restarting Visual Studio Code lets its terminal detect the new Python installation. The command confirms which interpreter the project now uses.

You should now see Python 3.11 or newer.

ⓧ Command not found

Windows cannot currently find a Python installation. Install a current stable release before continuing.

  • Visit the official Python Windows releases page.
  • Choose a current stable Python release.
  • Download its Windows installer.
  • Run the downloaded installer.
  • Complete the installation using the default options.
  • Restart Visual Studio Code after the installation finishes.
  • Return to the deployment-risk-checker workspace.
  • Start a new integrated PowerShell terminal.
  • Check the installed Python version by running:
python -c "import sys; print(sys.version)"

What does the result confirm?

The printed version confirms that PowerShell can find Python. Continue once the version is 3.11 or newer.

Add the reproducible project files

A dependency pin records the exact library release this project expects. This prevents a future installation from silently using a different scikit-learn version.

  • Create requirements.txt inside deployment-risk-checker using the Explorer sidebar.
  • Add the exact dependency pin below:
scikit-learn==1.9.1

What does this dependency pin do?

The double equals sign tells pip to install exactly scikit-learn 1.9.1. Every environment built from this file receives the same library release.

  • Save requirements.txt.
  • Confirm that requirements.txt appears in the Explorer sidebar.

Does the dependency file look different?

  • Check that the file is named requirements.txt with no extra file extension.
  • Check that the file contains only scikit-learn==1.9.1.

Help me fix my dependency file.

A .gitignore file keeps generated environment files out of the public repository. The project code stays visible without uploading local Python clutter.

  • Create .gitignore inside deployment-risk-checker using the Explorer sidebar.
  • Add the three exclusion patterns below:
.venv/
__pycache__/
*.pyc

What do these patterns exclude?

  • The .venv/ pattern excludes the local virtual environment.
  • The __pycache__/ pattern excludes Python cache directories.
  • The *.pyc pattern excludes compiled Python bytecode files.
  • Save .gitignore.
  • Confirm that .gitignore appears beside requirements.txt in the Explorer sidebar.

Is the ignore file missing?

  • Check that the filename starts with a full stop.
  • Remove any .txt extension that Windows added to the filename.
  • Check that all three patterns appear on separate lines.

Help me fix my .gitignore file.

✔️ Awesome, I've got everything!

Great. Save both files before continuing to the virtual environment.

ⓧ I'd like to double check the full code

scikit-learn==1.9.1

.venv/
__pycache__/
*.pyc
Create and verify the virtual environment

The project-specific interpreter lives inside .venv. Calling that interpreter by its full path removes any doubt about where scikit-learn gets installed.

  • Create the virtual environment by running this command in the integrated PowerShell terminal:
python -m venv .venv

What does this command do?

The venv module creates an isolated Python environment inside .venv. Its Windows layout includes a dedicated Python executable inside the Scripts directory.

  • Confirm that a .venv folder now appears in the Explorer sidebar.

Your isolated environment now exists inside the project. You can use its interpreter without activating the environment.

Don't see the .venv folder?

  • Check that the PowerShell prompt ends with deployment-risk-checker.
  • Check that your Python version is 3.11 or newer.
  • Rerun the environment command after correcting the folder or Python version.

Help me create the Windows virtual environment.

  • Install the pinned dependency into .venv by running:
.\.venv\Scripts\python.exe -m pip install -r requirements.txt

How does this installation stay isolated?

  • The .\.venv\Scripts\python.exe path selects the interpreter inside this project.
  • The -m pip arguments run the package installer belonging to that interpreter.
  • The -r requirements.txt arguments install the exact dependency recorded in the requirements file.

PowerShell should show package installation progress. The command should return to the prompt without an installation error.

Did the installation fail?

  • Check that requirements.txt contains the exact dependency pin.
  • Check that .\.venv\Scripts\python.exe exists inside the project.
  • Check that the computer has an internet connection before retrying the installation.

Help me install scikit-learn in .venv.

Before you run the final check, what version do you expect the isolated interpreter to report?

  • Verify the installed scikit-learn version by running this command:
.\.venv\Scripts\python.exe -c "import sklearn; print(sklearn.__version__)"

What does this verification prove?

The command imports scikit-learn using the interpreter inside .venv. Printing sklearn.__version__ confirms the installed release.

You should see 1.9.1 in the terminal. That output proves the project is using the required scikit-learn version.

Don't see version 1.9.1?

  • Check that the verification command starts with .\.venv\Scripts\python.exe.
  • Check that requirements.txt pins scikit-learn==1.9.1.
  • Rerun the installation command before repeating the version check.

Help me verify the scikit-learn installation.

That's your clean foundation in place. Next, you'll train a classifier whose perfect score hides a serious data problem.

Build a Leaky Model

Your isolated Python environment is ready with scikit-learn installed. The next challenge is deciding which historical deployment details belong in the model.

This step intentionally trains a first-pass decision tree from the available records. You will run it before judging whether its score deserves your trust.

In this step, get ready to:
  • Create a balanced dataset of 24 synthetic deployment records.
  • Train a classifier with a reproducible stratified split.
  • Investigate the feature behind the model's perfect score.
Create the deployment dataset

A CSV dataset stores each deployment as one row. Its columns capture engineering metrics plus the final risk label used for training.

  • Select New File... in the Explorer view.
  • Enter deployment_data.csv as the file name.
  • Press Enter.
  • Add the 24 deployment records by pasting this dataset:
lines_changed,test_coverage,failed_checks,services_touched,previous_incidents,rollback_occurred,high_risk
45,95,0,1,0,0,0
80,91,0,1,0,0,0
120,88,0,2,0,0,0
160,84,1,2,0,0,0
210,82,0,2,1,0,0
250,79,1,3,0,0,0
300,76,1,2,1,0,0
350,74,0,3,1,0,0
400,72,1,4,0,0,0
480,68,0,1,0,0,0
520,78,1,2,0,0,0
600,82,0,3,1,0,0
45,95,0,1,0,1,1
230,66,1,2,0,1,1
280,64,1,2,1,1,1
320,69,2,3,0,1,1
360,73,2,2,1,1,1
410,75,3,3,1,1,1
460,77,2,4,0,1,1
510,74,1,5,1,1,1
560,80,2,4,1,1,1
620,83,1,5,2,1,1
700,85,0,6,2,1,1
750,88,2,5,0,1,1

What does this dataset contain?

  • The first five columns describe information available before a deployment.
  • The rollback_occurred column records whether a rollback happened.
  • The high_risk column supplies the binary label that the model learns to predict.
  • The balanced labels give the model 12 low-risk examples plus 12 high-risk examples.
  • Save deployment_data.csv by pressing Ctrl+S.
  • Confirm that the final record appears on line 25 in the editor.

Missing rows from the dataset?

Check that the header occupies line 1. The 24 records should fill lines 2 through 25.

Make sure each record has seven comma-separated values. An omitted value shifts the remaining columns out of place.

Help me check my deployment dataset.

✔️ Awesome, I've got everything!

Your saved dataset contains one header plus 24 balanced deployment records.

ⓧ I'd like to double check the full code

lines_changed,test_coverage,failed_checks,services_touched,previous_incidents,rollback_occurred,high_risk
45,95,0,1,0,0,0
80,91,0,1,0,0,0
120,88,0,2,0,0,0
160,84,1,2,0,0,0
210,82,0,2,1,0,0
250,79,1,3,0,0,0
300,76,1,2,1,0,0
350,74,0,3,1,0,0
400,72,1,4,0,0,0
480,68,0,1,0,0,0
520,78,1,2,0,0,0
600,82,0,3,1,0,0
45,95,0,1,0,1,1
230,66,1,2,0,1,1
280,64,1,2,1,1,1
320,69,2,3,0,1,1
360,73,2,2,1,1,1
410,75,3,3,1,1,1
460,77,2,4,0,1,1
510,74,1,5,1,1,1
560,80,2,4,1,1,1
620,83,1,5,2,1,1
700,85,0,6,2,1,1
750,88,2,5,0,1,1
Build the naive classifier

The script needs a stable path to the dataset plus a defined feature list. It then converts each CSV value into a number the classifier can process.

  • Select New File... in the Explorer view.
  • Enter leaky_demo.py as the file name.
  • Press Enter.
  • Add the imports plus the data-loading logic by pasting this code:
import csv
from pathlib import Path

from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

DATA_PATH = Path(__file__).with_name("deployment_data.csv")
LEAKY_FEATURE_NAMES = [
    "lines_changed",
    "test_coverage",
    "failed_checks",
    "services_touched",
    "previous_incidents",
    "rollback_occurred",
]


def main():
    with DATA_PATH.open(newline="", encoding="utf-8") as csv_file:
        rows = list(csv.DictReader(csv_file))

    features = [
        [float(row[name]) for name in LEAKY_FEATURE_NAMES]
        for row in rows
    ]
    labels = [int(row["high_risk"]) for row in rows]

What does this code do?

  • The imports provide CSV loading plus model training utilities.
  • DATA_PATH locates deployment_data.csv beside the script.
  • LEAKY_FEATURE_NAMES defines the six inputs used by this first classifier.
  • main() converts the feature values to decimals plus the labels to integers.
  • Save leaky_demo.py by pressing Ctrl+S.
  • Confirm that leaky_demo.py appears beside deployment_data.csv in the Explorer view.

File missing from the Explorer view?

Check that you created leaky_demo.py inside the open deployment-risk-checker folder. Saving the file should remove the unsaved dot from its editor tab.

Help me locate my Python file.

The remaining logic creates a reproducible train/test split. It also reports which input contributes the most to the fitted tree.

  • Place your cursor directly below the labels = [int(row["high_risk"]) for row in rows] line.
  • Complete the classifier by adding this code:

    x_train, x_test, y_train, y_test = train_test_split(
        features,
        labels,
        test_size=0.25,
        random_state=42,
        stratify=labels,
    )

    model = DecisionTreeClassifier(max_depth=3, random_state=42)
    model.fit(x_train, y_train)
    predictions = model.predict(x_test)
    accuracy = accuracy_score(y_test, predictions)

    top_index = max(
        range(len(LEAKY_FEATURE_NAMES)),
        key=lambda index: model.feature_importances_[index],
    )
    top_feature = LEAKY_FEATURE_NAMES[top_index]

    print(f"Naive test accuracy: {accuracy:.1%}")
    print(f"Top feature: {top_feature}")
    print(
        "Problem: rollback_occurred happens after deployment, "
        "so the score leaks the answer."
    )


if __name__ == "__main__":
    main()

How does the model produce its score?

  • train_test_split() reserves 25 percent of the records for testing.
  • stratify=labels preserves the label balance across the training data plus the test data.
  • DecisionTreeClassifier(max_depth=3, random_state=42) fits a depth-limited tree with repeatable behavior.
  • feature_importances_ identifies the input that contributes most strongly to the tree's decisions.
  • Save leaky_demo.py by pressing Ctrl+S.
  • Confirm that the unsaved dot disappears from the leaky_demo.py editor tab.

Seeing code warnings in the editor?

Check that the training block stays indented inside main(). Keep the final condition flush with the left edge of the file.

Compare every bracket plus parenthesis with the complete file below. A missing closing character can make several later lines appear incorrect.

Help me check the classifier code.

✔️ Awesome, I've got everything!

Your saved script can now load the dataset plus train the first classifier.

ⓧ I'd like to double check the full code

import csv
from pathlib import Path

from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

DATA_PATH = Path(__file__).with_name("deployment_data.csv")
LEAKY_FEATURE_NAMES = [
    "lines_changed",
    "test_coverage",
    "failed_checks",
    "services_touched",
    "previous_incidents",
    "rollback_occurred",
]


def main():
    with DATA_PATH.open(newline="", encoding="utf-8") as csv_file:
        rows = list(csv.DictReader(csv_file))

    features = [
        [float(row[name]) for name in LEAKY_FEATURE_NAMES]
        for row in rows
    ]
    labels = [int(row["high_risk"]) for row in rows]

    x_train, x_test, y_train, y_test = train_test_split(
        features,
        labels,
        test_size=0.25,
        random_state=42,
        stratify=labels,
    )

    model = DecisionTreeClassifier(max_depth=3, random_state=42)
    model.fit(x_train, y_train)
    predictions = model.predict(x_test)
    accuracy = accuracy_score(y_test, predictions)

    top_index = max(
        range(len(LEAKY_FEATURE_NAMES)),
        key=lambda index: model.feature_importances_[index],
    )
    top_feature = LEAKY_FEATURE_NAMES[top_index]

    print(f"Naive test accuracy: {accuracy:.1%}")
    print(f"Top feature: {top_feature}")
    print(
        "Problem: rollback_occurred happens after deployment, "
        "so the score leaks the answer."
    )


if __name__ == "__main__":
    main()
Run the suspicious model

A perfect score deserves investigation because real engineering signals overlap. The most important feature reveals what the classifier actually learned.

Before you run the script, which recorded input do you think the tree will rely on most heavily?

  • Run the classifier in the PowerShell terminal with this command:
.\.venv\Scripts\python.exe leaky_demo.py

PowerShell prints Naive test accuracy: 100.0%.

The next line reads Top feature: rollback_occurred.

The final line reads Problem: rollback_occurred happens after deployment, so the score leaks the answer..

Why is 100% accuracy a warning?

This model contains data leakage. The rollback_occurred value is only known after a deployment has happened.

The synthetic dataset makes that column match the target label. The tree has effectively read the answer instead of predicting risk from pre-deployment evidence.

Didn't get the three expected lines?

Confirm that PowerShell is still in the deployment-risk-checker folder. Both leaky_demo.py plus deployment_data.csv must appear there.

Compare your dataset plus script with the full-file tabs above. A changed value can alter the fitted tree or its score.

Help me reproduce the leaky model output.

You found the intended trap. Next, you will rebuild the model around information that exists before a release decision.

Build a Leakage-Safe Model

Your leaky model proved that a perfect score can be meaningless when the inputs contain the answer. That deliberate case of data leakage exposed the weakness before it could hide inside a real project.

This step builds a second model that uses information available before deployment. You will add a feature guard, measure holdout accuracy, and produce a visible risk prediction.

In this step, get ready to:
  • Define five safe input features and block the leaked columns.
  • Train a reproducible decision tree with a stratified holdout split.
  • Predict the risk of one sample deployment.
Define the safe feature contract

A feature contract names the columns that the model may use. The same contract can also reject post-deployment outcomes before training begins.

  • Create risk_model.py inside the deployment-risk-checker folder using the file controls in Visual Studio Code.
  • Initialize risk_model.py with its data contract by pasting this code:
import csv
from pathlib import Path

from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

DATA_PATH = Path(__file__).with_name("deployment_data.csv")
FEATURE_NAMES = [
    "lines_changed",
    "test_coverage",
    "failed_checks",
    "services_touched",
    "previous_incidents",
]
FORBIDDEN_FEATURES = {"rollback_occurred", "high_risk"}

What does this code define?

  • The imports provide CSV loading, path handling, model evaluation, dataset splitting, and the scikit-learn classifier.
  • FEATURE_NAMES contains the five measurements known before deployment.
  • FORBIDDEN_FEATURES names the rollback result and target label that must never enter the model inputs.
  • DATA_PATH locates deployment_data.csv beside the Python file.
  • Save risk_model.py.
  • Search risk_model.py for rollback_occurred.

You should see one match inside FORBIDDEN_FEATURES. Its absence from FEATURE_NAMES confirms that the safe input list excludes the rollback result.

Is the leaked column still in the safe list?

Remove rollback_occurred from FEATURE_NAMES if your search shows a match there. Keep it inside FORBIDDEN_FEATURES so the guard can reject it.

Check that the filename is exactly risk_model.py.

Help me check my safe and forbidden feature lists.

  • Add the validation guard below FORBIDDEN_FEATURES by pasting this code:
def validate_feature_names(feature_names):
    blocked = sorted(set(feature_names) & FORBIDDEN_FEATURES)
    if blocked:
        raise ValueError(f"Leakage detected in features: {', '.join(blocked)}")

How does the guard stop leakage?

  • validate_feature_names() compares the requested inputs with FORBIDDEN_FEATURES.
  • blocked holds any forbidden names found in that comparison.
  • ValueError stops dataset loading when a forbidden name appears.
  • Save risk_model.py.
  • Search the file for FORBIDDEN_FEATURES.

You should see the set definition and the intersection inside validate_feature_names(). Those two matches connect the blocked names to the guard.

Does the guard look incomplete?

Check that the intersection uses feature_names with FORBIDDEN_FEATURES. Check that raise ValueError stays indented inside the condition.

Help me fix my feature validation guard.

Train with pre-deployment data

The safe loader applies the contract before reading values from each record. The training function then uses a reproducible stratified split for an honest holdout evaluation.

  • Add the dataset loader below validate_feature_names() by pasting this code:
def load_dataset(feature_names=None):
    selected_features = FEATURE_NAMES if feature_names is None else list(feature_names)
    validate_feature_names(selected_features)
    with DATA_PATH.open(newline="", encoding="utf-8") as csv_file:
        rows = list(csv.DictReader(csv_file))
    features = [[float(row[name]) for name in selected_features] for row in rows]
    labels = [int(row["high_risk"]) for row in rows]
    return features, labels

What does the loader protect?

  • selected_features defaults to the five names in FEATURE_NAMES.
  • validate_feature_names() checks the selected names before the CSV values are extracted.
  • features contains numeric input rows built from the approved columns.
  • labels contains the separate high_risk outcomes used for training.
  • Save risk_model.py.
  • Search the file for selected_features.

You should see three matches. The selected names are created, validated, and used to extract each input row.

Is the target mixed into the inputs?

Check that the feature comprehension reads from selected_features. The label comprehension should read high_risk separately.

Help me separate my model features from the target labels.

A decision tree can learn direct rules from these five numeric inputs. Limiting its depth keeps the small educational model interpretable.

  • Add the training function below load_dataset() by pasting this code:
def train_model():
    features, labels = load_dataset()
    x_train, x_test, y_train, y_test = train_test_split(features, labels, test_size=0.25, random_state=42, stratify=labels)
    model = DecisionTreeClassifier(max_depth=3, random_state=42)
    model.fit(x_train, y_train)
    predictions = model.predict(x_test)
    accuracy = float(accuracy_score(y_test, predictions))
    return model, accuracy

How is the model evaluated?

  • train_test_split() reserves 25 percent of the records for testing.
  • stratify=labels preserves the class balance across the split.
  • random_state=42 makes the split and model fitting reproducible.
  • accuracy_score() measures the fraction of holdout predictions that match their labels.
  • Save risk_model.py.
  • Search the file for random_state=42.

You should see the fixed random state in both the split and classifier. Those two uses make repeated runs comparable.

Does the training function show an editor warning?

Check that features and labels come from load_dataset(). Check that model.fit() receives the training portions.

Help me debug my model training function.

Predict one deployment risk

A useful command-line model needs to evaluate new deployment metrics after training. The prediction function also checks that every example contains exactly five values.

  • Add the prediction function below train_model() by pasting this code:
def predict_risk(model, metrics):
    if len(metrics) != len(FEATURE_NAMES):
        raise ValueError(f"Expected {len(FEATURE_NAMES)} metrics, got {len(metrics)}")
    return int(model.predict([[float(value) for value in metrics]])[0])

What does the prediction contract enforce?

  • len(FEATURE_NAMES) defines the required number of metrics.
  • ValueError stops incomplete or oversized examples.
  • model.predict() returns the predicted class for the deployment.
  • int() converts the result into a plain binary label.
  • Save risk_model.py.
  • Search the file for len(FEATURE_NAMES).

You should see two matches inside predict_risk(). The function uses the same feature count for validation and its error message.

Does the predictor reject five metrics?

Check that the condition uses != between the two lengths. Check that the returned prediction uses the first result at index 0.

Help me fix my prediction input check.

  • Add the command-line entry point below predict_risk() by pasting this code:
def main():
    model, accuracy = train_model()
    sample_deployment = [420, 68, 2, 4, 1]
    prediction = predict_risk(model, sample_deployment)
    risk_label = "HIGH" if prediction == 1 else "LOW"
    print(f"Leakage-safe features: {', '.join(FEATURE_NAMES)}")
    print(f"Holdout accuracy: {accuracy:.1%}")
    print(f"Sample deployment risk: {risk_label}")


if __name__ == "__main__":
    main()

What happens when the script runs?

  • main() trains the safe model and stores its holdout accuracy.
  • sample_deployment represents five pre-deployment measurements in feature-list order.
  • risk_label converts the binary prediction into HIGH or LOW.
  • The three print calls expose the safe inputs, evaluation result, and sample decision.
  • Save risk_model.py.

✔️ Awesome, I've got everything!

Great. Double check that risk_model.py is saved before you run it.

ⓧ I'd like to double check the full code

import csv
from pathlib import Path

from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

DATA_PATH = Path(__file__).with_name("deployment_data.csv")
FEATURE_NAMES = [
    "lines_changed",
    "test_coverage",
    "failed_checks",
    "services_touched",
    "previous_incidents",
]
FORBIDDEN_FEATURES = {"rollback_occurred", "high_risk"}


def validate_feature_names(feature_names):
    blocked = sorted(set(feature_names) & FORBIDDEN_FEATURES)
    if blocked:
        raise ValueError(f"Leakage detected in features: {', '.join(blocked)}")


def load_dataset(feature_names=None):
    selected_features = FEATURE_NAMES if feature_names is None else list(feature_names)
    validate_feature_names(selected_features)
    with DATA_PATH.open(newline="", encoding="utf-8") as csv_file:
        rows = list(csv.DictReader(csv_file))
    features = [[float(row[name]) for name in selected_features] for row in rows]
    labels = [int(row["high_risk"]) for row in rows]
    return features, labels


def train_model():
    features, labels = load_dataset()
    x_train, x_test, y_train, y_test = train_test_split(features, labels, test_size=0.25, random_state=42, stratify=labels)
    model = DecisionTreeClassifier(max_depth=3, random_state=42)
    model.fit(x_train, y_train)
    predictions = model.predict(x_test)
    accuracy = float(accuracy_score(y_test, predictions))
    return model, accuracy


def predict_risk(model, metrics):
    if len(metrics) != len(FEATURE_NAMES):
        raise ValueError(f"Expected {len(FEATURE_NAMES)} metrics, got {len(metrics)}")
    return int(model.predict([[float(value) for value in metrics]])[0])


def main():
    model, accuracy = train_model()
    sample_deployment = [420, 68, 2, 4, 1]
    prediction = predict_risk(model, sample_deployment)
    risk_label = "HIGH" if prediction == 1 else "LOW"
    print(f"Leakage-safe features: {', '.join(FEATURE_NAMES)}")
    print(f"Holdout accuracy: {accuracy:.1%}")
    print(f"Sample deployment risk: {risk_label}")


if __name__ == "__main__":
    main()

Before you run the corrected model, do you expect its output to contain rollback_occurred?

  • Run the leakage-safe model from the PowerShell terminal in deployment-risk-checker by using this command:
.\.venv\Scripts\python.exe risk_model.py

How do you read the result?

  • The first line lists lines_changed, test_coverage, failed_checks, services_touched, and previous_incidents as the leakage-safe features.
  • The second line reports the model's holdout accuracy.
  • The third line reports a LOW or HIGH risk prediction for the sample deployment.
  • The absence of rollback_occurred proves that the corrected model uses only pre-deployment inputs.

That is the key correction complete. Your model now reports an honest holdout result and produces a deployment risk decision without reading the rollback outcome.

Script does not print all three lines?

Check that PowerShell is still inside the deployment-risk-checker folder. Check that risk_model.py and deployment_data.csv appear in the same folder.

Compare the function names in your file with load_dataset(), train_model(), and predict_risk().

Help me debug my leakage-safe model output.

Your leakage-safe model is training and making predictions. Next, you will add automated tests that prevent the forbidden columns from returning.

Protect the Model with Tests

Your leakage-safe model now makes predictions from pre-deployment metrics. Future edits could quietly return a forbidden feature or break the prediction contract.

In this step, you will use Python's built-in unit testing framework to turn those safety rules into repeatable checks. Each test protects one promise that the model must keep.

In this step, get ready to:
  • Reject leaked features with an automated regression test.
  • Keep model accuracy inside its valid range.
  • Restrict prediction output to binary labels.
Guard against leaked features

A regression test captures behavior that must survive future changes. This first test injects rollback_occurred into the feature list to prove that validate_feature_names() raises ValueError.

  • Switch back to Visual Studio Code from earlier.
  • Create test_risk_model.py inside the deployment-risk-checker folder with this first regression test:
import unittest

from risk_model import (
    FEATURE_NAMES,
    predict_risk,
    train_model,
    validate_feature_names,
)


class RiskModelTests(unittest.TestCase):
    def test_forbidden_feature_is_rejected(self):
        with self.assertRaises(ValueError):
            validate_feature_names(FEATURE_NAMES + ["rollback_occurred"])


if __name__ == "__main__":
    unittest.main()

What Does This Test Protect?

  • The unittest module supplies the test runner plus its assertion methods.
  • The imports reuse the model functions that already exist in risk_model.py.
  • The assertRaises() check passes only when the injected rollback feature triggers ValueError.
  • The final guard starts the test runner when the file runs directly.
  • Save test_risk_model.py.

Before you run this, predict what the guard does with the injected rollback feature.

  • Run the current test module in the PowerShell terminal from earlier by entering:
.\.venv\Scripts\python.exe -m unittest -v test_risk_model

What Should I See?

The test runner discovers test_forbidden_feature_is_rejected. The run finishes with OK because the model rejects the leaked feature.

Test Module Not Running?

  • Confirm that PowerShell is still inside the deployment-risk-checker folder.
  • Check that the file is named test_risk_model.py.
  • Compare the imported names with the functions in risk_model.py.

Help me diagnose why my first unit test does not run.

Your first regression guard is working. A future edit cannot return the leaked rollback result without producing a failed test.

Check the accuracy contract

The model reports accuracy as a fraction. Its valid range begins at 0.0 and ends at 1.0.

  • In test_risk_model.py, find the test_forbidden_feature_is_rejected() method.
  • Add this method directly below it:
    def test_accuracy_is_a_fraction(self):
        _, accuracy = train_model()
        self.assertGreaterEqual(accuracy, 0.0)
        self.assertLessEqual(accuracy, 1.0)

What Does This Test Protect?

  • The test calls train_model() to get the same accuracy value used by the command-line model.
  • The lower-bound assertion rejects values below 0.0.
  • The upper-bound assertion rejects values above 1.0.
  • Save test_risk_model.py.
  • Run the expanded test module by entering:
.\.venv\Scripts\python.exe -m unittest -v test_risk_model

What Should I See?

The runner now executes two named tests. Both complete successfully before the run ends with OK.

Accuracy Test Failing?

  • Confirm that test_accuracy_is_a_fraction() is indented inside RiskModelTests.
  • Check that the test compares accuracy with 0.0 and 1.0.
  • Confirm that train_model() still returns the model followed by its accuracy.

Help me fix my accuracy contract test.

The evaluation metric now has its own boundary check. An invalid accuracy value becomes visible as soon as the tests run.

Protect the prediction contract

The risk model uses binary labels. Every prediction must therefore resolve to either 0 or 1.

  • In test_risk_model.py, find the test_accuracy_is_a_fraction() method.
  • Add this method directly below it:
    def test_prediction_is_a_binary_label(self):
        model, _ = train_model()
        prediction = predict_risk(model, [420, 68, 2, 4, 1])
        self.assertIn(prediction, (0, 1))

What Does This Test Protect?

  • The test trains a model through the existing train_model() function.
  • The sample metrics exercise the same five-value prediction path used by the command-line model.
  • The assertIn() assertion limits the returned label to 0 or 1.
  • Save test_risk_model.py.

Before you run the full suite, predict how many named tests should appear before the final status.

  • Run all three tests by entering:
.\.venv\Scripts\python.exe -m unittest -v test_risk_model

What Should I See?

  • The test_forbidden_feature_is_rejected test finishes successfully.
  • The test_accuracy_is_a_fraction test finishes successfully.
  • The test_prediction_is_a_binary_label test finishes successfully.
  • The complete run ends with OK.

One of the Three Tests Failing?

  • Check that all three test methods are indented inside RiskModelTests.
  • Confirm that the prediction test passes exactly five metrics to predict_risk().
  • Compare every imported identifier with its spelling in risk_model.py.

Help me diagnose the failing model test.

That is the model's safety net in place. The feature contract, accuracy range, and prediction labels are now checked together.

✔️ Awesome, I've got everything!

Your three tests pass. Double check that you saved test_risk_model.py.

ⓧ I'd like to double check the full code

import unittest

from risk_model import (
    FEATURE_NAMES,
    predict_risk,
    train_model,
    validate_feature_names,
)


class RiskModelTests(unittest.TestCase):
    def test_forbidden_feature_is_rejected(self):
        with self.assertRaises(ValueError):
            validate_feature_names(FEATURE_NAMES + ["rollback_occurred"])

    def test_accuracy_is_a_fraction(self):
        _, accuracy = train_model()
        self.assertGreaterEqual(accuracy, 0.0)
        self.assertLessEqual(accuracy, 1.0)

    def test_prediction_is_a_binary_label(self):
        model, _ = train_model()
        prediction = predict_risk(model, [420, 68, 2, 4, 1])
        self.assertIn(prediction, (0, 1))


if __name__ == "__main__":
    unittest.main()

Next, you will move these checks into GitHub Actions so every pushed version proves itself on a clean runner.

Run Tests with GitHub Actions

Your leakage-safe model already has three local tests. However, those checks protect the project only when someone remembers to run them.

GitHub Actions adds continuous integration to the repository. Every push runs the same tests on a clean runner.

In this step, get ready to:
  • Create a workflow that runs the model tests after every push.
  • Prepare a committed local repository with its project documentation.
  • Publish the repository and confirm its automated test job passes.
Create the CI workflow

GitHub Actions reads workflow definitions from .github/workflows. Your YAML file describes the clean environment plus every command needed to reproduce your local test check.

Why GitHub Actions instead of local-only testing?

Local testing depends on someone remembering to run the suite. GitHub Actions repeats the check whenever new code reaches the repository.

This approach keeps the project focused on automated verification. Standard GitHub-hosted runners are free for public repositories.

  • Switch back to the deployment-risk-checker project in Visual Studio Code.
  • Select the deployment-risk-checker folder in the file explorer sidebar.
  • Use the new folder control to create .github.

You should see .github inside the project folder.

  • Select the .github folder.
  • Use the new folder control to create workflows.

The file explorer should now show the nested path .github/workflows.

  • Select the workflows folder.
  • Use the new file control to create ci.yml.
  • Define the Python CI workflow in ci.yml by pasting this code:
name: Python CI

on: push

permissions:
  contents: read

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Check out repository
        uses: actions/checkout@v7
      - name: Set up Python
        uses: actions/setup-python@v7
        with:
          python-version: "3.13"
          cache: pip
      - name: Install dependencies
        run: python -m pip install -r requirements.txt
      - name: Run unit tests
        run: python -m unittest -v test_risk_model

What does this workflow do?

  • The push trigger starts the workflow whenever committed code reaches GitHub.
  • The test job uses ubuntu-latest as its clean hosted runner.
  • The checkout step uses actions/checkout@v7 to place the repository files on that runner.
  • The setup step uses actions/setup-python@v7 to provide Python 3.13 with pip caching.
  • The final steps install requirements.txt and run the same unit test module you checked locally.
  • Save ci.yml.
  • Expand the .github/workflows path in the file explorer sidebar.

You should see ci.yml inside .github/workflows. This confirms GitHub can discover the workflow after the repository is pushed.

Workflow file in the wrong place?

Check that ci.yml sits inside the workflows folder. That folder must sit inside .github.

Compare the spacing in your file with the full-code tab below. YAML uses indentation to group each job plus its steps.

Help me check my GitHub Actions workflow path and YAML indentation.

✔️ Awesome, I've got everything!

Great. Double check that you saved ci.yml inside .github/workflows.

ⓧ I'd like to double check the full code

name: Python CI

on: push

permissions:
  contents: read

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Check out repository
        uses: actions/checkout@v7
      - name: Set up Python
        uses: actions/setup-python@v7
        with:
          python-version: "3.13"
          cache: pip
      - name: Install dependencies
        run: python -m pip install -r requirements.txt
      - name: Run unit tests
        run: python -m unittest -v test_risk_model
Document and commit the project

A README explains what the project demonstrates before someone reads the code. This Markdown file also makes the educational data leakage example clear.

  • Select the deployment-risk-checker folder in the file explorer sidebar.
  • Use the new file control to create README.md.
  • Document the project in README.md by pasting this content:
# ML Deployment Risk Checker

A small, explainable machine-learning project that predicts deployment risk from pre-deployment engineering metrics.

## What it demonstrates

- Training and evaluating a decision-tree classifier
- Detecting target leakage from a post-deployment outcome
- Protecting the feature contract with unit tests
- Running the tests automatically with GitHub Actions

The dataset is synthetic and intentionally includes a leakage column for education. This model is not intended for production deployment decisions.

## Run on Windows

```powershell
python -m venv .venv
.\.venv\Scripts\python.exe -m pip install -r requirements.txt
.\.venv\Scripts\python.exe leaky_demo.py
.\.venv\Scripts\python.exe risk_model.py
.\.venv\Scripts\python.exe -m unittest -v test_risk_model
```

## Expected story

The naive demo reports perfect test accuracy because it uses `rollback_occurred`, which is only known after deployment. The corrected model excludes that column and uses only information available before release.

What does this README communicate?

The opening sections explain the model's purpose plus the engineering skills demonstrated by the repository. The Windows commands give another learner a repeatable path through the project.

The limitation statement identifies the dataset as synthetic. It also makes clear that the model is an educational example.

  • Save README.md.
  • Expand the deployment-risk-checker folder in the file explorer sidebar.

You should see README.md beside the model scripts plus requirements.txt.

README file missing?

Check that the filename is exactly README.md. Make sure it appears directly inside deployment-risk-checker.

Help me place README.md in the correct project folder.

✔️ Awesome, I've got everything!

Your project documentation is ready. Double check that README.md is saved.

ⓧ I'd like to double check the full code

# ML Deployment Risk Checker

A small, explainable machine-learning project that predicts deployment risk from pre-deployment engineering metrics.

## What it demonstrates

- Training and evaluating a decision-tree classifier
- Detecting target leakage from a post-deployment outcome
- Protecting the feature contract with unit tests
- Running the tests automatically with GitHub Actions

The dataset is synthetic and intentionally includes a leakage column for education. This model is not intended for production deployment decisions.

## Run on Windows

```powershell
python -m venv .venv
.\.venv\Scripts\python.exe -m pip install -r requirements.txt
.\.venv\Scripts\python.exe leaky_demo.py
.\.venv\Scripts\python.exe risk_model.py
.\.venv\Scripts\python.exe -m unittest -v test_risk_model
```

## Expected story

The naive demo reports perfect test accuracy because it uses `rollback_occurred`, which is only known after deployment. The corrected model excludes that column and uses only information available before release.

Before you run the suite again, do you expect the new documentation or workflow definition to change the model test results?

  • Run the complete test module again from the PowerShell window in deployment-risk-checker by using this command:
.\.venv\Scripts\python.exe -m unittest -v test_risk_model

What does this check prove?

This rerun proves the model contract still passes before you publish the repository. The workflow later runs the same test module with its own Python interpreter.

All three named tests should pass. The output should finish with OK.

That is the local checkpoint complete. Your repository is ready for an automated copy of the same test run.

Local tests stopped passing?

Check that PowerShell is still inside deployment-risk-checker. The test runner needs to find test_risk_model.py plus risk_model.py in that folder.

Help me diagnose my failing unit test run before I publish the repository.

A Git commit records the exact project state that GitHub receives. Your ignored .venv folder stays on your computer.

  • Initialize the main branch and create the first commit by running these commands:
git init -b main
git add .
git commit -m 'initial project version'

What do these Git commands do?

The first command turns deployment-risk-checker into a Git repository on the main branch. The second command stages every file that is not excluded by .gitignore.

The final command saves the staged project as its first version. This is the version you publish to GitHub.

You should see Git report a new commit on main. Its file summary should include the workflow plus the README.

Git commit did not complete?

If Git asks who you are, follow the instructions shown in PowerShell to configure your author name plus email. Run the commit commands again after Git accepts that configuration.

Check that you ran the commands from the deployment-risk-checker folder. Initializing Git from another folder tracks the wrong files.

Help me finish my first Git commit.

Push to GitHub and inspect CI

A public GitHub repository gives the workflow a clean runner plus a visible test result. That result shows whether the committed project works beyond your computer.

  • Go to GitHub in your browser.
  • Sign in to your GitHub account.
  • Click New repository.
  • Enter deployment-risk-checker as the repository name.
  • Select Public as the visibility.
  • Leave every initialization option unselected.
  • Click Create repository.

You should reach the Quick Setup page for an empty public repository named deployment-risk-checker.

A sign-in prompt may appear

The first push can ask you to authenticate with GitHub in your browser. This prompt connects Git on your computer to the repository you just created.

  • Copy the commands for pushing an existing repository from the Quick Setup page.
  • Return to the PowerShell window in deployment-risk-checker.
  • Paste the copied commands into PowerShell.
  • Run the copied commands.
  • Complete any browser sign-in prompt with the GitHub account that owns the repository.

Refresh the repository page after the push completes. You should see README.md, the Python files, plus the .github folder.

Your local project now has a public home. Every future push can prove that the leakage safeguards still pass.

Push rejected?

Check that the repository name on GitHub is deployment-risk-checker. Make sure the copied setup commands came from that repository.

If authentication failed, complete the browser sign-in prompt with the account that created the repository. Return to PowerShell after GitHub confirms the sign-in.

Help me push my local project to its GitHub repository.

Before you inspect the workflow, do you expect a clean hosted runner to reproduce the passing result from your computer?

  • Click the Actions tab in the repository.
  • Select Python CI from the workflow list.
  • Open the latest workflow run.
  • Select the test job.

You should see the latest test job marked as completed successfully. Its steps should show that Python 3.13 installed the dependency plus ran all three tests.

That is the full delivery loop working. A push now turns your local model tests into visible repository evidence.

Workflow job failed?

Open the failed step to read its log. An installation failure usually points to requirements.txt, while a test failure points to the committed Python files.

Confirm that ci.yml is committed under .github/workflows. Check that the workflow runs the same test module that passed locally.

Help me diagnose my failed Python CI job.

Your deployment risk checker is documented, published, and protected by automated tests on every push.

Secret mission

Compare Tree Depths with Cross-Validation

One holdout score can make a tree look stronger or weaker because of a lucky split. Compare four tree depths across four cross-validation folds. You will choose a depth using both average performance and score stability.

Clean Up Your Resources

Clean Up Your Resources

Your public GitHub repository uses GitHub Actions on a standard hosted runner at no cost while the repository remains public. Choose whether to keep everything, pause the workflow, or delete the project.

Resources you used:

  • Your local deployment-risk-checker folder containing the .venv environment, dataset, model scripts, tests, tuning experiment, README, workflow file, plus Git history.
  • Your public GitHub deployment-risk-checker repository containing the Python CI workflow in .github/workflows/ci.yml.

Keep everything running

No action needed. Choose this if you want to keep developing the model or show the tested repository as a finished project.

  • Keep the local deployment-risk-checker folder on your computer.
  • Leave the public repository available on GitHub.
  • Leave the Python CI workflow enabled so future pushes continue running the unit tests.

Pause - I'll come back to this later

Pause future CI runs while keeping every project file available. Your local files plus repository history remain intact.

  • Return to the public deployment-risk-checker repository on GitHub.
  • Select the Actions tab.
  • Select the Python CI workflow.
  • Use Disable workflow on the workflow page.
  • Keep the local deployment-risk-checker folder on your computer.
  • Re-enable the workflow from the same page when you are ready to resume automatic tests.

You now have a clean pause. The repository remains available while future pushes stop triggering the workflow.

Delete - I don't want to use this again

Deletion is the most drastic option, so pause for a moment before confirming it. This removes the public repository plus the local project folder.

  • Return to the public deployment-risk-checker repository on GitHub.
  • Select the Settings tab.
  • Scroll to the Danger Zone section.
  • Select Delete this repository.
  • Complete the confirmation prompts to remove the repository.
  • Return to the parent folder that contains deployment-risk-checker in File Explorer.
  • Select the deployment-risk-checker folder.
  • Delete the selected folder permanently using File Explorer's permanent-delete option.
  • Confirm that deployment-risk-checker is no longer listed in its parent folder.

Nice Work!

Nice Work!

You did it! Your deployment risk checker now tells an honest machine-learning story from a suspicious result to a tested repository with automated checks.

You've learned how to:

  • Built an interpretable binary classifier with a decision tree that predicts deployment risk from pre-deployment metrics.
  • Exposed data leakage in a perfect score by tracing it to the post-deployment rollback_occurred feature.
  • Protected the feature contract with unit tests that run on every push through GitHub Actions.
  • Secret Mission: Compared tree depths with four-fold cross-validation to choose complexity from average performance plus score stability.

Ready to quiz yourself?