Compile and Load-Test a Fraud Model

Introduction

30 Second Summary

A fraud score can look convincing during a quiet test. Moving that model into a busy service can expose mismatched predictions or slow requests.

In this project, you will build a local FastAPI fraud-scoring service powered by a LightGBM model that Hummingbird compiles to ONNX. You will use Locust to prove how model loading changes performance under concurrent requests.

What You'll Build

Your local API returns a fraud probability plus a binary flag for one transaction while CSV reports reveal the performance gained from sharing one loaded model.

By the end of this project, you'll have:

• A time-ordered fraud evaluation where the temporal holdout reveals average precision on future transactions.
• A portable ONNX artifact that loads in a separate current runtime. Its numerical parity confirms that its probabilities match LightGBM.
• A fraud-scoring endpoint at /score that returns a probability plus a fraud flag. Repeatable load-test evidence compares per-request loading with one shared session.
• Secret Mission: Tune ONNX Runtime intra-op threads under the same fixed load profile. You will identify the best request-rate and tail-latency trade-off for your machine.

Are there any prerequisites?

You need Windows 10 or newer. You also need internet access plus enough free disk space for two Python environments.

Step 1 includes the Python 3.11.9 setup, so an existing Python installation is optional.

Before We Start

Before the hands-on work begins, lock in what you are building and why it matters. You are committing to a portable fraud-scoring service whose runtime design holds up under concurrent requests.

Set Up Isolated Environments

The Hummingbird converter at version 0.4.12 depends on an older ONNX stack. A shared environment would carry that dependency ceiling into the fraud API.

Two Python virtual environments create a boundary between conversion and production inference. The converter lane holds the pinned model-building packages.

The serving lane uses ONNX Runtime version 1.31.0 without Hummingbird or LightGBM. This split proves the exported model can stand on its own.

In this step, get ready to:

• Verify Python 3.11.9 and create the project folder.
• Create two isolated environments with pinned dependencies.
• Run version checks that prove the environments are separate.

Prepare Python isolation

The Windows Python launcher selects the 3.11 release line for this project. Start by checking whether PowerShell can find it.

• Press the Windows key to open Windows search.
• Type PowerShell and press Enter to open it.
• Check the available Python 3.11 version by running this command:

py -3.11 --version

✔️ I see Python 3.11.9

Good, Python 3.11.9 is ready for both environments. Keep this PowerShell window open.

ⓧ I see another Python version

This project uses Python 3.11.9 so both dependency stacks resolve against the same interpreter.

• Visit the Python 3.11.9 release page.
• Download the Windows installer (64-bit).
• Run the downloaded installer with the recommended settings.
• Close PowerShell after the installation finishes.
• Reopen PowerShell through Windows search.
• Confirm the installed version by running this command:

py -3.11 --version

Python version still unchanged?

Close every PowerShell window before checking again. This gives the launcher a fresh view of the installed interpreters.

Still stuck? Help me make py -3.11 select Python 3.11.9 on Windows.

ⓧ Python 3.11 is unavailable

Python 3.11 or its Windows launcher is missing. Installing the recommended release provides both pieces.

• Visit the Python 3.11.9 release page.
• Download the Windows installer (64-bit).
• Run the downloaded installer with the recommended settings.
• Approve the installation if Windows requests permission.
• Close PowerShell after the installation finishes.
• Reopen PowerShell through Windows search.
• Confirm the installation by running this command:

py -3.11 --version

Launcher still unavailable?

Restart Windows if PowerShell still cannot find the newly installed launcher.

Need another pair of eyes? Help me troubleshoot the Python launcher installation.

A fixed project folder keeps the environments beside every file they support. PowerShell uses its current location as the destination for the environment folders.

• Move to your Desktop by running this command:

Set-Location -Path "$HOME\Desktop" -PassThru

PowerShell prints your Desktop path. This confirms where the project folder is created.

• Create the fraud-model-load-test folder and enter it by running these commands:

New-Item -ItemType Directory -Path "fraud-model-load-test"
Set-Location -Path "fraud-model-load-test" -PassThru

What do these commands do?

• New-Item creates the named directory on your Desktop.
• Set-Location makes that directory PowerShell's current location.
• PassThru prints the resulting path so you can confirm the move.

You should see a path ending in Desktop\fraud-model-load-test.

Project folder command failed?

If the folder already exists, keep it. Run only the location command from the code block to enter it.

Still stuck? Help me create and enter the project folder in PowerShell.

Each environment receives its own interpreter plus package directory. Their dependencies stay independent even though both folders sit inside the same project.

• Create both Python environments by running these commands:

py -3.11 -m venv .venv-convert
py -3.11 -m venv .venv-serve

Why use two environments?

The .venv-convert environment contains the older compilation stack. It exists only to train and export the model.

The .venv-serve environment contains the current API stack. It later proves the ONNX file works without the training framework.

• Confirm both environment folders exist by running:

Get-ChildItem -Name

You should see .venv-convert and .venv-serve in the output.

Missing an environment folder?

• Check that PowerShell's path ends in fraud-model-load-test.
• Rerun the environment command for the missing folder.

Need help? Help me find why one of my Python virtual environments was not created.

Install the pinned dependency stacks

Requirements files turn each environment into a repeatable dependency set. You will create both manifests in Visual Studio before installing them with the matching interpreter.

• Press the Windows key to open Windows search.
• Type Visual Studio and press Enter to open it.
• Select Open a local folder in the start window.
• Select the fraud-model-load-test folder on your Desktop.
• Click Select Folder.

You should see fraud-model-load-test in the Solution Explorer pane. The two hidden environment folders may stay out of view.

• Select the fraud-model-load-test folder in Solution Explorer.
• Press Ctrl+Shift+A to open the Quick Add dialog.
• Type requirements-converter.txt.
• Press Enter to create the file.
• Fill requirements-converter.txt with the pinned converter dependencies by pasting this content:

hummingbird-ml==0.4.12
lightgbm==4.7.0
numpy==1.26.4
onnx==1.16.1
onnxconverter-common==1.16.0
onnxruntime==1.17.3
scikit-learn==1.5.2
scipy==1.14.1

What does the converter manifest hold?

• LightGBM version 4.7.0 trains the fraud classifier.
• Hummingbird version 0.4.12 compiles the classifier into tensor operations.
• ONNX version 1.16.1 and ONNX Runtime version 1.17.3 satisfy Hummingbird's dependency ceiling.
• NumPy version 1.26.4 keeps the converter stack below the NumPy 2 boundary.

• Save requirements-converter.txt by pressing Ctrl+S.
• Confirm requirements-converter.txt appears in Solution Explorer.

Converter manifest missing?

Make sure the file name ends with .txt exactly once. Confirm it sits directly inside fraud-model-load-test.

Still stuck? Help me create the converter requirements file in Visual Studio.

• Select the fraud-model-load-test folder in Solution Explorer.
• Press Ctrl+Shift+A to open the Quick Add dialog.
• Type requirements-runtime.txt.
• Press Enter to create the file.
• Fill requirements-runtime.txt with the serving dependencies by pasting this content:

fastapi==0.143.3
locust==2.46.7
numpy==2.4.6
onnxruntime==1.31.0
pydantic==2.14.1
uvicorn==0.54.0

What does the runtime manifest hold?

• FastAPI version 0.143.3 provides the scoring API.
• Uvicorn version 0.54.0 runs the local service.
• Locust version 2.46.7 generates concurrent scoring requests.
• Pydantic version 2.14.1 validates each transaction payload.

• Save requirements-runtime.txt by pressing Ctrl+S.
• Confirm requirements-runtime.txt appears in Solution Explorer.

Runtime manifest missing?

Check that the Quick Add dialog created the file inside fraud-model-load-test. Check the filename for spelling differences.

Need help? Help me create the runtime requirements file in Visual Studio.

Use these tabs to compare both manifests with the required project state.

✔️ Awesome, I've got everything!

Both requirements files are saved in fraud-model-load-test. Switch back to PowerShell for installation.

ⓧ I'd like to double check the full code

Your requirements-converter.txt file should match this reference.

hummingbird-ml==0.4.12
lightgbm==4.7.0
numpy==1.26.4
onnx==1.16.1
onnxconverter-common==1.16.0
onnxruntime==1.17.3
scikit-learn==1.5.2
scipy==1.14.1

Your requirements-runtime.txt file should match this reference.

fastapi==0.143.3
locust==2.46.7
numpy==2.4.6
onnxruntime==1.31.0
pydantic==2.14.1
uvicorn==0.54.0

Keep the converter isolated

The converter environment contains older ONNX and PyTorch versions. Use it only with trusted project files.

The serving environment stays on the current inference stack. This is the boundary you validate at the end of the step.

The package downloads can take several minutes. A steady stream of download messages means the installation is still working.

• Switch back to the PowerShell window in fraud-model-load-test.
• Install the CPU-only PyTorch 2.4.1 wheel into .venv-convert by running:

.\.venv-convert\Scripts\python.exe -m pip install torch==2.4.1 --index-url https://download.pytorch.org/whl/cpu

Why install the CPU wheel?

CPU inference keeps the exported artifact portable across machines. It also keeps the later load tests on one repeatable execution provider.

PowerShell returns to the prompt after the CPU wheel finishes installing.

PyTorch installation failed?

• Confirm your internet connection can reach the official PyTorch package index.
• Confirm PowerShell's current path ends in fraud-model-load-test.
• Confirm the .venv-convert folder appears in the project directory.

Need help? Help me troubleshoot the CPU PyTorch installation.

• Install the remaining converter dependencies from requirements-converter.txt by running:

.\.venv-convert\Scripts\python.exe -m pip install -r requirements-converter.txt

What does this install change?

This command uses the Python executable inside .venv-convert. Every pinned converter package lands in that environment.

Calling the environment's interpreter directly removes any ambiguity about where pip installs the packages.

You should see the converter packages finish installing before PowerShell returns to the prompt.

Converter requirements failed?

• Confirm requirements-converter.txt is saved directly inside fraud-model-load-test.
• Compare every package pin with the full-code reference above.
• Rerun the command after correcting the file.

Still blocked? Help me troubleshoot the converter requirements installation.

• Install the serving dependencies from requirements-runtime.txt by running:

.\.venv-serve\Scripts\python.exe -m pip install -r requirements-runtime.txt

What does the runtime install contain?

This environment receives the API server and load-testing packages. It also receives the current ONNX Runtime version 1.31.0.

LightGBM and Hummingbird stay outside this environment. Their absence makes the later portability check meaningful.

You should see the runtime packages finish installing before PowerShell returns to the prompt.

Runtime requirements failed?

• Confirm requirements-runtime.txt is saved in fraud-model-load-test.
• Confirm the .venv-serve folder exists.
• Compare each runtime pin with the full-code reference.

Need another pair of eyes? Help me troubleshoot the runtime requirements installation.

Prove the runtime split

Package version checks expose the dependency boundary directly. Each command asks its own environment to report the packages that matter in that lane.

Before you run these checks, do you expect both environments to report the same ONNX Runtime version?

• Verify the converter stack and serving stack by running both commands:

.\.venv-convert\Scripts\python.exe -c "from importlib.metadata import version; print(version('hummingbird-ml'), version('onnxruntime'), version('torch'))"
.\.venv-serve\Scripts\python.exe -c "from importlib.metadata import version; print(version('onnxruntime'), version('fastapi'), version('locust'))"

What should the checks prove?

The converter line includes Hummingbird 0.4.12 plus ONNX Runtime 1.17.3. It also reports the PyTorch 2.4.1 CPU build.

The serving line includes ONNX Runtime 1.31.0. It also includes FastAPI 0.143.3 plus Locust 2.46.7.

That is the isolation boundary working. The converter can keep its older compatible stack while the fraud service moves forward with current inference packages.

Version check failed?

• Confirm PowerShell remains inside fraud-model-load-test.
• Confirm both requirements files were saved before installation.
• Rerun the failed environment's installation command.

Still seeing the wrong versions? Help me diagnose dependency drift between my two environments.

Your converter and serving stacks are now separated by a tested dependency boundary. Next up, you will train the time-ordered fraud baseline inside the converter environment.

Train a Time-Ordered Baseline

Your isolated converter stack is ready for model training. Now you need a baseline result that proves fraud scoring works before compilation begins.

A random split can place later transaction patterns into training. A temporal holdout protects the baseline by reserving the final 20 percent for evaluation.

Fraud cases are rare compared with legitimate transactions. Average precision gives you a score that reflects how well the model ranks those positive cases.

In this step, get ready to:

• Fetch the public fraud dataset in chronological order.
• Train a class-weighted fraud classifier against a future holdout.
• Save four artifacts for conversion plus serving.

Load the fraud dataset

The OpenML dataset with ID 43627 contains numeric transaction features plus a fraud label. Your script converts the features to float32 before training.

• Press the Windows key to open Windows search.
• Type Visual Studio and press Enter to open it.
• Use Visual Studio's folder opener to select the project folder shown in PowerShell.
• Create train_model.py beside the two requirements files using the file tree.
• Start the training script by pasting this code into train_model.py:

import json
import pickle
from pathlib import Path

import numpy as np
from lightgbm import LGBMClassifier
from sklearn.datasets import fetch_openml
from sklearn.metrics import average_precision_score

DATA_ID = 43627
TRAIN_FRACTION = 0.80
ARTIFACT_DIR = Path("artifacts")
MODEL_PATH = ARTIFACT_DIR / "fraud_model.pkl"
METADATA_PATH = ARTIFACT_DIR / "metadata.json"
SAMPLE_PATH = ARTIFACT_DIR / "sample_request.json"
PARITY_PATH = ARTIFACT_DIR / "parity_batch.npz"


def main() -> None:
    ARTIFACT_DIR.mkdir(exist_ok=True)

    dataset = fetch_openml(
        data_id=DATA_ID,
        as_frame=False,
        parser="auto",
    )
    features = np.asarray(dataset.data, dtype=np.float32)
    labels = np.asarray(dataset.target, dtype=np.int64)
    feature_names = list(dataset.feature_names)

What does this code do?

• The path constants keep every generated file inside artifacts.
• The call to fetch_openml() retrieves the fixed dataset ID used throughout the project.
• The features array uses float32 so its values match the later ONNX inference input.
• The labels array uses integers so each row has a binary class.

• Save train_model.py.
• Confirm the file tree shows train_model.py beside requirements-converter.txt.

Don't see the training file?

• Check that Visual Studio opened the same folder whose path appears in PowerShell.
• Check that the filename ends in .py.

Still stuck? Help me place train_model.py in the correct project folder.

Train with a future holdout

The Time feature gives each transaction a chronological position. A stable sort preserves the original order when two transactions share the same time value.

The first 80 percent becomes training data. The final 20 percent becomes the future holdout used to calculate average precision.

Why weight the positive class?

The class imbalance means legitimate transactions can dominate training. The calculated positive weight gives fraud examples more influence when LightGBM fits the classifier.

• In train_model.py, place the cursor after feature_names = list(dataset.feature_names).
• Add the chronological split plus classifier by pasting this code:

    time_column = feature_names.index("Time")
    chronological_order = np.argsort(features[:, time_column], kind="stable")
    features = features[chronological_order]
    labels = labels[chronological_order]

    split_index = int(len(features) * TRAIN_FRACTION)
    x_train = features[:split_index]
    y_train = labels[:split_index]
    x_holdout = features[split_index:]
    y_holdout = labels[split_index:]

    negatives = int(np.sum(y_train == 0))
    positives = int(np.sum(y_train == 1))
    positive_weight = negatives / max(positives, 1)

    model = LGBMClassifier(
        objective="binary",
        n_estimators=100,
        num_leaves=31,
        max_depth=8,
        learning_rate=0.05,
        class_weight={0: 1.0, 1: positive_weight},
        random_state=42,
        n_jobs=4,
    )
    model.fit(x_train, y_train)

if __name__ == "__main__":
    main()

How does the baseline work?

• The stable sort places rows into chronological order using Time.
• The array slices keep the chronologically first 80 percent for training.
• The positive_weight value scales fraud examples using the observed training labels.
• The classifier uses the fixed parameters from the project so later load tests share one reproducible model.

• Save train_model.py.

The first dataset fetch can take longer because the transaction data must be downloaded. Before you run the script, do you expect the future holdout to participate in model fitting?

• Run the initial training pass in the PowerShell window from earlier by running:

.\.venv-convert\Scripts\python.exe train_model.py

What does this command prove?

The command uses the pinned converter environment to fetch the data plus fit the classifier. The holdout remains outside model.fit().

PowerShell returns to its prompt after the initial training pass completes. That proves the chronological training path can run with your isolated dependencies.

Training didn't finish?

• Confirm PowerShell is still in the project folder containing train_model.py.
• Confirm your internet connection is available for the first OpenML download.
• Confirm the command starts with the Python executable inside .venv-convert.

Need a hand? Help me diagnose why the initial fraud-model training pass failed.

Save the evaluation artifacts

The trained classifier alone cannot describe its input contract. The service also needs a real request plus reference probabilities for later parity checks.

• In train_model.py, place the cursor on the blank line immediately above if __name__ == "__main__":.
• Add the evaluation plus first two artifact writers by pasting this code:

    holdout_probabilities = model.predict_proba(x_holdout)[:, 1]
    average_precision = float(
        average_precision_score(y_holdout, holdout_probabilities)
    )

    with MODEL_PATH.open("wb") as model_file:
        pickle.dump(model, model_file, protocol=pickle.HIGHEST_PROTOCOL)

    fraud_indices = np.flatnonzero(y_holdout == 1)
    sample_index = int(fraud_indices[0])
    SAMPLE_PATH.write_text(
        json.dumps(
            {"features": x_holdout[sample_index].astype(float).tolist()},
            indent=2,
        ),
        encoding="utf-8",
    )

What do these artifacts capture?

• The probability calculation scores every row in the untouched future holdout.
• The pickle stores the fitted classifier for Hummingbird conversion in the isolated environment.
• The sample request uses a real fraud-labelled holdout row with the same feature order used for training.

• Save train_model.py.
• Return to the PowerShell window from earlier.
• Create the model pickle plus sample request by running:

.\.venv-convert\Scripts\python.exe train_model.py

What should this run create?

The completed run writes fraud_model.pkl plus sample_request.json inside artifacts.

The sample contains one future-holdout transaction labelled as fraud. This gives the API a realistic request for later scoring.

• Refresh Visual Studio's file tree.

You should see fraud_model.pkl plus sample_request.json inside the new artifacts directory.

Missing the first artifacts?

• Check that the artifact-writing code remains indented inside main().
• Check that the new code appears above the final entry-point block.
• Check the PowerShell output for a Python exception before refreshing the file tree.

Still missing a file? Help me troubleshoot why the model pickle or sample request was not written.

The final artifact group preserves a compact probability reference plus the complete feature contract. These files let the next step test portability without fetching or retraining the model.

• Place the cursor on the blank line immediately above if __name__ == "__main__": again.
• Add the parity batch plus metadata writers by pasting this code:

    parity_size = min(2048, len(x_holdout))
    np.savez_compressed(
        PARITY_PATH,
        features=x_holdout[:parity_size],
        native_probabilities=holdout_probabilities[:parity_size],
    )

    metadata = {
        "data_id": DATA_ID,
        "feature_count": len(feature_names),
        "feature_names": feature_names,
        "train_rows": len(x_train),
        "holdout_rows": len(x_holdout),
        "holdout_average_precision": average_precision,
    }
    METADATA_PATH.write_text(
        json.dumps(metadata, indent=2),
        encoding="utf-8",
    )

    print(f"Train rows: {len(x_train):,}")
    print(f"Future holdout rows: {len(x_holdout):,}")
    print(f"Future holdout average precision: {average_precision:.6f}")
    print(f"Saved model: {MODEL_PATH}")
    print(f"Saved metadata: {METADATA_PATH}")
    print(f"Saved sample request: {SAMPLE_PATH}")
    print(f"Saved parity batch: {PARITY_PATH}")

How does this finish the contract?

• The compressed parity batch stores up to the first 2048 future-holdout rows plus their native model probabilities.
• The metadata records the dataset ID plus the ordered feature names.
• The metadata also records the split sizes plus the future-holdout score.
• The final print statements expose each result in PowerShell for a visible verification.

• Save train_model.py.

Before you run the finished script, predict which row count should be larger. Do you expect the training count or future-holdout count to win?

• Run the finished training pipeline by running:

.\.venv-convert\Scripts\python.exe train_model.py

What should I see?

PowerShell prints lines beginning with Train rows:, Future holdout rows:, plus Future holdout average precision:.

It also prints the paths for the model, metadata, sample request, plus parity batch. The training row count is larger because it represents 80 percent of the chronological data.

You have a working future-facing baseline. Its score comes from transactions that never participated in fitting.

• Expand the artifacts directory in Visual Studio's file tree.
• Open metadata.json.
• Confirm feature_count is 30.
• Open sample_request.json.
• Confirm its features array contains 30 numeric values.
• Confirm the directory also contains parity_batch.npz.

Output or contract doesn't match?

• Check that TRAIN_FRACTION remains 0.80.
• Check that the sort uses the Time feature plus kind="stable".
• Check that the sample comes from x_holdout at the first fraud-labelled index.

Need help comparing the result? Help me diagnose my row counts, average precision output, or 30-feature contract.

✔️ Awesome, I've got everything!

Your full training script is saved. The terminal output plus four artifacts confirm that the baseline is ready.

ⓧ I'd like to double check the full code

import json
import pickle
from pathlib import Path

import numpy as np
from lightgbm import LGBMClassifier
from sklearn.datasets import fetch_openml
from sklearn.metrics import average_precision_score

DATA_ID = 43627
TRAIN_FRACTION = 0.80
ARTIFACT_DIR = Path("artifacts")
MODEL_PATH = ARTIFACT_DIR / "fraud_model.pkl"
METADATA_PATH = ARTIFACT_DIR / "metadata.json"
SAMPLE_PATH = ARTIFACT_DIR / "sample_request.json"
PARITY_PATH = ARTIFACT_DIR / "parity_batch.npz"


def main() -> None:
    ARTIFACT_DIR.mkdir(exist_ok=True)

    dataset = fetch_openml(
        data_id=DATA_ID,
        as_frame=False,
        parser="auto",
    )
    features = np.asarray(dataset.data, dtype=np.float32)
    labels = np.asarray(dataset.target, dtype=np.int64)
    feature_names = list(dataset.feature_names)

    time_column = feature_names.index("Time")
    chronological_order = np.argsort(features[:, time_column], kind="stable")
    features = features[chronological_order]
    labels = labels[chronological_order]

    split_index = int(len(features) * TRAIN_FRACTION)
    x_train = features[:split_index]
    y_train = labels[:split_index]
    x_holdout = features[split_index:]
    y_holdout = labels[split_index:]

    negatives = int(np.sum(y_train == 0))
    positives = int(np.sum(y_train == 1))
    positive_weight = negatives / max(positives, 1)

    model = LGBMClassifier(
        objective="binary",
        n_estimators=100,
        num_leaves=31,
        max_depth=8,
        learning_rate=0.05,
        class_weight={0: 1.0, 1: positive_weight},
        random_state=42,
        n_jobs=4,
    )
    model.fit(x_train, y_train)

    holdout_probabilities = model.predict_proba(x_holdout)[:, 1]
    average_precision = float(
        average_precision_score(y_holdout, holdout_probabilities)
    )

    with MODEL_PATH.open("wb") as model_file:
        pickle.dump(model, model_file, protocol=pickle.HIGHEST_PROTOCOL)

    fraud_indices = np.flatnonzero(y_holdout == 1)
    sample_index = int(fraud_indices[0])
    SAMPLE_PATH.write_text(
        json.dumps(
            {"features": x_holdout[sample_index].astype(float).tolist()},
            indent=2,
        ),
        encoding="utf-8",
    )

    parity_size = min(2048, len(x_holdout))
    np.savez_compressed(
        PARITY_PATH,
        features=x_holdout[:parity_size],
        native_probabilities=holdout_probabilities[:parity_size],
    )

    metadata = {
        "data_id": DATA_ID,
        "feature_count": len(feature_names),
        "feature_names": feature_names,
        "train_rows": len(x_train),
        "holdout_rows": len(x_holdout),
        "holdout_average_precision": average_precision,
    }
    METADATA_PATH.write_text(
        json.dumps(metadata, indent=2),
        encoding="utf-8",
    )

    print(f"Train rows: {len(x_train):,}")
    print(f"Future holdout rows: {len(x_holdout):,}")
    print(f"Future holdout average precision: {average_precision:.6f}")
    print(f"Saved model: {MODEL_PATH}")
    print(f"Saved metadata: {METADATA_PATH}")
    print(f"Saved sample request: {SAMPLE_PATH}")
    print(f"Saved parity batch: {PARITY_PATH}")


if __name__ == "__main__":
    main()

Your fraud baseline now has a future-holdout score plus the reference data needed for portability checks. Next up, you'll compile the classifier with Hummingbird and test it in the current runtime.

Compile and Verify the ONNX Model

Your time-ordered LightGBM baseline now has a future-holdout score. Its saved reference probabilities give you a benchmark for every compiled prediction.

The conversion only earns trust when Hummingbird preserves those probabilities. The resulting ONNX graph must also load in the separate ONNX Runtime environment that serves it.

In this step, get ready to:

• Compile the trained classifier into an opset-13 ONNX graph.
• Validate probability parity inside the converter environment.
• Recheck the artifact with ONNX Runtime 1.31.0 in the serving environment.

Convert the LightGBM model

Hummingbird turns the trained classifier into tensor operations. This converter uses tree_trav with opset 13 to produce the portable graph.

• Use Visual Studio's new-file control to create convert_model.py inside the project folder.
• Paste the converter imports and artifact paths into convert_model.py using this code:

import json
import pickle
from pathlib import Path

import numpy as np
import onnx
from hummingbird.ml import constants, convert

ARTIFACT_DIR = Path("artifacts")
MODEL_PATH = ARTIFACT_DIR / "fraud_model.pkl"
PARITY_PATH = ARTIFACT_DIR / "parity_batch.npz"
ONNX_PATH = ARTIFACT_DIR / "fraud_hummingbird.onnx"
METADATA_PATH = ARTIFACT_DIR / "metadata.json"

What does this setup establish?

• The imports provide model loading, array handling, graph validation, conversion, and metadata updates.
• The path constants keep every generated artifact inside the existing artifacts directory.
• The converter reads the trained pickle and the saved parity batch from the previous step.

• Save convert_model.py.
• Confirm Visual Studio shows convert_model.py inside the project folder.

File missing from the project folder?

Check the file location in Visual Studio. Save it beside train_model.py instead of inside the artifacts directory.

Still stuck? Help me place convert_model.py in the correct project folder.

The first runnable slice loads the trained model and the parity batch. It compiles the model in memory before comparing Hummingbird probabilities with the saved LightGBM reference.

• Paste this code directly below the METADATA_PATH line in convert_model.py.

def main() -> None:
    with MODEL_PATH.open("rb") as model_file:
        model = pickle.load(model_file)

    with np.load(PARITY_PATH) as parity_data:
        features = parity_data["features"].astype(np.float32)
        native_probabilities = parity_data["native_probabilities"]

    hummingbird_model = convert(
        model,
        "onnx",
        test_input=features[:32],
        extra_config={
            constants.TREE_IMPLEMENTATION: "tree_trav",
            constants.ONNX_TARGET_OPSET: 13,
        },
    )

    hummingbird_probabilities = hummingbird_model.predict_proba(features)[:, 1]
    np.testing.assert_allclose(
        hummingbird_probabilities,
        native_probabilities,
        rtol=1e-3,
        atol=1e-5,
    )

    print("Hummingbird parity check passed.")


if __name__ == "__main__":
    main()

How does the first parity check work?

• The script loads only the model and reference data created by your training script.
• A 32-row sample gives Hummingbird the input shape it needs for ONNX conversion.
• The full parity batch then passes through the compiled model.
• The comparison allows a relative tolerance of 1e-3 and an absolute tolerance of 1e-5.

• Save convert_model.py.

Before you run this, predict whether compilation preserves the saved probabilities within tolerance.

• Run the in-memory conversion with the converter environment by running:

.\.venv-convert\Scripts\python.exe convert_model.py

What should you see?

You should see Hummingbird parity check passed. in PowerShell. This proves the compiled probability output matches the saved native reference within the defined tolerance.

Parity check did not pass?

• Confirm PowerShell is still in the project folder that contains convert_model.py.
• Confirm the existing artifacts directory contains fraud_model.pkl and parity_batch.npz.
• Check that the conversion configuration uses tree_trav and opset 13.

Need a hand? Help me diagnose my Hummingbird parity failure.

Save the portable ONNX artifact

The compiled graph currently exists only in memory. The next chunk validates its structure before saving it and records the graph contract in your existing metadata.

• Find the print("Hummingbird parity check passed.") line inside main().
• Paste this graph-saving block immediately above that line:

    onnx.checker.check_model(hummingbird_model.model)
    onnx.save(hummingbird_model.model, str(ONNX_PATH))

    metadata = json.loads(METADATA_PATH.read_text(encoding="utf-8"))
    metadata.update(
        {
            "onnx_input_name": hummingbird_model.model.graph.input[0].name,
            "onnx_output_names": [
                output.name for output in hummingbird_model.model.graph.output
            ],
            "onnx_opset": 13,
            "tree_implementation": "tree_trav",
        }
    )
    METADATA_PATH.write_text(
        json.dumps(metadata, indent=2),
        encoding="utf-8",
    )

    max_delta = float(
        np.max(np.abs(hummingbird_probabilities - native_probabilities))
    )
    print(f"Saved ONNX model: {ONNX_PATH}")
    print(f"Maximum converter-environment probability delta: {max_delta:.8f}")

What does this saving block do?

• The graph checker validates the in-memory model before it reaches disk.
• The save operation writes artifacts/fraud_hummingbird.onnx as an ONNX ModelProto.
• The metadata update records the input name, output names, opset, and tree implementation needed by later runtime code.
• The maximum delta turns parity into a concrete value you can inspect.

• Save convert_model.py.

Before you rerun the converter, predict which artifact path and probability delta it reports.

• Generate the validated ONNX artifact by running:

.\.venv-convert\Scripts\python.exe convert_model.py

What should you see now?

PowerShell should report artifacts\fraud_hummingbird.onnx as the saved model. It should also print the maximum converter-environment probability delta.

The final line should confirm that the Hummingbird parity check passed. Your portable graph now has a measured link back to the native classifier.

ONNX artifact was not saved?

• Check that the saving block remains indented inside main().
• Check that onnx.save receives hummingbird_model.model.
• Confirm the artifacts directory still contains metadata.json.

Still blocked? Help me troubleshoot why fraud_hummingbird.onnx was not created.

✔️ Awesome, I've got everything!

Your converter script now checks parity, validates the graph, saves the ONNX artifact, and updates the metadata.

ⓧ I'd like to double check the full code

import json
import pickle
from pathlib import Path

import numpy as np
import onnx
from hummingbird.ml import constants, convert

ARTIFACT_DIR = Path("artifacts")
MODEL_PATH = ARTIFACT_DIR / "fraud_model.pkl"
PARITY_PATH = ARTIFACT_DIR / "parity_batch.npz"
ONNX_PATH = ARTIFACT_DIR / "fraud_hummingbird.onnx"
METADATA_PATH = ARTIFACT_DIR / "metadata.json"


def main() -> None:
    with MODEL_PATH.open("rb") as model_file:
        model = pickle.load(model_file)

    with np.load(PARITY_PATH) as parity_data:
        features = parity_data["features"].astype(np.float32)
        native_probabilities = parity_data["native_probabilities"]

    hummingbird_model = convert(
        model,
        "onnx",
        test_input=features[:32],
        extra_config={
            constants.TREE_IMPLEMENTATION: "tree_trav",
            constants.ONNX_TARGET_OPSET: 13,
        },
    )

    hummingbird_probabilities = hummingbird_model.predict_proba(features)[:, 1]
    np.testing.assert_allclose(
        hummingbird_probabilities,
        native_probabilities,
        rtol=1e-3,
        atol=1e-5,
    )

    onnx.checker.check_model(hummingbird_model.model)
    onnx.save(hummingbird_model.model, str(ONNX_PATH))

    metadata = json.loads(METADATA_PATH.read_text(encoding="utf-8"))
    metadata.update(
        {
            "onnx_input_name": hummingbird_model.model.graph.input[0].name,
            "onnx_output_names": [
                output.name for output in hummingbird_model.model.graph.output
            ],
            "onnx_opset": 13,
            "tree_implementation": "tree_trav",
        }
    )
    METADATA_PATH.write_text(
        json.dumps(metadata, indent=2),
        encoding="utf-8",
    )

    max_delta = float(
        np.max(np.abs(hummingbird_probabilities - native_probabilities))
    )
    print(f"Saved ONNX model: {ONNX_PATH}")
    print(f"Maximum converter-environment probability delta: {max_delta:.8f}")
    print("Hummingbird parity check passed.")


if __name__ == "__main__":
    main()

How to use this reference

Compare this complete file with convert_model.py in Visual Studio. Pay close attention to indentation inside main() and the order of the parity check, graph save, metadata update, and output lines.

Verify the artifact in the serving runtime

The converter environment contains the training framework and Hummingbird. A portable artifact must reproduce the same probabilities in .venv-serve without importing either package.

• Use Visual Studio's new-file control to create verify_runtime.py inside the project folder.
• Paste the runtime imports and artifact paths into verify_runtime.py using this code:

from pathlib import Path

import numpy as np
import onnxruntime as ort

ARTIFACT_DIR = Path("artifacts")
MODEL_PATH = ARTIFACT_DIR / "fraud_hummingbird.onnx"
PARITY_PATH = ARTIFACT_DIR / "parity_batch.npz"

What defines the runtime boundary?

This file imports NumPy and ONNX Runtime only. The absence of Hummingbird and LightGBM proves the saved graph carries the prediction logic across environments.

• Save verify_runtime.py.
• Confirm Visual Studio shows verify_runtime.py beside convert_model.py.

• Paste this first runtime check directly below the PARITY_PATH line:

def main() -> None:
    with np.load(PARITY_PATH) as parity_data:
        features = parity_data["features"].astype(np.float32)
        native_probabilities = parity_data["native_probabilities"]

    session = ort.InferenceSession(
        str(MODEL_PATH),
        providers=["CPUExecutionProvider"],
    )
    input_name = session.get_inputs()[0].name
    probability_name = session.get_outputs()[1].name
    runtime_probabilities = session.run(
        [probability_name],
        {input_name: features},
    )[0][:, 1]

    print(f"ONNX Runtime version: {ort.__version__}")
    print(f"Input: {input_name}")
    print(f"Probability output: {probability_name}")


if __name__ == "__main__":
    main()

What does the runtime check load?

• The serving environment opens the saved graph with the CPU execution provider.
• The graph input comes from output metadata stored inside the ONNX model.
• Output index 1 selects the classifier probability tensor.
• The complete parity batch runs through the current runtime in one call.

• Save verify_runtime.py.
• Load the ONNX graph with the serving environment by running:

.\.venv-serve\Scripts\python.exe verify_runtime.py

What should the first runtime check show?

PowerShell should print ONNX Runtime version 1.31.0. It should also print the graph's input name and probability output name.

This proves the current runtime can load and execute the artifact. The final edit checks whether its probabilities still match.

Current runtime cannot load the model?

• Confirm you ran the command with the Python executable inside .venv-serve.
• Confirm artifacts/fraud_hummingbird.onnx exists in the project folder.
• Check that the provider name in verify_runtime.py is CPUExecutionProvider.

Need more help? Help me diagnose why ONNX Runtime cannot load my converted model.

The current ending reports graph names without enforcing numerical parity. Replace it with the assertion and measured delta that make portability testable.

• Find these three lines at the end of main() in verify_runtime.py.

    print(f"ONNX Runtime version: {ort.__version__}")
    print(f"Input: {input_name}")
    print(f"Probability output: {probability_name}")

What is missing from this ending?

These lines prove the graph executed. They do not yet reject probabilities that drift beyond the accepted tolerance.

• Replace those three lines with this final verification block:

    np.testing.assert_allclose(
        runtime_probabilities,
        native_probabilities,
        rtol=1e-3,
        atol=1e-5,
    )
    max_delta = float(np.max(np.abs(runtime_probabilities - native_probabilities)))

    print(f"ONNX Runtime version: {ort.__version__}")
    print(f"Input: {input_name}")
    print(f"Probability output: {probability_name}")
    print(f"Maximum current-runtime probability delta: {max_delta:.8f}")
    print("Current-runtime parity check passed.")

Why is this the decisive check?

The same relative and absolute tolerances now guard the serving runtime. A passing assertion connects current-runtime output directly to the original LightGBM probabilities.

The maximum delta shows the largest observed difference across the saved batch. You now have both a hard limit and a readable measurement.

• Save verify_runtime.py.

Before the final run, predict whether the old-stack artifact reproduces its reference probabilities in ONNX Runtime 1.31.0.

• Run the complete current-runtime parity check by running:

.\.venv-serve\Scripts\python.exe verify_runtime.py

What confirms portability?

PowerShell should print ONNX Runtime version 1.31.0 and the maximum current-runtime probability delta. The final line should report Current-runtime parity check passed.

That result proves the graph works outside the converter environment. The serving runtime needs neither Hummingbird nor LightGBM to reproduce the reference probabilities.

Current-runtime parity did not pass?

• Confirm both parity checks use rtol=1e-3.
• Confirm both parity checks use atol=1e-5.
• Check that the runtime selects output index 1 before slicing the fraud probability column.

Still seeing a mismatch? Help me compare my converter and current-runtime probability paths.

✔️ Awesome, I've got everything!

Your serving environment can load the compiled model and reproduce the saved native probabilities without the training stack.

ⓧ I'd like to double check the full code

from pathlib import Path

import numpy as np
import onnxruntime as ort

ARTIFACT_DIR = Path("artifacts")
MODEL_PATH = ARTIFACT_DIR / "fraud_hummingbird.onnx"
PARITY_PATH = ARTIFACT_DIR / "parity_batch.npz"


def main() -> None:
    with np.load(PARITY_PATH) as parity_data:
        features = parity_data["features"].astype(np.float32)
        native_probabilities = parity_data["native_probabilities"]

    session = ort.InferenceSession(
        str(MODEL_PATH),
        providers=["CPUExecutionProvider"],
    )
    input_name = session.get_inputs()[0].name
    probability_name = session.get_outputs()[1].name
    runtime_probabilities = session.run(
        [probability_name],
        {input_name: features},
    )[0][:, 1]

    np.testing.assert_allclose(
        runtime_probabilities,
        native_probabilities,
        rtol=1e-3,
        atol=1e-5,
    )
    max_delta = float(np.max(np.abs(runtime_probabilities - native_probabilities)))

    print(f"ONNX Runtime version: {ort.__version__}")
    print(f"Input: {input_name}")
    print(f"Probability output: {probability_name}")
    print(f"Maximum current-runtime probability delta: {max_delta:.8f}")
    print("Current-runtime parity check passed.")


if __name__ == "__main__":
    main()

How to use this reference

Compare this complete file with verify_runtime.py in Visual Studio. Check the provider, probability output index, tolerances, and final output lines.

That is the runtime boundary proven. Next, you will put the portable model behind an API and expose the cost of loading a new session for every request.

Load-Test the Naive Service

Your portable ONNX model already reproduces LightGBM probabilities in the current serving environment. The next question is whether its API design survives concurrent scoring.

One successful score can hide repeated model setup inside the request path. You will serve a deliberately naive FastAPI endpoint before measuring its behavior with a fixed Locust profile.

In this step, get ready to:

• Build a naive scoring API that creates a fresh ONNX Runtime session for every request.
• Create a repeatable load profile that submits the saved fraud transaction.
• Run 25 concurrent users before preserving the CSV baseline.

Build and probe the naive API

The API reads its required feature count from the metadata created earlier. Its validator rejects any request that does not contain the same 30 numeric features.

• Create app_naive.py inside the project folder with the file panel in Visual Studio using this first section:

import json
from pathlib import Path

import numpy as np
import onnxruntime as ort
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel

ARTIFACT_DIR = Path("artifacts")
MODEL_PATH = ARTIFACT_DIR / "fraud_hummingbird.onnx"
METADATA_PATH = ARTIFACT_DIR / "metadata.json"
FEATURE_COUNT = int(
    json.loads(METADATA_PATH.read_text(encoding="utf-8"))["feature_count"]
)

app = FastAPI(title="Naive Fraud Scoring API")


class FraudRequest(BaseModel):
    features: list[float]


def validate_features(features: list[float]) -> np.ndarray:
    if len(features) != FEATURE_COUNT:
        raise HTTPException(
            status_code=422,
            detail=f"Expected {FEATURE_COUNT} features, received {len(features)}.",
        )
    return np.asarray([features], dtype=np.float32)

What does this code do?

• The path constants point the service at the converted model plus its metadata.
• The FraudRequest model requires a list of numeric feature values.
• The validate_features() function compares the incoming list length with FEATURE_COUNT before converting one transaction into a 32-bit NumPy array.

• Save app_naive.py.
• Confirm app_naive.py appears beside the existing project scripts in Visual Studio.

Seeing import or metadata warnings?

• Check that app_naive.py sits in the same project folder as the artifacts folder.
• Confirm the selected Python environment is .venv-serve because the converter environment is isolated from the API stack.

Still stuck? Help me diagnose the imports or metadata path in app_naive.py.

A health route gives you a quick view of the service lifecycle. This version reports that no model session is held at application scope.

• Add this health route below validate_features() in app_naive.py using the following code:

@app.get("/health")
def health() -> dict[str, object]:
    return {"status": "ok", "model_loaded": False, "session_scope": "per_request"}

What does the health route show?

• The /health route confirms that the API process is responding.
• The model_loaded value stays false because this application does not keep a shared session in memory.
• The session_scope value records the lifecycle choice that this load test measures.

The Uvicorn process keeps this terminal occupied while the service is running. You will use another PowerShell terminal for the load test.

• Save app_naive.py.
• Start the health probe from the PowerShell window in your project folder by running this command:

.\.venv-serve\Scripts\python.exe -m uvicorn app_naive:app --host 127.0.0.1 --port 8000

What does this command do?

This starts app_naive:app on the local address 127.0.0.1:8000 using the serving environment. The terminal remains attached to the server so you can see request logs.

• Visit http://127.0.0.1:8000/health in your browser.

You will see a healthy response with model_loaded: false plus session_scope: per_request. That response confirms the API is live before its scoring route exists.

• Return to the server terminal.
• Press Ctrl+C to stop the health probe.

You will see the PowerShell prompt again. The file is ready for its scoring route.

Health page not responding?

• Confirm the server command was run from the project folder containing app_naive.py.
• Check the server terminal for an import failure or a missing artifacts/metadata.json path.
• Stop any other local process already using port 8000 before retrying.

Need a hand? Help me get the naive API health route responding.

The scoring route performs validation before running the converted model. Every accepted request also constructs a fresh CPU inference session inside the handler.

• Add the scoring route below health() in app_naive.py using this code:

@app.post("/score")
def score_transaction(request: FraudRequest) -> dict[str, object]:
    features = validate_features(request.features)
    session = ort.InferenceSession(
        str(MODEL_PATH),
        providers=["CPUExecutionProvider"],
    )
    input_name = session.get_inputs()[0].name
    probability_name = session.get_outputs()[1].name
    fraud_probability = float(
        session.run([probability_name], {input_name: features})[0][0, 1]
    )
    return {
        "fraud_probability": fraud_probability,
        "fraud_flag": fraud_probability >= 0.5,
        "session_scope": "per_request",
    }

What happens during each score?

• The handler passes the request through validate_features() before inference.
• The InferenceSession constructor loads the ONNX model with the CPU execution provider for every accepted request.
• The handler selects probability output index 1 before reading the fraud probability.
• The response applies a threshold of 0.5 before returning the fraud flag.

• Save app_naive.py.
• Start the completed naive service by running this command:

.\.venv-serve\Scripts\python.exe -m uvicorn app_naive:app --host 127.0.0.1 --port 8000

What is running now?

The completed API now accepts local requests at /score. Keep this PowerShell window running so the load test can reach it.

You will see server startup logs in PowerShell. The service is now waiting for the saved fraud payload.

Server fails after adding the score route?

• Check that the scoring route sits below the complete health() function.
• Confirm artifacts/fraud_hummingbird.onnx still exists in the project folder.
• Compare the indentation inside score_transaction() with the reference code.

Still seeing a failure? Help me troubleshoot the naive scoring route.

✔️ Awesome, I've got everything!

Your complete naive API is saved. Keep the server running for the load test.

ⓧ I'd like to double check the full code

import json
from pathlib import Path

import numpy as np
import onnxruntime as ort
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel

ARTIFACT_DIR = Path("artifacts")
MODEL_PATH = ARTIFACT_DIR / "fraud_hummingbird.onnx"
METADATA_PATH = ARTIFACT_DIR / "metadata.json"
FEATURE_COUNT = int(
    json.loads(METADATA_PATH.read_text(encoding="utf-8"))["feature_count"]
)

app = FastAPI(title="Naive Fraud Scoring API")


class FraudRequest(BaseModel):
    features: list[float]


def validate_features(features: list[float]) -> np.ndarray:
    if len(features) != FEATURE_COUNT:
        raise HTTPException(
            status_code=422,
            detail=f"Expected {FEATURE_COUNT} features, received {len(features)}.",
        )
    return np.asarray([features], dtype=np.float32)


@app.get("/health")
def health() -> dict[str, object]:
    return {"status": "ok", "model_loaded": False, "session_scope": "per_request"}


@app.post("/score")
def score_transaction(request: FraudRequest) -> dict[str, object]:
    features = validate_features(request.features)
    session = ort.InferenceSession(
        str(MODEL_PATH),
        providers=["CPUExecutionProvider"],
    )
    input_name = session.get_inputs()[0].name
    probability_name = session.get_outputs()[1].name
    fraud_probability = float(
        session.run([probability_name], {input_name: features})[0][0, 1]
    )
    return {
        "fraud_probability": fraud_probability,
        "fraud_flag": fraud_probability >= 0.5,
        "session_scope": "per_request",
    }

Create and run the fixed load profile

A fixed load profile keeps the payload plus request timing consistent. That consistency turns the CSV into a baseline you can compare with the same test later.

• Create locustfile.py inside the project folder with this code:

import json
from pathlib import Path

from locust import HttpUser, between, task

PAYLOAD = json.loads(
    Path("artifacts/sample_request.json").read_text(encoding="utf-8")
)


class FraudScoringUser(HttpUser):
    wait_time = between(0.01, 0.05)

    @task
    def score_transaction(self) -> None:
        self.client.post("/score", json=PAYLOAD)

What does this load profile do?

• The PAYLOAD constant loads the 30-feature fraud transaction saved during training.
• The FraudScoringUser class models one concurrent API user.
• The wait time varies between 0.01 seconds and 0.05 seconds between tasks.
• Each task sends the same payload to /score so model lifecycle remains the changing pressure point.

• Save locustfile.py.
• Confirm locustfile.py appears beside app_naive.py in Visual Studio.

Load profile cannot find the payload?

• Check that locustfile.py is in the project folder containing the artifacts folder.
• Confirm artifacts/sample_request.json still contains the saved transaction from training.

Need help? Help me fix the Locust payload path.

✔️ Awesome, I've got everything!

Your repeatable load profile is saved beside the API.

ⓧ I'd like to double check the full code

import json
from pathlib import Path

from locust import HttpUser, between, task

PAYLOAD = json.loads(
    Path("artifacts/sample_request.json").read_text(encoding="utf-8")
)


class FraudScoringUser(HttpUser):
    wait_time = between(0.01, 0.05)

    @task
    def score_transaction(self) -> None:
        self.client.post("/score", json=PAYLOAD)

• Create a results folder inside the project folder with Visual Studio's file panel.
• Confirm the results folder appears beside artifacts.
• Create a second PowerShell terminal in Visual Studio's terminal panel.
• Confirm the second terminal points to the project folder containing .venv-serve.

Before you run the profile, do you expect 25 concurrent users to reveal a lifecycle cost that one health request could not show?

This test runs for 30 seconds. Keep the API terminal running until the load-test terminal prints its final statistics.

• Run the fixed 25-user load profile from the second PowerShell terminal with this command:

.\.venv-serve\Scripts\python.exe -m locust -f locustfile.py --headless --users 25 --spawn-rate 5 --run-time 30s --host http://127.0.0.1:8000 --csv results\naive

What does this command measure?

• The --headless option runs the profile without the Locust web interface.
• The --users 25 setting raises the test to 25 concurrent users.
• The --spawn-rate 5 setting adds five users each second.
• The --run-time 30s setting fixes the measurement window at 30 seconds.
• The --csv results\naive setting saves the statistics under the naive baseline name.

The test completes with zero application failures. Every recorded scoring latency still includes construction of a fresh inference session, which is the intended lifecycle bottleneck.

Load test fails or records errors?

• Confirm the first PowerShell terminal still shows the Uvicorn server running.
• Check that the load-test command runs from the folder containing locustfile.py.
• Confirm the saved payload still contains exactly 30 numeric feature values.

Still seeing failures? Help me troubleshoot the naive Locust run.

Stop the service and inspect the baseline

Before you inspect the CSV, do you expect the concurrent run to leave the per-request session cost visible in its latency statistics?

• Return to the first PowerShell terminal running Uvicorn.
• Press Ctrl+C to stop the naive service.
• Open results/naive_stats.csv in Visual Studio.
• Find the aggregate row at the bottom of the report.
• Confirm the aggregate failure count is zero.
• Identify the recorded response-time statistics plus request rate.

You now have results/naive_stats.csv with zero application failures plus measured latency and request rate. That report preserves the cost of constructing an inference session inside every scoring request.

You have a repeatable bottleneck baseline backed by CSV evidence. Next, you will change the model lifecycle before running the same profile again.

Load Once and Compare Results

Your naive service proved that a correct response can still hide an expensive lifecycle decision. Its baseline now gives you evidence to test a better design.

This version uses FastAPI lifespan to create one shared ONNX Runtime session before traffic arrives. Each request then validates input before using that session for inference.

In this step, get ready to:

• Create a lifespan-managed shared inference session.
• Complete the optimized health and scoring API.
• Benchmark the optimized service against the naive baseline.

Create the shared session during startup

The optimized service needs one place to hold the model session throughout the application lifecycle. The shared RUNTIME dictionary gives the startup function that place.

• In Visual Studio, create app.py beside app_naive.py.
• Add the imports and shared runtime state by pasting this code into app.py:

import json
from contextlib import asynccontextmanager
from pathlib import Path

import numpy as np
import onnxruntime as ort
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel

ARTIFACT_DIR = Path("artifacts")
MODEL_PATH = ARTIFACT_DIR / "fraud_hummingbird.onnx"
METADATA_PATH = ARTIFACT_DIR / "metadata.json"
FEATURE_COUNT = int(
    json.loads(METADATA_PATH.read_text(encoding="utf-8"))["feature_count"]
)
RUNTIME: dict[str, object] = {}

What does this code set up?

• The path constants point the service at the portable model and its metadata.
• FEATURE_COUNT reads the expected input size from the training metadata.
• RUNTIME starts empty so startup can populate it with shared inference objects.

• Save app.py.
• Confirm Visual Studio lists app.py beside app_naive.py.

Can't find app.py?

Confirm Visual Studio is showing the folder that contains app_naive.py and the artifacts folder.

Still stuck? Help me place app.py in the correct project folder.

A lifespan function runs its setup code before the API accepts requests. Its cleanup code runs when the server shuts down.

• Add the lifespan function below RUNTIME: dict[str, object] = {} by pasting this code:

@asynccontextmanager
async def lifespan(app: FastAPI):
    session = ort.InferenceSession(
        str(MODEL_PATH),
        providers=["CPUExecutionProvider"],
    )
    RUNTIME["session"] = session
    RUNTIME["input_name"] = session.get_inputs()[0].name
    RUNTIME["probability_name"] = session.get_outputs()[1].name
    print("Loaded one shared ONNX Runtime session.")
    yield
    RUNTIME.clear()


app = FastAPI(title="Fraud Scoring API", lifespan=lifespan)

How does the lifespan work?

• InferenceSession loads the ONNX model once with the CPU execution provider.
• The input and probability output names are discovered once during startup.
• yield marks the point where FastAPI begins accepting requests.
• RUNTIME.clear() releases the shared references during shutdown.

• Save app.py.
• Return to the PowerShell window that ran the naive server earlier.
• Start the optimized service by running this command:

.\.venv-serve\Scripts\python.exe -m uvicorn app:app --host 127.0.0.1 --port 8000

You'll see Loaded one shared ONNX Runtime session. before the server begins accepting requests. That message proves model construction has moved into application startup.

Service failed during startup?

• Confirm PowerShell is using the folder that contains app.py.
• Confirm artifacts\fraud_hummingbird.onnx remains inside that folder.
• Confirm every earlier code chunk in app.py is saved.

Need another pair of eyes? Help me debug the optimized service startup.

• Stop the startup check by pressing Ctrl+C in PowerShell.

Add the API routes and verify health

The shared session is now ready before traffic arrives. The remaining code preserves the same 30-feature request contract.

• Add the request model and validation function below app = FastAPI(title="Fraud Scoring API", lifespan=lifespan) by pasting this code:

class FraudRequest(BaseModel):
    features: list[float]


def validate_features(features: list[float]) -> np.ndarray:
    if len(features) != FEATURE_COUNT:
        raise HTTPException(
            status_code=422,
            detail=f"Expected {FEATURE_COUNT} features, received {len(features)}.",
        )
    return np.asarray([features], dtype=np.float32)

What does validation protect?

• FraudRequest requires a numeric feature list in the request body.
• validate_features() rejects payloads that do not contain exactly 30 values.
• The accepted values become one float32 row for ONNX Runtime.

• Save app.py.
• Confirm validate_features() contains the status_code=422 response for an invalid feature count.

Seeing validation syntax errors?

Check the indentation beneath validate_features(). Confirm the closing parenthesis for HTTPException appears before the NumPy return line.

Need help matching the function? Help me fix validate_features().

The health route exposes whether startup populated the shared runtime state. It gives you a direct check before load reaches the scoring route.

• Add the health route below validate_features() by pasting this code:

@app.get("/health")
def health() -> dict[str, object]:
    return {
        "status": "ok",
        "model_loaded": "session" in RUNTIME,
        "session_scope": "application_lifespan",
    }

What does health reveal?

• model_loaded checks whether the lifespan function stored the shared session.
• session_scope identifies the session as an application-level resource.

• Save app.py.
• Return to the server PowerShell window.
• Start the service with its health route by running this command:

.\.venv-serve\Scripts\python.exe -m uvicorn app:app --host 127.0.0.1 --port 8000

The startup message appears once because one session now belongs to the application lifecycle.

Before you check the endpoint, do you expect its lifecycle fields to match the naive service or show a different model state?

• Enter http://127.0.0.1:8000/health in your browser address bar.

You'll see model_loaded: true with session_scope: application_lifespan. That is the lifecycle fix working before a scoring request arrives.

Health still shows the naive lifecycle?

• Confirm the server target is app:app.
• Confirm the naive server is no longer using port 8000.
• Refresh the health page after restarting the optimized service.

Still seeing the old values? Help me identify which FastAPI application is running.

• Stop the server by pressing Ctrl+C in PowerShell.

The scoring route now needs to reuse the objects stored during startup. Model construction stays completely outside the request path.

• Add the optimized scoring route below health() by pasting this code:

@app.post("/score")
def score_transaction(request: FraudRequest) -> dict[str, object]:
    features = validate_features(request.features)
    session = RUNTIME["session"]
    input_name = RUNTIME["input_name"]
    probability_name = RUNTIME["probability_name"]
    fraud_probability = float(
        session.run([probability_name], {input_name: features})[0][0, 1]
    )
    return {
        "fraud_probability": fraud_probability,
        "fraud_flag": fraud_probability >= 0.5,
        "session_scope": "application_lifespan",
    }

What changed in the request path?

• The handler validates the same 30 numeric features as the naive service.
• The session and model input names come from RUNTIME.
• session.run() performs inference without rebuilding the model session.
• The response preserves the probability and 0.5 fraud threshold.

• Save app.py.
• Return to the server PowerShell window.
• Start the completed optimized service by running this command:

.\.venv-serve\Scripts\python.exe -m uvicorn app:app --host 127.0.0.1 --port 8000

You'll see the shared-session startup message once. The service is now ready for the fixed scoring workload.

Scoring service won't start?

• Confirm the scoring route sits below the completed health() function.
• Confirm RUNTIME uses the keys session, input_name, and probability_name.
• Confirm the lifespan function saves each of those keys before yield.

Need help tracing the startup error? Help me debug app.py.

✔️ Awesome, I've got everything!

Great, your optimized API now loads one shared session during startup. Double-check that app.py is saved.

ⓧ I'd like to double check the full code

import json
from contextlib import asynccontextmanager
from pathlib import Path

import numpy as np
import onnxruntime as ort
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel

ARTIFACT_DIR = Path("artifacts")
MODEL_PATH = ARTIFACT_DIR / "fraud_hummingbird.onnx"
METADATA_PATH = ARTIFACT_DIR / "metadata.json"
FEATURE_COUNT = int(
    json.loads(METADATA_PATH.read_text(encoding="utf-8"))["feature_count"]
)
RUNTIME: dict[str, object] = {}


@asynccontextmanager
async def lifespan(app: FastAPI):
    session = ort.InferenceSession(
        str(MODEL_PATH),
        providers=["CPUExecutionProvider"],
    )
    RUNTIME["session"] = session
    RUNTIME["input_name"] = session.get_inputs()[0].name
    RUNTIME["probability_name"] = session.get_outputs()[1].name
    print("Loaded one shared ONNX Runtime session.")
    yield
    RUNTIME.clear()


app = FastAPI(title="Fraud Scoring API", lifespan=lifespan)


class FraudRequest(BaseModel):
    features: list[float]


def validate_features(features: list[float]) -> np.ndarray:
    if len(features) != FEATURE_COUNT:
        raise HTTPException(
            status_code=422,
            detail=f"Expected {FEATURE_COUNT} features, received {len(features)}.",
        )
    return np.asarray([features], dtype=np.float32)


@app.get("/health")
def health() -> dict[str, object]:
    return {
        "status": "ok",
        "model_loaded": "session" in RUNTIME,
        "session_scope": "application_lifespan",
    }


@app.post("/score")
def score_transaction(request: FraudRequest) -> dict[str, object]:
    features = validate_features(request.features)
    session = RUNTIME["session"]
    input_name = RUNTIME["input_name"]
    probability_name = RUNTIME["probability_name"]
    fraud_probability = float(
        session.run([probability_name], {input_name: features})[0][0, 1]
    )
    return {
        "fraud_probability": fraud_probability,
        "fraud_flag": fraud_probability >= 0.5,
        "session_scope": "application_lifespan",
    }

Repeat the load test and compare reports

The optimized service now faces the same Locust workload as the naive version. Keeping the payload and load profile fixed isolates the model lifecycle as the changed variable.

• Switch back to the second PowerShell window used for the earlier Locust run.

This run lasts 30 seconds with the same 25 concurrent users. Let it finish so both CSV reports cover the same test duration.

Before you run this, do you expect removing session construction from /score to affect latency, request rate, or both?

• Run the fixed profile against the optimized service with this command:

.\.venv-serve\Scripts\python.exe -m locust -f locustfile.py --headless --users 25 --spawn-rate 5 --run-time 30s --host http://127.0.0.1:8000 --csv results\optimized

When the run finishes, you'll see results\optimized_stats.csv beside the naive report. You now have two measurements from the same workload.

Optimized report missing?

• Confirm the optimized Uvicorn server is still running in the first PowerShell window.
• Confirm the Locust command completed its full 30-second duration.
• Confirm Visual Studio still shows the existing results folder.

Need help with the load run? Help me troubleshoot the missing optimized CSV report.

• Open results\naive_stats.csv in Visual Studio.
• Open results\optimized_stats.csv in Visual Studio.
• Locate the aggregate row in each report.
• Confirm the optimized aggregate row records zero application failures.
• Compare the request rate across both aggregate rows.
• Compare the median response time across both aggregate rows.
• Compare the tail percentiles across both aggregate rows.

How should you read the comparison?

• Failures confirm whether both services completed the same workload successfully.
• Request rate shows how much scoring work each service completed per second.
• Median response time describes the typical request experience.
• Tail percentiles expose the slower requests that averages can hide.

Together, these measures show the effect of moving model construction out of the request path.

• Return to the server PowerShell window.
• Stop the optimized server by pressing Ctrl+C.

You have replaced per-request model construction with one lifespan-managed session. Your two CSV reports now turn that lifecycle decision into measurable evidence.

Secret mission

Tune ONNX Runtime Threads Under Load

One shared session removed the per-request lifecycle cost. Now you will compare two ONNX Runtime intra-op thread settings under the same fixed load profile. Your CSV evidence will reveal which setting gives your machine the better request-rate and tail-latency trade-off.

Clean Up Your Resources

Clean Up Your Resources

All work in this project runs locally. Keeping the files on your machine creates no ongoing cost.

Choose whether to preserve the benchmark evidence, pause any active processes, or remove the local resources.

Resources you used:

• The .venv-convert environment containing Hummingbird, LightGBM, PyTorch, and the legacy ONNX Runtime stack.
• The .venv-serve environment containing FastAPI, Uvicorn, Locust, and the current ONNX Runtime packages.
• The artifacts folder containing fraud_hummingbird.onnx, fraud_model.pkl, metadata.json, parity_batch.npz, and sample_request.json.
• The results folder containing naive_stats.csv, optimized_stats.csv, threads_1_stats.csv, and threads_0_stats.csv.
• The project scripts train_model.py, convert_model.py, verify_runtime.py, app_naive.py, app.py, app_tuned.py, and locustfile.py.
• The dependency files requirements-converter.txt and requirements-runtime.txt.

Protect the Converter Boundary

The converter environment uses the older package lane required by Hummingbird. It belongs only to the trusted files created during this project.

• Keep untrusted pickle files out of .venv-convert.
• Keep untrusted ONNX files out of .venv-convert.

Keep everything running

No action is needed. Choose this option if you want to repeat the benchmarks or continue experimenting with the service.

• Keep both virtual environments if you plan to retrain the model or repeat the conversion.
• Retain artifacts/fraud_hummingbird.onnx with verify_runtime.py so you can repeat the current-runtime parity check.
• Retain the reports in results so you can compare future lifecycle or thread-tuning experiments.
• Keep requirements-converter.txt and requirements-runtime.txt so you can recreate either environment.

Pause - I'll come back to this later

Shut down any local test processes to free their CPU and memory. Your environments, artifacts, and reports remain available at no cost.

• Switch back to the PowerShell window running Uvicorn if it is still active.
• Stop Uvicorn by pressing Ctrl+C.
• Switch back to the PowerShell window running Locust if it is still active.
• Stop Locust by pressing Ctrl+C.

Your machine is now idle while every model artifact and benchmark report stays ready for another run.

Delete - I don't want to use this again

Deleting these local resources is permanent. A backup of any CSV evidence protects the comparisons you may want later.

• Copy any benchmark reports you want to preserve to a separate folder.
• Switch back to the PowerShell window for this project.
• Delete the temporary converter environment by running this command:

Remove-Item -Recurse -Force .venv-convert

What Does This Remove?

This command permanently removes .venv-convert and every package installed inside it. The validated ONNX artifact remains in artifacts.

• Confirm that .venv-convert is no longer present in the project folder.

Converter Environment Still Present?

A local Python process may still be using files inside the environment.

• Stop any Python process that is using .venv-convert.
• Run the removal command again.
• Help me remove a locked converter environment.

• Press the Windows key to open system search.
• Type File Explorer into the search field.
• Press Enter to open File Explorer.
• Navigate to the folder that contains app.py, artifacts, and results.
• Delete the .venv-serve folder.
• Delete the artifacts folder.
• Delete the results folder.
• Delete the project scripts and both requirements files listed above.

That closes the local runtime boundary. The old converter stack and the current serving stack are both off your machine.

• Confirm that the folder no longer contains any resource listed in this cleanup section.

Nice Work!

Nice Work!

You did it! Your local FastAPI fraud-scoring API now serves a portable ONNX model through one shared ONNX Runtime session.

You've learned how to:

• Trained a class-weighted LightGBM fraud classifier with a temporal holdout. Measured average precision on future transactions.
• Compiled the model into a portable ONNX artifact at opset 13 using Hummingbird's tree_trav implementation. Proved numerical parity inside the converter environment. Repeated the check in current ONNX Runtime 1.31.0.
• Reproduced the per-request lifecycle bottleneck under concurrent Locust traffic. Replaced it with a FastAPI lifespan-managed session. Compared results/optimized_stats.csv against results/naive_stats.csv under the same fixed load profile.
• Completed the optional Secret Mission by benchmarking INTRA_OP_THREADS = 1. Repeated the fixed profile with INTRA_OP_THREADS = 0. Identified the local setting with the better request-rate and tail-latency trade-off. Kept the GNN as a separate upstream enrichment project.

Ready to quiz yourself?